Publications

2025

1. NeurIPS

   egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-World Tasks

   Matthias Jammot^*, Björn Braun^*, Paul Streli, Rafael Wampfler, and Christian Holz

   To appear in Advances in Neural Information Processing Systems (NeurIPS), 2025

   *Equal contribution

   Abstract Bib

   Understanding affect is central to anticipating human behavior, yet current egocentric vision benchmarks largely ignore the person’s emotional states that shape their decisions and actions. Existing tasks in egocentric perception focus on physical activities, hand-object interactions, and attention modeling—assuming neutral affect and uniform personality. This limits the ability of vision systems to capture key internal drivers of behavior. In this paper, we present egoEMOTION, the first dataset that couples egocentric visual and physiological signals with dense self-reports of emotion and personality across controlled and real-world scenarios. Our dataset includes over 50 hours of recordings from 43 participants, captured using Meta’s Project Aria glasses. Each session provides synchronized eye-tracking video, head-mounted photoplethysmography, inertial motion data, and physiological baselines for reference. Participants completed emotion-elicitation tasks and naturalistic activities while self-reporting their affective state using the Circumplex Model and Mikels’ Wheel as well as their personality via the Big Five model. We define three benchmark tasks: (1) continuous affect classification (valence, arousal, dominance); (2) discrete emotion classification; and (3) trait-level personality inference. We show that a classical learning-based method, as a simple baseline in real-world affect prediction, produces better estimates from signals captured on egocentric vision systems than processing physiological signals. Our dataset establishes emotion and personality as core dimensions in egocentric perception and opens new directions in affect-driven modeling of behavior, intent, and interaction.

   @article{jammot2025egoemotion,
     title = {egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-World Tasks},
     author = {Jammot, Matthias and Braun, Bj{\"o}rn and Streli, Paul and Wampfler, Rafael and Holz, Christian},
     journal = {Advances in Neural Information Processing Systems},
     year = {2025},
     highlighted = {true}
   }

2. NeurIPS

   Contimask: Explaining Irregular Time Series Models via Perturbations in Continuous Time

   Max Moebus, Björn Braun, and Christian Holz

   To appear in Advances in Neural Information Processing Systems (NeurIPS), 2025

   Bib

   @article{moebus2025contimask,
     title = {Contimask: Explaining Irregular Time Series Models via Perturbations in Continuous Time},
     author = {Moebus, Max and Braun, Bj{\"o}rn and Holz, Christian},
     journal = {Advances in Neural Information Processing Systems},
     year = {2025},
   }

3. ICCV

   

   egoPPG: Heart Rate Estimation from Eye-Tracking Cameras in Egocentric Systems to Benefit Downstream Vision Tasks

   Björn Braun, Rayan Armani, Manuel Meier, Max Moebus, and Christian Holz

   To appear in IEEE/CVF International Conference on Computer Vision (ICCV), 2025

   Abstract arXiv PDF Bib

   Egocentric vision systems aim to understand the spatial surroundings and the wearer’s behavior inside it, including motions, activities, and interactions. We argue that egocentric systems must additionally detect physiological states to capture a person’s attention and situational responses, which are critical for context-aware behavior modeling. In this paper, we propose egoPPG, a novel vision task for egocentric systems to recover a person’s cardiac activity to aid downstream vision tasks. We introduce PulseFormer, a method to extract heart rate as a key indicator of physiological state from the eye tracking cameras on unmodified egocentric vision systems. PulseFormer continuously estimates the photoplethysmogram (PPG) from areas around the eyes and fuses motion cues from the headset’s inertial measurement unit to track HR values. We demonstrate egoPPG’s downstream benefit for a key task on EgoExo4D, an existing egocentric dataset for which we find PulseFormer’s estimates of HR to improve proficiency estimation by 14%. To train and validate PulseFormer, we collected a dataset of 13+ hours of eye tracking videos from Project Aria and contact-based PPG signals as well as an electrocardiogram (ECG) for ground-truth HR values. Similar to EgoExo4D, 25 participants performed diverse everyday activities such as office work, cooking, dancing, and exercising, which induced significant natural motion and HR variation (44–164 bpm). Our model robustly estimates HR (MAE=7.67 bpm) and captures patterns (r=0.85). Our results show how egocentric systems may unify environmental and physiological tracking to better understand users and that egoPPG as a complementary task provides meaningful augmentations for existing datasets and tasks. We release our code, dataset, and HR augmentations for EgoExo4D to inspire research on physiology-aware egocentric tasks.

   @article{braun2025egoppg,
     title = {egoPPG: Heart Rate Estimation from Eye-Tracking Cameras in Egocentric Systems to Benefit Downstream Vision Tasks},
     author = {Braun, Bj{\"o}rn and Armani, Rayan and Meier, Manuel and Moebus, Max and Holz, Christian},
     journal = {IEEE/CVF International Conference on Computer Vision (ICCV)},
     year = {2025},
     highlighted = {true}
   }

2024

1. BHI

   

   SympCam: Remote Optical Measurement of Sympathetic Arousal

   Björn Braun, Daniel McDuff, Tadas Baltrusaitis, Paul Streli, Max Moebus, and Christian Holz

   IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), 2024

   Abstract arXiv PDF Dataset Bib

   Recent work has shown that a person’s sympathetic arousal can be estimated from facial videos alone using basic signal processing. This opens up new possibilities in the field of telehealth and stress management, providing a non-invasive method to measure stress only using a regular RGB camera. In this paper, we present SympCam, a new 3D convolutional architecture tailored to the task of remote sympathetic arousal prediction. Our model incorporates a temporal attention module (TAM) to enhance the temporal coherence of our sequential data processing capabilities. The predictions from our method improve accuracy metrics of sympathetic arousal in prior work by 48% to a mean correlation of 0.77. We additionally compare our method with common remote photoplethysmography (rPPG) networks and show that they alone cannot accurately predict sympathetic arousal "out-of-the-box". Furthermore, we show that the sympathetic arousal predicted by our method allows detecting physical stress with a balanced accuracy of 90% - an improvement of 61% compared to the rPPG method commonly used in related work, demonstrating the limitations of using rPPG alone. Finally, we contribute a dataset designed explicitly for the task of remote sympathetic arousal prediction. Our dataset contains synchronized face and hand videos of 20 participants from two cameras synchronized with electrodermal activity (EDA) and photoplethysmography (PPG) measurements. We will make this dataset available to the community and use it to evaluate the methods in this paper. To the best of our knowledge, this is the first dataset available to other researchers designed for remote sympathetic arousal prediction.

   @inproceedings{braun2024sympcam,
     title = {SympCam: Remote Optical Measurement of Sympathetic Arousal},
     author = {Braun, Bj{\"o}rn and McDuff, Daniel and Baltrusaitis, Tadas and Streli, Paul and Moebus, Max and Holz, Christian},
     booktitle = {IEEE EMBS International Conference on Biomedical and Health Informatics},
     year = {2024},
     highlighted = {true}
   }

2. CVPRW

   

   How Suboptimal is Training rPPG Models with Videos and Targets from Different Body Sites?

   Björn Braun, Daniel McDuff, and Christian Holz

   IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024

   Abstract arXiv PDF Dataset Bib

   Remote camera measurement of the blood volume pulse via photoplethysmography (rPPG) is a compelling technology for scalable, low-cost, and accessible assessment of cardiovascular information. Neural networks currently provide the state-of-the-art for this task and supervised training or fine-tuning is an important step in creating these models. However, most current models are trained on facial videos using contact PPG measurements from the fingertip as targets/ labels. One of the reasons for this is that few public datasets to date have incorporated contact PPG measurements from the face. Yet there is copious evidence that the PPG signals at different sites on the body have very different morphological features. Is training a facial video rPPG model using contact measurements from another site on the body suboptimal? Using a recently released unique dataset with synchronized contact PPG and video measurements from both the hand and face, we can provide precise and quantitative answers to this question. We obtain up to 40 % lower mean squared errors between the waveforms of the predicted and the ground truth PPG signals using state-of-the-art neural models when using PPG signals from the forehead compared to using PPG signals from the fingertip. We also show qualitatively that the neural models learn to predict the morphology of the ground truth PPG signal better when trained on the forehead PPG signals. However, while models trained from the forehead PPG produce a more faithful waveform, models trained from a finger PPG do still learn the dominant frequency (i.e., the heart rate) well.

   @inproceedings{braun2024suboptimal,
     title = {How Suboptimal is Training rPPG Models with Videos and Targets from Different Body Sites?},
     author = {Braun, Bj{\"o}rn and McDuff, Daniel and Holz, Christian},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
     year = {2024},
   }

3. JBHI

   

   Nightbeat: Heart Rate Estimation From a Wrist-Worn Accelerometer During Sleep

   Max Moebus, Lars Hauptmann, Nicolas Kopp, Berken Utku Demirel, Björn Braun, and Christian Holz

   IEEE Journal of Biomedical and Health Informatics (JBHI), 2024

   Abstract arXiv PDF Code Dataset Bib

   Today’s fitness bands and smartwatches typically track heart rates (HR) using optical sensors. Large behavioral studies such as the UK Biobank use activity trackers without such optical sensors and thus lack HR data, which could reveal valuable health trends for the wider population. In this paper, we present the first dataset of wrist-worn accelerometer recordings and electrocardiogram references in uncontrolled at-home settings to investigate the recent promise of IMU-only HR estimation via ballistocardiograms. Our recordings are from 42 patients during the night, totaling 310 hours. We also introduce a frequency-based method to extract HR via curve tracing from IMU recordings while rejecting motion artifacts. Using our dataset, we analyze existing baselines and show that our method achieves a mean absolute error of 0.88 bpm – 76% better than previous approaches. Our results validate the potential of IMU-only HR estimation as a key indicator of cardiac activity in existing longitudinal studies to discover novel health insights. Our dataset, Nightbeat-DB, and our source code are available on GitHub.

   @article{moebus2024nightbeat,
     title = {Nightbeat: Heart Rate Estimation From a Wrist-Worn Accelerometer During Sleep},
     author = {Moebus, Max and Hauptmann, Lars and Kopp, Nicolas and Demirel, Berken Utku and Braun, Bj{\"o}rn and Holz, Christian},
     journal = {IEEE Journal of Biomedical and Health Informatics},
     year = {2024},
   }

2023

1. BOE

   

   Video-Based Sympathetic Arousal Assessment via Peripheral Blood Flow Estimation

   Björn Braun, Daniel McDuff, Tadas Baltrusaitis, and Christian Holz

   Biomedical Optics Express (BOE), 2023

   Abstract arXiv PDF Dataset Bib

   Electrodermal activity (EDA) is considered a standard marker of sympathetic activity. However, traditional EDA measurement requires electrodes in steady contact with the skin. Can sympathetic arousal be measured using only an optical sensor, such as an RGB camera? This paper presents a novel approach to infer sympathetic arousal by measuring the peripheral blood flow on the face or hand optically. We contribute a self-recorded dataset of 21 participants, comprising synchronized videos of participants’ faces and palms and gold-standard EDA and photoplethysmography (PPG) signals. Our results show that we can measure peripheral sympathetic responses that closely correlate with the ground truth EDA. We obtain median correlations of 0.57 to 0.63 between our inferred signals and the ground truth EDA using only videos of the participants’ palms or foreheads or PPG signals from the foreheads or fingers. We also show that sympathetic arousal is best inferred from the forehead, finger, or palm.

   @article{braun2023video,
     title = {Video-Based Sympathetic Arousal Assessment via Peripheral Blood Flow Estimation},
     author = {Braun, Bj{\"o}rn and McDuff, Daniel and Baltrusaitis, Tadas and Holz, Christian},
     journal = {Biomedical Optics Express},
     year = {2023},
   }