Performance Evaluation of the Circadia Contactless Respiratory Monitor and Circadia Sleep Analysis Algorithm
Abstract—The current study aims to evaluate the performance of the Circadia Sleep Analysis Algorithm, which - in combination with the non-contact and minimally invasive Circadia bedside monitor - allows for long-term sleep assessment and continuous evaluation of sleep therapy outcomes. The Circadia Contactless Respiratory Monitor was initially used to record 17 nights of sleep data from 9 participants alongside polysomnography (PSG), with a subsequent 24 nights of PSG data for validation purposes. Vital sign and body movement features were extracted from sensor data, and a machine learning algorithm was developed to perform sleep stage prediction. The algorithm was trained using PSG data, and validated by leave-one-subject-out cross validation. An epoch-by-epoch recall (true positive rate) of 75.0%, 59.9%, 74.8% and 57.1%, was found for Deep, Light, REM and Wake respectively, in the initial 17 night dataset. Highly similar results were obtained in the independent validation dataset of 24 nights, indicating robustness of results and generalisability of the sleep staging algorithm. The Circadia device and Sleep Analysis Algorithm were found to outperform both consumer and medical grade wrist-worn actimetry devices (Fitbit Alta HR and Philips Respironics Actiwatch) on sleep metric estimation accuracy. In a comparison of published sleep staging performance against alternative non-contact consumer sleep tracking devices, the Circadia device achieves highest overall performance.
These results show that contactless sleep tracking using the developed monitor is highly accurate, and PSG accuracy is being approached (considering a PSG inter-rater-agreement rate of 82%). This suggests that the developed non-contact monitor forms a viable alternative to existing clinically used wrist-worn methods, and that longitudinal monitoring of sleep stages in a home environment becomes feasible.
Sleep tracking, both through wrist-worn actigraphy devices and polysomnography (PSG), is extensively used in sleep medicine to diagnose sleep disorders and to evaluate treatment outcomes. Actigraphy data are used for clinical evaluation of insomnia, circadian rhythm disorders and to exclude insufficient sleep as a possible cause for complaints. PSG data are often used to diagnose or exclude sleepdisordered breathing and periodic limb movement disorder. In addition, the portion of the night spent in different sleep stages and their timing are of clinical significance: adequate non-rapid eye movement (NREM) sleep is considered essential for general restoration, to avoid hypertension, diabetes, weight gain, cognitive problems, and even early death. The rapid-eyemovement (REM) sleep stage is important for memory, and its timing is a marker for depression, circadian phase, and neurological disorders such as narcolepsy. Finally, sleep stages and their timing are being used for non-invasive estimation of circadian rhythm. However, PSG studies are inconvenient for patients and very costly. Circadia has developed a non-contact and non-invasive bedside respiratory monitor. This study evaluates the aleep analysis performance of the Circadia device in combination with the Circadia Sleep Analysis Algorithm against the current gold standard PSG, and compares obtained performance results against alternative consumer sleep trackers.
The Circadia contactless monitoring functionality relies on accurate sensing of both vital signs and body motion through radar technology. A coherent pulsed ultra-wideband radaron-chip is used for precise measurements of user range and motion, for distances up to 3 meters. Both respiration and the heart beat cause a slight (sub-millimeter) displacement of the chest wall [1], which is easily detected by the Circadia device. Respiration signals, which include respiratory rate, respiratory rate variability and respiration patterns, as well as body motion, vary in different sleep stages. These signals can therefore be used to predict sleep stages through machine learning algorithms. By distinguishing deep, light, and REM sleep, sleeping patterns can be tracked and analysed. The specific radar architecture used ensures that, in addition to motion and respiration signals, the range from a user to the device can be determined from the received data with high accuracy. This is of importance for two-person-tracking [2] in case of a bed partner, for filtering out noise sources (such as nurses passing by in a clinical setting), or to ascertain when the user is in bed. User position, together with analysis of the sleep environment through light and sound sensing, allows for accurate determination of time spent in bed. Especially for users with insomnia, time-in-bed and sleep efficiency (the percentage of time in bed spent asleep) are important clinical outcomes, which are difficult to monitor using a wearable device [3].
The Circadia Contactless Respiratory Monitor was initially used to record 17 nights of sleep data from nine healthy participants (three females; mean age 25.3, SD 1.73) over consecutive nights of testing. Alongside the Circadia device, PSG data were recorded (Somte PSG, Compumedics). PSG data ´ were scored by a RPSGT-certified sleep technician, according to standards set forth by the American Academy of Sleep Medicine (AASM): each 30-second epoch was scored as either Wake, N1, N2, N3, or REM. Stages N1 and N2 were grouped together into Light sleep. N3 was relabelled as Deep sleep for the purposes of this study. For further validation purposes, participants were asked to wear a Fitbit Alta HR watch and a clinical grade wrist-worn actimetry device (Philips Respironics Actiwatch Spectrum Plus).
Respiration signals and body movement features were obtained from raw Circadia data using proprietary digital signal processing algorithms, and processed by the Circadia Sleep Analysis Algorithm. The algorithm consists of an ensemble of machine learning algorithm which was trained on PSG data. One of four sleep stages (Wake, REM, Light, Deep) was predicted for each 30 s epoch of sensor data. The accuracy of sleep stage predictions was assessed through epoch-by-epoch comparison with PSG data. Precision and recall rates were used to evaluate the Circadia model performance. In addition, hypnogram (plot of sleep stages throughout the night) concordance between PSG and Circadia prediction was evaluated by visual inspection (not quantified). To evaluate the performance of sleep metric estimation (such as total sleep time, sleep efficiency and sleep latency, all derived according to AASM standard) across all recordings, mean absolute percentage error (MAPE) was used. The results are given as 100-MAPE%.
Despite the use of cross validation, a risk of model overfitting exists when using machine learning techniques for sleep stage prediction. To truly validate performance, the Circadia system was independently tested by a sleep and memory research group based at a European university. Sleep data were collected from 24 participants (healthy sleepers, 18 females, mean age 23.1, SD 3.42) in a sleep lab using PSG and the Circadia device. None of the participants from the validation study had previously participated in the initial data collection study. PSG data were scored by sleep technicians according to standards set forth by the AASM. Sleep technicians from the validation study were different from those who scored the data from the initial training set. Obtained datasets were only used for performance evaluation, not for training of the Circadia Sleep Analysis Algorithm. The validation dataset was thus completely independent from the training dataset. After obtaining the validation dataset, Circadia algorithms were run to predict sleep stages, and sleep staging performance was assessed through epoch-by-epoch comparison with PSG data.