From heart rate, blood pressure, sleep to energy consumption, fatigue status, and maximum oxygen uptake, the metrics displayed by smartwatches and fitness bands are increasing, but do some values seem inaccurate?
Your feeling is correct,the accuracy of various metrics varies greatly; some are accurate enough for doctors to use as reference, while others are only suitable for casual observation.
The table below summarizes the accuracy of some common metrics, with the highest accuracy rated as 5 stars and the lowest as 1 star.
This table rates based on all conditions; different manufacturers may have different algorithms, and the grading has a certain subjectivity. 5 stars indicate very good accuracy (measured by gold standard), 4 stars indicate good, 3 stars indicate average, 2 stars indicate poor, and 1 star indicates very poor.
No metric achieves 5 stars, meaning no data is absolutely accurate. Why is there inaccuracy? It may relate to measurement methods, sensors, algorithms, wearing and interpretation methods.
Next, we will fill in more detailed content in the table:Why is it accurate or not, how accurate is it, and how to achieve more accurate measurements.By the end of the article, you will obtain a table with doubled information and how to use the judgment ability of various metrics.
Accuracy depends on whether the metric is measured
or estimated or newly created
Now, a smartwatch weighing several dozen grams can integrate nearly 10 types of sensors, such as optical sensors for measuring heart rate, GPS sensors for measuring latitude and longitude, as well as barometric, temperature, and accelerometer sensors, etc.
Wearable devices (smartwatches, fitness bands, rings, etc.)measure directly through sensors to display a limited set of basic metrics(the raw data also needs algorithm processing, but for ease of understanding, it is written as direct measurement);this batch of metrics is integrated and calculated, continuously producing an infinite number of new metrics.That is, as long as there is physiology and exercise physiology as a foundation, several basic metrics can be manipulated to derive a plethora of metrics.
The number of metrics is increasing, but are they all reliable? As long as there is measurement, errors are inevitable,but most metrics have a commonly accepted method with the least measurement error, generally referred to as the “gold standard.”For example, the gold standard for measuring heart rate is an electrocardiogram, for measuring sleep time and stages is polysomnography, and for measuring energy expenditure is the doubly labeled water method.
The gold standard is usually measured under laboratory conditions, most devices are expensive, measurement steps are complex, and require experienced operators to assist. Currently, none of the metrics provided by smartwatches, fitness bands, or rings are measured by the gold standard. Therefore, in the table at the beginning of the article, no metric can achieve 5 stars. By sacrificing some data accuracy, wearable devices offer a more convenient and cost-effective measurement method.
The gold standard for measuring heart rate is an electrocardiogram; smartwatches can continuously measure heart rate, which is convenient but slightly less accurate丨medpick/Sina Testing
Among the metrics with gold standards, some data is obtained directly through measurement or through simple calculations, such as measuring heart rate through optical sensors or calculating pace through distance and time.
Some data is estimated based on directly measured data through algorithms, like estimating energy expenditure from heart rate and accelerometer data. Different manufacturers may have different algorithms, and even the same manufacturer’s algorithms may constantly improve, leading to significant result discrepancies.In most cases, estimated data is less accurate than directly measured data.
Additionally, some metrics without gold standards are generally less accurate.These metrics often only exist in the concepts of exercise science (e.g., load, fatigue, recovery) and cannot be accurately measured; sometimes subjective feelings are used as standards. Some metrics even lack scientific definitions and are created through “arms races” between manufacturers.
How big is the gap?
Just compare with the gold standard
If you want to know how accurate the metrics are, measure with both wearable devices and the gold standard, and compare the results to find out.
In fact, most manufacturers do this, but they usually do not disclose how big the gap is. However, by analyzing how the data is obtained and looking at the articles published by researchers, one can still roughly understand the accuracy of the data.
1
Heart Rate, One of the Most Accurate Metrics
Heart rate is related to many health and exercise-related metrics; smartwatches and fitness bands will directly display heart rate and can also provide many metrics estimated based on heart rate. Thus, the accuracy of heart rate measurement determines the accuracy of many other metrics.
The gold standard for measuring heart rate is an electrocardiogram, which detects the electrical activity of the heart and measures heart rate through electrodes placed on the chest and limbs.
When wearable devices continuously display heart rate, the measurement method is usually photoplethysmography(PPG).This measurement method can be affected by various factors, such as exercise intensity, type of exercise, wrist movement, strap tightness, skin pigmentation and surface dirt, arrhythmias, etc.
According to a comprehensive test of 18 studies, heart rate measurement is more accurate at rest or during low-intensity exercise; as exercise intensity increases, the probability of obtaining data and the reliability of the data significantly decrease. In an analysis of 249 studies,the average error in heart rate measurement is ±3%.
Therefore,when wearable devices display stable values at rest, heart rate data is relatively credible and can help assess health and exercise conditions.During intense exercise, data accuracy decreases; if you want more accurate data, you can wear a chest heart rate monitor.
Chest heart rate monitor.Image copyright, reprinting may cause copyright disputes.
2
Sleep, Total Time Slightly Better than Stages and Quality
Some people check last night’s sleep metrics as soon as they wake up; feeling like they slept well, they see a low overall score and suddenly feel fatigued, which is unnecessary.
The gold standard for measuring sleep is polysomnography, which simultaneously measures multiple signals, including EEG, ECG, EOG, and EMG. After obtaining the raw data,sleep experts will integrate the results to derive sleep duration and manually score to analyze sleep stages.
Polysomnography schematic ｜ verywell
Wearable devices evaluate sleep by measuring heart rate and wrist activity(accelerometer), calculating heart rate variability and respiratory rate, and combining personal background information such as age, height, weight, and gender, based on a neural network model, ultimately obtaining bed and wake-up times, sleep onset and offset times, total sleep duration and sleep latency, awake duration, durations and proportions of various sleep stages, and overall sleep score derived from this information.
Wearable device sleep evaluation method ｜ Image provided by the author
From the measurement method perspective,if one remains still for a long time before falling asleep, it may be misjudged as entering sleep state, overestimating total sleep duration.
The specific algorithms of different brands lead to different errors. A review article on the application of wearable technology in sleep mentioned that compared to polysomnography,smartwatches perform relatively well in estimating total sleep time, with overall accuracy around 70% to 90%; however, their performance in measuring sleep stages is poor, with light sleep judgment accuracy around 50% to 90%, and deep sleep and REM sleep accuracy around 30% to 80%.
As for overall sleep scores, there is no corresponding score in medicine. When doctors evaluate sleep quality and treatment, they will analyze many indicators, including sleep onset time, sleep duration, efficiency, abnormal states, hypnotic drugs, and daytime life and work conditions.
For estimated metrics like sleep, those that are relatively accurate can be used as references, such as total sleep time; other metrics should not cause anxiety.If one feels good overall, there is no need to worry about a low total sleep score;if one always feels that they are not sleeping well, they can go to the hospital for polysomnography to identify issues.
3
Recovery Status, One of the Least Accurate Metrics
The above metrics all have gold standards, while some metrics do not; they are created based on certain theories, such as recovery status.
To make progress in training, one must continuously increase training stress without crossing the line of overtraining, so measuring and detecting recovery status is crucial.However, recovery status is a very comprehensive and complex metricthat is influenced by training (training volume, type, intensity, etc.), non-training (work, relationships, illness, medication, etc.), and recovery (sleep, diet, recovery time, recovery methods, etc.) factors.
Recovery status is influenced by various factors including training, sleep, and diet.Image copyright, reprinting may cause copyright disputes.
When measuring the stress the body undergoes and recovery status, autonomic nervous system activity is a crucial indicator. When the body is under stress, it physiologically manifests as increased sympathetic nervous system activity and decreased parasympathetic nervous system activity; during recovery, the opposite occurs. Studies show that when analyzing the interaction between sympathetic and parasympathetic nervous systems, heart rate variability is a powerful tool.
Due to the lack of a gold standard,some wearable device manufacturers use weighted models to estimate recovery status.The specific method involves collecting a series of indicators that may affect recovery, such as heart rate, sleep, and training status, calculating heart rate variability, respiratory rate, oxygen consumption, etc., and then summing these values based on sports science principles to represent recovery status.
The downside of this approach is that it cannot exhaust all influencing factors,such as physiological cycles and relationships can affect recovery but may not be included in the model, leading to discrepancies between displayed data and actual status, resulting in undertraining or overtraining when using data as guidance.
However, it makes sense for manufacturers to introduce such metrics, as not everyone has enough knowledge to analyze and interpret the original data related to recovery. By sacrificing some accuracy and making simple assumptions (e.g., less sleep and more activity equals poor recovery),a recovery status score reminder may be much better than presenting complex physiological data.
How should these metrics be utilized?
According to the classification method at the beginning of the article, all metrics can be divided into three categories: measured, estimated, and created.
Measured metrics usually have smaller errors, such as heart rate, distance, heart rate variability, and pace.These metrics are relatively credible and can be used as references for observing health status, adjusting lifestyle, and exercise plans.For example, if the heart rate is higher than usual this morning, was it because of poor sleep last night? Or has there been excessive training lately? Should one reduce the training load or take a rest day?
Estimated metrics are derived from measured metrics based on algorithms, such as sleep, energy expenditure, and oxygen uptake. At this point, measurement errors add to algorithm errors, which may reduce the accuracy of estimated metrics.Interpreting such metrics requires greater caution.For instance, overall sleep scores may sometimes align with drowsiness levels and sometimes differ; energy expenditure estimated by wearable devices for walking may be relatively accurate, but resistance (strength) training expenditure may be underestimated.
The above two are metrics with gold standards, and even if current measurements are not very accurate, we can expect advancements in measurement technology or algorithms to bring data closer to accurate values.
Created metrics are those without measurement gold standards; they are generated based on definitions or ideas derived from the first two categories of metrics, such as recovery status and training effects. Due to the lack of a measurement gold standard for comparison, coupled with inconsistencies in sensors and algorithms between different manufacturers, and the lack of transparency in metric algorithms, it is difficult to verify the accuracy of the data.
Therefore, for these created metrics, we should not overly fixate on the absolute values of numbers,but rather understand the trends of the metrics and combine them with subjective feelings to actively comprehend how the body responds to daily life and exercise.
Additionally, device manufacturers regularly release software updates,so timely checking and installing these updatescan ensure that devices are always using the latest algorithms, which can improve the accuracy of metrics to some extent.
The final table summarizes the important content of the entire article; referring to it to interpret the data provided by wearable devices may help reduce confusion and increase control over health and exercise.
References
[1]Altini M, Plews D. What is behind changes in resting heart rate and heart rate variability? A large-scale analysis of longitudinal measurements acquired in free-living[J]. Sensors, 2021, 21(23): 7932.
[2]Cudejko T, Button K, Al-Amri M. Validity and reliability of accelerations and orientations measured using wearable sensors during functional activities[J]. Scientific reports, 2022, 12(1): 14619.
[3]Shei R J, Holder I G, Oumsang A S, et al. Wearable activity trackers–advanced technology or advanced marketing?[J]. European Journal of Applied Physiology, 2022, 122(9): 1975-1990.
[4]Miller D J, Sargent C, Roach G D. A validation of six wearable devices for estimating sleep, heart rate and heart rate variability in healthy adults[J]. Sensors, 2022, 22(16): 6317.
[5]Germini F, Noronha N, Borg Debono V, et al. Accuracy and acceptability of wrist-wearable activity-tracking devices: systematic review of the literature[J]. Journal of medical Internet research, 2022, 24(1): e30791.
[6]Li Y I, Zhong-Hua L V, Shun-Ying H U, et al. Validating the accuracy of a multifunctional smartwatch sphygmomanometer to monitor blood pressure[J]. Journal of Geriatric Cardiology: JGC, 2022, 19(11): 843.
[7]de Zambotti M, Goldstein C, Cook J, et al. State of the science and recommendations for using wearable technology in sleep and circadian research[J]. Sleep, 2023: zsad325.
[8]https://www.firstbeat.com/en/athletes-recovery-analysis-firstbeat-white-paper-2/
[9]https://www.firstbeat.com/en/firstbeat-white-paper-sleep-analysis-method-based-on-heart-rate-variability/
[10]Doherty C, Baldwin M, Keogh A, Caulfield B, Argent R. Keeping Pace with Wearables: A Living Umbrella Review of Systematic Reviews Evaluating the Accuracy of Consumer Wearable Technologies in Health Measurement. Sports Med. 2024 Jul 30. doi: 10.1007/s40279-024-02077-2. Epub ahead of print. PMID: 39080098.
Source: Science Popularization China
