The metrics displayed by smartwatches and fitness bands are increasing, from heart rate, blood pressure, sleep, to energy expenditure, fatigue status, and maximum oxygen uptake. However, some values seem inaccurate, right?
Your feeling is correct, the accuracy of various metrics varies greatly. Some are accurate enough for doctors to reference, while others are only suitable for casual observation.
The table below summarizes the accuracy of some common metrics, with a maximum accuracy of 5 stars and a minimum of 1 star.
This table rates the metrics based on various conditions. Different manufacturers may have different algorithms, and the grading involves a degree of subjectivity. A rating of 5 stars indicates very good accuracy (measured against the gold standard), 4 stars indicates good, 3 stars indicates moderate, 2 stars indicates poor, and 1 star indicates very poor.
No metric achieves a rating of 5 stars, meaning no data is absolutely accurate. Why is there inaccuracy? It may relate to measurement methods, sensors, algorithms, wearing and interpretation methods.
Next, we will fill in more detailed content in the table: why it is accurate or not, how accurate it is, and how to achieve more accurate measurements. By the end of this article, you will receive a table with doubled information and how to utilize the judgment capabilities of each metric.
Accuracy depends on whether the metric is measured
or estimated or newly created
Currently, a smartwatch weighing only a few grams can integrate nearly 10 types of sensors, such as photoplethysmographic sensors for measuring heart rate, GPS sensors for measuring latitude and longitude, as well as barometric, temperature, and accelerometer sensors, among others.
Wearable devices (smartwatches, bands, rings, etc.) measure a limited set of basic metrics directly through sensors (raw data also requires algorithm processing, but for ease of understanding, we refer to it as direct measurement); this set of metrics is integrated and calculated, continuously producing an infinite number of new metrics. In other words, as long as there is a basis in physiology and exercise physiology, a few basic metrics can be manipulated to yield a plethora of metrics.
The number of metrics is increasing, but are they all reliable? Measurement inevitably involves errors, but most metrics have a recognized measurement method with the least error, commonly referred to as the “gold standard.” For example, the gold standard for measuring heart rate is an electrocardiogram, for measuring sleep time and stages is polysomnography, and for measuring energy expenditure is the doubly labeled water method.
The gold standard is generally measured under laboratory conditions, and most devices are expensive, with complex measurement steps that require experienced operators to assist. Currently, none of the metrics provided by smartwatches, bands, or rings are measured using the gold standard. Therefore, in the table at the beginning of the article, no metric can achieve a 5-star rating. By sacrificing some data accuracy, wearable devices provide a more convenient and cost-effective measurement method.
The gold standard for measuring heart rate is the electrocardiogram; smartwatches can continuously measure heart rate, which is convenient but slightly less accurate丨medpick/Sina Testing
Among the metrics with gold standards, some data is obtained through direct measurement or simple calculations, such as measuring heart rate with photoplethysmographic sensors or calculating pace based on distance and time.
Other data is estimated based on directly measured data through algorithms, like estimating energy expenditure using heart rate and accelerometer data. Different manufacturers may use different algorithms, and even the same manufacturer may continuously improve their algorithms, leading to potentially significant discrepancies in results.In most cases, estimated data is less accurate than directly measured data.
Moreover, some metrics without gold standards are generally considered inaccurate. These metrics often exist only as concepts in exercise science (e.g., load, fatigue, recovery), which cannot be accurately measured, and sometimes rely on subjective feelings as standards. Some metrics may even lack scientific definitions and are created through a “arms race” among manufacturers.
How big is the gap?
Just compare with the gold standard
If you want to know how accurate a metric is, measure it using both wearable devices and the gold standard, and then compare the results, right?
Actually, most manufacturers do this as well, but they generally do not disclose how big the gap is. However, by analyzing how the data is obtained and looking at published articles by researchers, one can get a rough idea of the accuracy of the data.
1
Heart Rate, One of the Most Accurate Metrics
Heart rate is related to many health and exercise-related metrics. Smartwatches and bands directly display heart rate and can also provide many metrics estimated based on heart rate. Therefore, the accuracy of heart rate measurement determines the accuracy of many other metrics.
The gold standard for measuring heart rate is an electrocardiogram, which detects the heart’s electrical activity and measures heart rate by placing electrodes on the chest and limbs.
When wearable devices continuously display heart rate, the measurement method is usually photoplethysmography (PPG). This measurement method can be affected by various factors, such as exercise intensity, type of exercise, wrist movement, strap tightness, skin pigmentation, surface dirt, arrhythmias, etc.
According to a comprehensive test of 18 studies, heart rate measurement is more accurate at rest or during low-intensity exercise. As exercise intensity increases, the likelihood of obtaining data and the reliability of the data significantly decrease. In an analysis of 249 studies, the average error in heart rate measurement is ±3%.
Therefore, when the wearable device shows stable values at rest, the heart rate data is relatively credible and can help assess health and exercise status. During intense exercise, data accuracy decreases. If you want more accurate data, you can wear a chest heart rate monitor.
Chest heart rate monitor. Image copyright, unauthorized use may lead to copyright disputes
2
Sleep, Total Time Slightly Better Than Stages and Quality
Some people check their sleep metrics first thing in the morning. They may feel they slept well, but upon seeing a low overall score, they feel fatigued, which is often unnecessary.
The gold standard for measuring sleep is polysomnography, which simultaneously measures multiple signals, including EEG, ECG, EOG, and EMG. After obtaining raw data, sleep experts synthesize the results to determine sleep time and manually score to analyze sleep stages.
Polysomnography illustration ｜ verywell
Wearable devices evaluate sleep by measuring heart rate and wrist activity (using accelerometers), calculating metrics like heart rate variability and respiratory rate, and combining background information such as age, height, weight, and gender, based on neural network models to finally determine bed and wake times, sleep onset and offset times, total sleep duration, sleep latency, wake times, and the duration and proportion of each sleep stage, as well as an overall sleep score derived from this information.
Wearable device sleep evaluation method ｜ provided by the author
From the measurement method, if one remains still for a long time before falling asleep, it may be mistakenly judged as entering sleep state, thus overestimating total sleep duration.
Different brands have inconsistent algorithms leading to different errors. A review article on the application of wearable technology in sleep mentioned that compared to polysomnography, smartwatches perform relatively well in estimating total sleep time, with an overall accuracy of about 70% to 90%; however, their performance in measuring sleep stages is poor. The accuracy of identifying light sleep is about 50% to 90%, while deep sleep and REM sleep accuracy is about 30% to 80%.
As for the overall sleep score, there is no corresponding score in medicine. When doctors evaluate sleep quality and provide treatment, they consider many indicators, including sleep onset time, sleep duration, efficiency, abnormal states, sleep medications, and daily life and work conditions.
For estimated metrics like sleep, some of the more accurate ones can serve as references, such as total sleep time, while other metrics should not cause anxiety.If you feel good overall, there is no need to worry about a low total sleep score. If you consistently feel you are not sleeping well, you can undergo polysomnography in a hospital to identify issues.
3
Recovery Status, One of the Least Accurate Metrics
The above metrics all have gold standards, but some metrics lack them and are created based on certain theories, such as recovery status.
To make progress in training, one must continuously increase training stress without crossing the line of overtraining, making the measurement and assessment of recovery status very important.However, recovery status is a very comprehensive and complex metric affected by training (volume, type, intensity, etc.), non-training (work, interpersonal relationships, illness, medication, etc.), and recovery (sleep, diet, recovery time, recovery methods, etc.) factors.
Recovery status is influenced by various factors such as training, sleep, and diet. Image copyright, unauthorized use may lead to copyright disputes
When measuring the stress on the body and recovery status, autonomic nervous system activity is an important indicator. When the body is under stress, it physiologically manifests as increased sympathetic nervous system activity and decreased parasympathetic nervous system activity, and the opposite occurs during recovery. Studies show that when analyzing the interaction between the sympathetic and parasympathetic nervous systems, heart rate variability is a powerful tool.
Since there is no gold standard, some wearable device manufacturers use weighted models to estimate recovery status. The specific method involves collecting a series of metrics that may affect recovery, such as heart rate, sleep, and training status, calculating heart rate variability, respiratory rate, oxygen consumption, etc., and then summing the weighted indicators based on exercise science principles to obtain a value representing recovery status.
The drawback of this approach is that it cannot exhaustively account for all influencing factors, such as physiological cycles and interpersonal relationships, which can affect recovery but may not be included in the model, leading to discrepancies between displayed data and actual status, resulting in undertraining or overtraining when using the data as guidance.
However, manufacturers’ introduction of such metrics makes sense because not everyone has sufficient knowledge to analyze and interpret all the raw data related to recovery. By sacrificing some accuracy and making simple assumptions (e.g., less sleep and more activity equals poor recovery), a recovery status score may be more effective in reminding users than presenting complex physiological data.
How should these metrics be utilized?
According to the classification method mentioned at the beginning of the article, all metrics can be divided into three categories: measured, estimated, and created.
Measured metrics usually have smaller errors, such as heart rate, distance, heart rate variability, and pace.These metrics are relatively reliable and can be used as references for observing health status, adjusting lifestyle, and exercise plans. For example, if your heart rate is higher than usual this morning, did you not sleep well last night? Or have you been overtraining recently? Should you reduce your training or rest for a day?
Estimated metrics are derived from measured metrics through algorithms, such as sleep, energy expenditure, and oxygen uptake. At this point, measurement errors add to algorithm errors, which may reduce the accuracy of estimated metrics.Interpreting these metrics requires more caution. For instance, total sleep scores may sometimes align with fatigue levels, while at other times they may differ; the energy expenditure estimated by wearable devices for walking may be relatively accurate, but the expenditure for resistance (strength) training may be underestimated.
The above two types are metrics with gold standards. Even if current measurements are not very accurate, we can expect advancements in measurement technology or algorithms to bring data closer to accurate values.
Created metrics are those that lack measurement gold standards; they are defined or conceptualized based on the first two types of metrics through algorithms, such as recovery status and training effects. Due to the absence of gold standards for comparison and inconsistencies in sensor hardware and algorithms between different manufacturers, as well as the lack of transparency in metric algorithms, it is challenging to verify the accuracy of the data.
Therefore, for these created metrics, we need not obsess over the absolute values of the numbers; we can understand our body’s responses to daily life and exercise more proactively by observing trends in the metrics and combining them with our subjective feelings.
Additionally, device manufacturers regularly release software updates, so it is important to check and install these updates promptly to ensure the device is using the latest algorithms, which can improve the accuracy of the metrics to some extent.
Finally, this table summarizes the important content of the entire article. Referring to it can help you interpret the data provided by wearable devices, potentially reducing confusion and increasing your control over health and exercise.
References
[1]Altini M, Plews D. What is behind changes in resting heart rate and heart rate variability? A large-scale analysis of longitudinal measurements acquired in free-living[J]. Sensors, 2021, 21(23): 7932.
[2]Cudejko T, Button K, Al-Amri M. Validity and reliability of accelerations and orientations measured using wearable sensors during functional activities[J]. Scientific reports, 2022, 12(1): 14619.
[3]Shei R J, Holder I G, Oumsang A S, et al. Wearable activity trackers–advanced technology or advanced marketing?[J]. European Journal of Applied Physiology, 2022, 122(9): 1975-1990.
[4]Miller D J, Sargent C, Roach G D. A validation of six wearable devices for estimating sleep, heart rate and heart rate variability in healthy adults[J]. Sensors, 2022, 22(16): 6317.
[5]Germini F, Noronha N, Borg Debono V, et al. Accuracy and acceptability of wrist-wearable activity-tracking devices: systematic review of the literature[J]. Journal of medical Internet research, 2022, 24(1): e30791.
[6]Li Y I, Zhong-Hua L V, Shun-Ying H U, et al. Validating the accuracy of a multifunctional smartwatch sphygmomanometer to monitor blood pressure[J]. Journal of Geriatric Cardiology: JGC, 2022, 19(11): 843.
[7]de Zambotti M, Goldstein C, Cook J, et al. State of the science and recommendations for using wearable technology in sleep and circadian research[J]. Sleep, 2023: zsad325.
[8]https://www.firstbeat.com/en/athletes-recovery-analysis-firstbeat-white-paper-2/
[9]https://www.firstbeat.com/en/firstbeat-white-paper-sleep-analysis-method-based-on-heart-rate-variability/
[10]Doherty C, Baldwin M, Keogh A, Caulfield B, Argent R. Keeping Pace with Wearables: A Living Umbrella Review of Systematic Reviews Evaluating the Accuracy of Consumer Wearable Technologies in Health Measurement. Sports Med. 2024 Jul 30. doi: 10.1007/s40279-024-02077-2. Epub ahead of print. PMID: 39080098.
Planning and Production
Source丨Guokr (id: Guokr42)
Author丨ZIYI
Editor丨Yang Yaping
Proofreader丨Xu Lai, Lin Lin
Recommended Articles
1. How Does the Heart Gradually Fail? Full Process Exposed!
2. Vitamin C Content Higher Than Oranges, Low Calories! Missing This Autumn Fruit Will Really Make You Regret
3. This Mysterious Seismic Wave Spread for Nine Days Globally, but Few Noticed the Serious Consequences
4. Did Netizens’ Grandmother’s Jujube Tree Suddenly Go “Crazy”? Is It Contagious? The Truth Is Like This!
5. Carotene Is 29 Times Higher Than Broccoli! This Vegetable Is Antioxidant, Promotes Bowel Movements, Controls Blood Pressure, Missing It Is a Waste
The cover image and images in the article are from copyright libraries
Reproducing may lead to copyright disputes; for original text and images, please reply “reprint” in the background.
Light Up “Looking”
Let’s Gain Knowledge Together!
