The metrics displayed by smartwatches and fitness bands are increasing, ranging from heart rate, blood pressure, sleep, energy expenditure, fatigue status, to maximum oxygen uptake. However, do some values seem inaccurate?
Your feeling is correct,the accuracy of various metrics varies greatly; some are accurate enough for doctors to reference, while others are only suitable for casual observation.
The table below summarizes the accuracy of some common metrics, with the highest accuracy rated as 5 stars and the lowest as 1 star.
This table integrates all conditions for rating; different manufacturers may have different algorithms, and the grading is somewhat subjective. 5 stars indicate very good accuracy (measured against gold standards), 4 stars indicate good, 3 stars indicate moderate, 2 stars indicate poor, and 1 star indicates very poor.
No metric achieved 5 stars here, meaning no data is absolutely accurate. Why might there be inaccuracies? It may relate to the measurement methods, sensors, algorithms, wearing, and interpretation methods.
Next, we will fill in more detailed content in the table:Why is it accurate or not, how accurate is it, and how can we measure more accurately?By the end of this article, you will receive a table with doubled information, along with how to use the judgment capabilities of various metrics.
Accuracy is determined by the measurement of metrics
Whether estimated or newly created
Currently, a smartwatch weighing a few dozen grams can integrate nearly 10 types of sensors, such as optical sensors for measuring heart rate, GPS sensors for measuring latitude and longitude, as well as barometric, temperature, and accelerometer sensors, etc.
Wearable devices (smartwatches, bands, rings, etc.)directly measure a limited set of basic metrics through sensors(raw data also requires algorithm processing; for ease of understanding, this is referred to as direct measurement);this set of metrics is integrated and calculated to continuously produce an infinite number of new metrics.That is, as long as there is physiology and exercise physiology as a foundation, several basic metrics can be cycled to yield many metrics.
The number of metrics is increasing, but are they all reliable? Measurement inevitably involves errors,but most metrics have a commonly accepted method of measurement with the least recognized error, generally referred to as the “gold standard.”For example, the gold standard for measuring heart rate is the electrocardiogram, for measuring sleep time and stages, it is polysomnography, and for measuring energy expenditure, it is the doubly labeled water method.
The gold standard is generally measured under laboratory conditions; most devices are expensive, the measurement steps are complex, and require experienced operators to assist. Currently, none of the metrics provided by smartwatches, bands, or rings are measured using gold standards. Therefore, in the table at the beginning of the article, no metric can achieve 5 stars. By sacrificing some data accuracy, wearable devices offer a more convenient and cost-effective measurement method.
The gold standard for measuring heart rate is the electrocardiogram; smartwatches can continuously measure heart rate, which is convenient but slightly less accurate | medpick/Sina Testing
For metrics that have a gold standard, some data is obtained through direct measurement or simple calculations, such as measuring heart rate with optical sensors or calculating pace based on distance and time.
Other data is estimated based on directly measured data through algorithms, such as estimating energy expenditure based on heart rate and accelerometer data. Different manufacturers may have different algorithms, and the same company’s algorithms may continuously improve, leading to significant differences in results.In most cases, estimated data is less accurate than directly measured data.
Additionally, some metrics without a gold standard can be said to be mostly inaccurate.These metrics often exist only as concepts in exercise science (e.g., load, fatigue, recovery) and cannot be accurately measured; sometimes, subjective feelings are used as standards. Some metrics even lack scientific definitions and are created in a “arms race” between manufacturers.
How big is the gap?
Let’s compare it with the gold standard
To know how accurate a metric is, measure it with wearable devices and the gold standard, and then compare the results. Isn’t that clear?
Actually, most manufacturers do this, but they generally do not disclose how significant the gap is. However, by analyzing how the data is obtained and looking at articles published by researchers, one can roughly understand the accuracy of the data.
1
Heart Rate, One of the Most Accurate Metrics
Heart rate is related to many health and exercise-related metrics; smartwatches and bands display heart rate directly and can provide many metrics estimated based on heart rate. Thus, the accuracy of heart rate measurement determines the accuracy of many other metrics.
The gold standard for measuring heart rate is the electrocardiogram, which detects the heart’s electrical activity and measures heart rate using electrodes placed on the chest and limbs.
When wearable devices continuously display heart rate, the measurement method is typically photoplethysmography(photoplethysmography, PPG).This measurement method can be affected by various factors, such as exercise intensity, type of exercise, wrist activity, tightness of the band, skin pigmentation, surface dirt, arrhythmias, etc.
According to a comprehensive test of 18 studies, heart rate measurement is relatively accurate at rest or during low-intensity exercise; as exercise intensity increases, the likelihood of obtaining data and the reliability of the data significantly decrease. In an analysis of a comprehensive 249 studies,the average error in heart rate measurement is ±3%.
Therefore,when the wearable device displays stable values at rest, the heart rate data is relatively credible and can help assess health and exercise conditions.During vigorous exercise, data accuracy decreases; if more accurate data is desired, a chest heart rate monitor can be worn.
Chest heart rate monitor. Image copyright, reprinting may lead to copyright disputes
2
Sleep, Total Time Slightly Better than Stages and Quality
Some people check last night’s sleep metrics as soon as they wake up; they may feel they slept well, but upon seeing a lower overall score, they feel fatigued unnecessarily.
The gold standard for measuring sleep is polysomnography, which measures multiple signals simultaneously, including electroencephalograms, electrocardiograms, eye movement, and electromyograms. After obtaining the raw data,sleep experts will integrate various results to determine sleep time and manually score to analyze sleep stages.
Polysomnography diagram | verywell
Wearable devices evaluate sleep by measuring heart rate and wrist activity(accelerometer), calculating heart rate variability and respiratory rate, etc., combined with personal background information such as age, height, weight, and gender, based on neural network models, ultimately determining bedtimes and wake-up times, total sleep time and sleep latency, duration and proportion of each sleep stage, and overall sleep scoring based on this information.
Wearable device sleep evaluation method | Image provided by the author
From a measurement method perspective,if one remains motionless for a long time before falling asleep, they may be misjudged as having entered a sleep state,overestimating total sleep time.
The specific algorithms of each brand vary, leading to different errors. A review article on the application of wearable technology in sleep mentioned that compared to polysomnography,smartwatches perform relatively well in estimating total sleep time, with an overall accuracy of about 70% to 90%; their performance in measuring sleep stages is poorer,with accuracy rates for light sleep judgment around 50% to 90%, and deep sleep and REM sleep accuracy around 30% to 80%.
As for overall sleep scoring, there is no corresponding score in medicine. When doctors evaluate sleep quality and treatment, they analyze many indicators, including sleep onset time, sleep duration, efficiency, abnormal states, hypnotics, daytime life, and work status.
For estimated metrics like sleep, relatively accurate metrics such as total sleep time can serve as a reference, while other metrics should not cause anxiety.If the overall state is good, there is no need to worry about a low total sleep score.If one feels they are not sleeping well, they can undergo polysomnography at a hospital to identify issues in a timely manner.
3
Recovery Status, One of the Least Accurate Metrics
The above metrics have gold standards, while some metrics do not; they are created based on certain theories, such as recovery status.
To achieve progress in training, it is essential to continuously increase training stress without crossing the line of overtraining, so measuring and detecting recovery status is very important.However, recovery status is a very comprehensive and complex metricthat is influenced by training (training volume, type, intensity, etc.), non-training (work, interpersonal relationships, illness, medication, etc.), and recovery (sleep, diet, recovery time, recovery methods, etc.) factors.
Recovery status is influenced by various factors such as training, sleep, and diet. Image copyright, reprinting may lead to copyright disputes
When measuring the stress the body endures and recovery status, autonomic nervous system activity is an important metric. When the body is under stress, it physiologically usually manifests as increased sympathetic nervous system activity and decreased parasympathetic nervous system activity; during recovery, the opposite occurs. Research shows that when analyzing the interaction between sympathetic and parasympathetic nervous system, heart rate variability is a powerful tool.
Due to the lack of a gold standard,some wearable device manufacturers use weighted models to estimate recovery status.The specific method is to collect a series of metrics that may affect recovery, such as heart rate, sleep, and training status, calculate heart rate variability, respiratory rate, oxygen consumption, etc., and then sum the weighted values based on exercise science principles to obtain a value representing recovery status.
The downside of this approach is that it cannot exhaustively account for all influencing factors;for example, physiological cycles and interpersonal relationships can affect recovery but may not be included in the model, leading to discrepancies between displayed data and actual states, resulting in undertraining or overtraining when using data for guidance.
However, it makes sense for manufacturers to introduce such metrics, as not everyone has sufficient knowledge to analyze and interpret the raw data related to recovery. By sacrificing some accuracy and making some simple assumptions (e.g., less sleep and more activity equals poor recovery),a recovery status score reminder may be much more effective than presenting complex physiological data.
How should these metrics be utilized?
According to the classification method at the beginning of the article, all metrics can be divided into three categories: measurement, estimation, and creation.
Measured metrics typically have smaller errors, such as heart rate, distance, heart rate variability, and pace.These metrics are relatively credible and can be used as references for observing health status, adjusting lifestyles, and exercise plans.For example, if your heart rate is higher than usual when you wake up this morning, could it be that you didn’t sleep well last night? Or have you been overtraining recently? Should you reduce your training or take a day off?
Estimated metrics are derived from measured metrics through algorithms, such as sleep, energy expenditure, and oxygen consumption. At this point, measurement errors combined with algorithm errors may reduce the accuracy of estimated metrics.Interpreting this type of metric requires greater caution.For instance, overall sleep scores may sometimes align with levels of drowsiness and at other times differ; the energy expenditure estimated by wearable devices during walking may be relatively accurate, but the expenditure during resistance (strength) training may be underestimated.
The above two types are metrics with gold standards; even if current measurements are not very accurate, we can expect advancements in measurement technology or algorithms to bring data closer to accurate values.
Created metrics are those that lack measurement gold standards; they are developed based on the first two types, where algorithms create metrics based on certain definitions or ideas, such as recovery status and training effects. Due to the absence of measurement gold standards for comparison, coupled with inconsistencies in sensor hardware and algorithms between different manufacturers, and the lack of transparency regarding the algorithms for these metrics, it is challenging to verify the accuracy of the data.
Therefore, for these created metrics, we need not overly fixate on the absolute values of the numbers;we can understand the trends in the metrics, combine them with our subjective feelings, and actively comprehend how our bodies respond to daily life and exercise.
Additionally, device manufacturers regularly release software updates,timely checking and installing these updatesensures that the device is always using the latest algorithms, which can improve the accuracy of the metrics to some extent.
This final table summarizes the important content of the entire article; referring to it can help you interpret the data provided by wearable devices, potentially reducing confusion and increasing your control over health and exercise.
