From heart rate, blood pressure, sleep to energy expenditure, fatigue state, and maximum oxygen uptake, the metrics displayed by smartwatches are increasing, but it seems that some values are not measured accurately?
Your feeling is correct, the accuracy of various metrics varies greatly; some are accurate enough for doctors to reference, while others are only suitable for casual observation.
The table below summarizes the accuracy of some common metrics, with the highest accuracy rated at 5 stars and the lowest at 1 star.
This table rates based on all conditions; different manufacturers may have different algorithms, and the grading has a certain subjectivity. 5 stars indicate very good accuracy (measured against the gold standard), 4 stars indicate good, 3 stars indicate average, 2 stars indicate poor, and 1 star indicates very poor.
None of the metrics achieved 5 stars, meaning no data is absolutely accurate. Why might there be inaccuracies? It could be related to the measurement methods, sensors, algorithms, wearing and interpretation methods.
Next, we will fill in more detailed content in the table: Why they are accurate or not, how accurate they are, and how to measure them more accurately. By the end of this article, you will receive a table with double the information, along with how to utilize the judgment capabilities of each metric.
Accuracy depends on whether the metrics are measured
or estimated or newly created
Currently, a smartwatch weighing a few dozen grams can integrate nearly 10 types of sensors, such as photoplethysmographic sensors for measuring heart rate, GPS sensors for measuring latitude and longitude, as well as barometric, temperature, and accelerometer sensors, etc.
Wearable devices (smartwatches, bands, rings, etc.) measure a limited set of basic metrics directly through sensors (the raw data also needs to be processed by algorithms, but for ease of understanding, we write it as direct measurement);this set of metrics is integrated and calculated, continuously generating an infinite number of new metrics. That is to say, as long as physiology and exercise physiology serve as the foundation, a few basic metrics can be manipulated to yield a plethora of metrics.
As the number of metrics increases, are they all reliable? As long as there is measurement, errors are unavoidable, but most metrics have a widely recognized method of measurement with the least error, generally referred to as the “gold standard.” For example, the gold standard for measuring heart rate is an electrocardiogram, for measuring sleep time and stages it is polysomnography, and for measuring energy expenditure it is the doubly labeled water method.
The gold standard is usually measured under laboratory conditions, most devices are expensive, the measurement steps are complex, and require experienced operators to assist. Currently, none of the metrics provided by smartwatches, bands, or rings are measured using the gold standard. Therefore, in the table at the beginning of the article, none of the metrics can achieve 5 stars. By sacrificing some data accuracy, wearable devices offer more convenient and cost-effective measurement methods.
The gold standard for measuring heart rate is the electrocardiogram; smartwatches can continuously measure heart rate, which is convenient but slightly less accurate | medpick/Sina Testing
Among the metrics that have a gold standard, some data is obtained through direct measurements or simple calculations, such as measuring heart rate through photoplethysmographic sensors or calculating pace through distance and time.
Other data are estimated based on directly measured data through algorithms, like estimating energy expenditure using heart rate and accelerometer data. Different manufacturers may have different algorithms, and the same manufacturer’s algorithms may also be continuously improved, leading to significant differences in results.In most cases, estimated data is less accurate than directly measured data.
Additionally, some metrics without a gold standard can be said to be generally inaccurate. These metrics often only exist in the concepts of sports science (e.g., load, fatigue, recovery), and cannot be accurately measured, sometimes using subjective feelings as the standard. Some metrics may not even have scientific definitions and are created through “arms races” among manufacturers.
How big is the gap?
Just compare it with the gold standard
To know how accurate a metric is, measure it using wearable devices and the gold standard, then compare the results. Isn’t that clear?
Actually, most manufacturers do this, but they generally won’t disclose how big the gap is. However, by analyzing how the data is obtained and looking at the articles published by researchers, one can still get a rough idea of the accuracy of the data.
1
Heart Rate, one of the most accurate metrics
Heart rate is related to many health and exercise-related metrics, and smartwatches can directly display heart rate and provide many metrics estimated based on heart rate. Therefore, the accuracy of heart rate measurement determines the accuracy of many other metrics.
The gold standard for measuring heart rate is the electrocardiogram, which detects the electrical activity of the heart and measures heart rate using electrodes placed on the chest and limbs.
When wearable devices continuously display heart rate, they usually measure it using photoplethysmography (PPG). This measurement method can be affected by various factors, such as exercise intensity, type of exercise, wrist activity, strap tightness, skin pigmentation, surface dirt, arrhythmias, etc.
According to a comprehensive test of 18 studies, heart rate measurement is more accurate at rest or during low-intensity exercise, and as exercise intensity increases, the likelihood of obtaining data and the reliability of the data significantly decrease. In an analysis of 249 comprehensive studies,the average error in heart rate measurement is ±3%.
Therefore, when the wearable device shows stable values at rest, heart rate data is relatively reliable and can be used to help assess health and exercise conditions. During intense exercise, the accuracy of the data decreases; if more accurate data is desired, a chest strap heart rate monitor can be worn.
Chest strap heart rate monitor. Image copyright, reprinting may cause copyright disputes
2
Sleep, total time slightly better than stages and quality
Some people check the sleep metrics of last night as soon as they wake up; they may feel they slept well, but seeing a low overall score can make them feel fatigued, which is actually unnecessary.
The gold standard for measuring sleep is polysomnography, which simultaneously measures multiple signals, including EEG, ECG, EOG, and EMG. After obtaining raw data, sleep experts will synthesize the results to derive sleep time and manually score to analyze sleep stages.
Polysomnography diagram | verywell
Wearable devices evaluate sleep by measuring heart rate and wrist activity (accelerometer), calculating heart rate variability and respiratory rate, and combining personal background information such as age, height, weight, and gender, based on neural network models, ultimately obtaining bedtimes and wake times, sleep start and end times, total sleep duration and sleep latency, wake time, durations, and proportions of various sleep stages, as well as an overall sleep score based on this information.
Wearable devices evaluate sleep methods | Image provided by the author
From the measurement methods, if one remains still for a long time before falling asleep, it may be misjudged as entering a sleep state, overestimating total sleep time.
The specific algorithms of each brand differ, leading to different errors. A review article on the application of wearable technology in sleep mentions that compared to polysomnography, smartwatches perform relatively well in estimating total sleep time, with an overall accuracy of about 70% to 90%; however, their performance in measuring sleep stages is poor, with the accuracy of light sleep judgment around 50% to 90% and deep sleep and REM sleep accuracy around 30% to 80%.
As for the overall sleep score, there is no corresponding score in medicine. When doctors evaluate sleep quality and treatment, they analyze many indicators, including sleep onset time, sleep duration, efficiency, abnormal states, hypnotic medications, daytime life, and work conditions.
For estimated metrics like sleep, some relatively accurate ones can serve as references, such as total sleep time; the remaining metrics should not cause anxiety.If one feels good overall, there is no need to worry about a low total sleep score. If one consistently feels they are not sleeping well, they can go to the hospital for polysomnography to identify issues promptly.
3
Recovery Status, one of the least accurate metrics
The above metrics all have gold standards, while some metrics do not; they are created based on certain theories, such as recovery status.
To achieve progress in training, it is essential to continually increase training stress without crossing the line of overtraining, making the measurement and detection of recovery status very important.However, recovery status is a very comprehensive and complex metric, influenced by training (volume, type, intensity, etc.), non-training (work, relationships, illness, medications, etc.), and recovery (sleep, diet, recovery time, recovery methods, etc.) factors.
Recovery status is influenced by various factors such as training, sleep, diet, etc. Image copyright, reprinting may cause copyright disputes
When measuring the stress experienced by the body and recovery status, autonomic nervous system activity is an important indicator. When the body is under stress, it physiologically usually manifests as increased sympathetic nervous system activity and decreased parasympathetic nervous system activity; during recovery, the opposite occurs. Research shows that when analyzing the interaction between the sympathetic and parasympathetic nervous systems, heart rate variability is a powerful tool.
Due to the lack of a gold standard, some wearable device manufacturers use weighted models to estimate recovery status. The specific method is to collect a series of indicators that may affect recovery, such as heart rate, sleep, and training status, calculate heart rate variability, respiratory rate, oxygen consumption, etc., and then weight and sum different indicators based on exercise science principles, with the resulting value representing recovery status.
The drawback of this approach is that it cannot exhaustively account for all influencing factors, such as physiological cycles and interpersonal relationships can affect recovery, but may not be included in the model calculations, leading to discrepancies between displayed data and actual status, causing undertraining or overtraining when using data for guidance.
However, it makes sense for manufacturers to introduce such metrics, as not everyone has enough knowledge to analyze and interpret all the raw data related to recovery. By sacrificing some accuracy and making some simple assumptions (like less sleep and more activity equals poor recovery), a recovery status score reminder may be much better than presenting complex physiological data.
How should these metrics be utilized?
According to the classification method at the beginning of the article, all metrics can be divided into three categories: measured, estimated, and created.
Measured metrics usually have smaller errors, such as heart rate, distance, heart rate variability, and pace, etc. These metrics are relatively reliable and can be used as references for observing health status, adjusting lifestyle, and exercise plans. For example, if the heart rate is higher than usual when waking up, could it be due to poor sleep last night? Or has there been excessive training recently? Should one reduce the amount or take a day off?
Estimated metrics are derived from measured metrics through algorithms, such as sleep, energy expenditure, and oxygen consumption. At this point, measurement errors compounded with algorithmic errors may reduce the accuracy of estimated metrics. When interpreting these types of metrics, one needs to be more cautious. For instance, overall sleep scores may sometimes align with levels of drowsiness, while at other times there may be discrepancies; energy expenditure estimated by wearable devices for walking may be relatively accurate, but the expenditure for resistance (strength) training may be underestimated.
Both of the above are metrics with gold standards; even if current measurements are not very accurate, we can expect advancements in measurement technology or algorithms to bring data closer to accurate values.
Created metrics are those that do not have a measurement gold standard; they are created based on definitions or ideas from the first two categories of metrics, such as recovery status, training effects, etc. Due to the lack of a measurement gold standard for comparison, combined with inconsistencies in sensor hardware and algorithms among different manufacturers, and the non-disclosure of algorithms for metrics, it is difficult to verify the accuracy of the data.
Therefore, for these created metrics, we need not overly fixate on the absolute values of the numbers, but can understand the trends of the metrics in conjunction with our subjective feelings, to more proactively comprehend how our bodies respond to daily life and exercise.
In addition, device manufacturers regularly release software updates, so it’s important to check and install these updates promptly, ensuring that devices are always using the latest algorithms, which can improve the accuracy of the metrics to some extent.
The final table summarizes the important content of the entire article; referring to it can help you interpret the data provided by wearable devices, perhaps reducing some confusion and increasing your control over health and exercise.
Source: Science Popularization China
——————
New Media Editor: Zhang Yuehong
New Media Review: Qin Jinwen
Director: Zhang Yishui
