Intraclass Correlation Coefficient (ICC) is commonly used to evaluate the degree of similarity of a certain quantitative attribute among individuals with established kinship (such as twins, siblings, etc.). It is also applied to assess the reproducibility or consistency of different measurement methods or raters on the same quantitative measurement results. In diagnostic tests, we often use the ICC indicator to evaluate the reproducibility of different researchers diagnosing the same set of test results.
1. Problem and Data
Suppose there are 2 researchers who measure the blood glucose levels of 25 subjects using the same diagnostic test. Part of the raw data is shown in Table 1.
Table 1: Part of the Raw Data
Of course, to evaluate the reproducibility of this diagnostic test, we can set a diagnostic cutoff point, artificially converting blood glucose levels into a binary variable, and then use the Kappa analysis discussed earlier for judgment. However, converting continuous variables into binary variables results in a loss of information. So what should we do?
Next, we will introduce the analysis method of the intraclass correlation coefficient for diagnostic tests, using SPSS statistical software as an example.
2. SPSS Analysis Method
1. Data Entry into SPSS
2. Select Analyze → Scale → Reliability Analysis
3. Option Settings
(1) Main Dialog Box Settings
We will place the two groups of data to be observed into the Items box.
(2) Statistics Settings
Click on Statistics and select Intraclass correlation coefficient.
Model Settings:
There are three models for calculating the intraclass correlation coefficient: One-way random, Two-way random, and Two-way mixed. The One-way random model is used to test whether the means of each subject are completely equal and should not be used to evaluate the reproducibility of the diagnostic test. The Two-way random model and the Two-way mixed model are similar; they consider the effects of both subjects and researchers and can theoretically be used to evaluate the reproducibility of diagnostic tests. However, the inference scope of the results from these two models differs. The results of the Two-way random model can be generalized to all similar, possible researchers, while the results of the Two-way mixed model are limited to a given researcher and cannot be inferred to others.
Therefore, the evaluation of the reproducibility of diagnostic tests should choose the Two-way random model.
The Two-way random model has two calculation types: absolute agreement and consistency. Absolute agreement considers the systematic error of the researchers and can be used to measure whether different researchers give the same absolute values to the subjects. Consistency does not consider the systematic error of the researchers and is only suitable for evaluating whether the scores between different researchers are highly correlated.
Specifically, if we have two sets of results: 2, 4, 6 and 4, 6, 8, the absolute values are not equal, and calculating the ICC value using absolute agreement gives only 0.67 (only the absolute values of 4 and 6 are the same); however, these two sets of results are highly correlated, and using consistency gives an ICC value of 1 (these two sets of numbers are highly correlated).
For the evaluation of the reproducibility of diagnostic tests, we want different researchers to obtain consistent results, not just “highly correlated” results.Therefore, we should choose the absolute agreement calculation type.
→ Continue→ Return to Main Interface→ OK
3. Result Interpretation
The calculation results of the SPSS intraclass correlation coefficient have three tables, but we only need to focus on one, as follows:
This table provides two estimated results of the intraclass correlation coefficient: Single Measures and Average Measures. What is the difference between these two estimated results? The analysis unit of Single Measures is the result of each researcher, which can estimate the situation of a single researcher. The analysis unit of Average Measures is the mean of the results from multiple researchers, which has limited applicability.
Therefore,we base our judgment of the intraclass correlation coefficient for the reproducibility evaluation of the diagnostic test on the estimated result of Single Measures, which is ICC=0.987 (P<0.001).
4. Writing Conclusions
Generally, the ICC value ranges from 0 to 1. For diagnostic tests, if the ICC value is less than 0.4, we consider the reproducibility of the diagnostic test to be poor; if the ICC value is greater than 0.75, then the reproducibility of the diagnostic test is considered good.
In summary, the ICC value for this blood glucose level diagnostic test is 0.987 (P<0.001), indicating good reproducibility.
Recommended Reading
1. Mcgraw K O, Wong S P. Forming inferences about some intraclass correlation coefficients.[J]. Psychological Methods, 1996, 1(4):390-390.
2. Shrout P E, Fleiss J L. Intraclass correlations: uses in assessing rater reliability.[J]. Psychological Bulletin, 1979, 86(2):420.
Recommended course in Medical Coffee Club: How to Understand the Quality of Evidence and Recommendation Strength in Clinical Practice Guidelines!
This course explains how to search for evidence, how to conduct systematic reviews, how to use the GRADE system to grade the quality of evidence and the strength of recommendations, and the processes and key points of consensus method application.
Use your computer to open Medical Coffee Club and watch the course video:
https://www.mediecogroup.com/zhuanlan/courses/38/
Follow Medical Coffee Club to get the latest statistical tutorials in a timely manner.
Click the lower left corner “Read the Original” to view all free statistical tutorials. Or use your computer to open the website:http://www.mediecogroup.com/, and browse all statistical Tutorials.
Add the personal WeChat of Xiaoka (xys2019ykh), and join the statistical discussion group to exchange and learn with many research enthusiasts.