A graphical method for assessing agreement with the mean between multiple observers using continuous measures

Jones, Mark, Dobson, Annette and O'Brian, Sue (2011) A graphical method for assessing agreement with the mean between multiple observers using continuous measures. International Journal of Epidemiology, 40 5: 1308-1313.


Author Jones, Mark
Dobson, Annette
O'Brian, Sue
Title A graphical method for assessing agreement with the mean between multiple observers using continuous measures
Journal name International Journal of Epidemiology  (ERA 2012 Listed)    (ERA 2010 Rank A*)   Check publisher's open access policy
Publication date 2011-10
Sub-type Article
DOI 10.1093/ije/dyr109
Volume number 40
Issue number 5
ISSN 0300-5771
1464-3685
Start page 1308
End page 1313
Total pages 6
Place of publication Oxford, United Kingdom
Publisher Oxford University Press
Collection year 2012
Language eng
Formatted abstract Background
Currently, we are not aware of a method to assess graphically on one simple plot agreement between more than two observers making continuous measurements on the same subjects.

Methods
We aimed to develop a simple graphical method to assess agreement between multiple observers using continuous measurements. The Bland–Altman graphical method for assessing agreement between two observers using continuous measures was modified and extended to accommodate multiple observers. Mathematical formulae are derived and real data examples used to illustrate the proposed method.

Results
The examples show that the proposed graphical method of assessing agreement provides clinically useful information. This information includes estimates of the limits of agreement with the mean and a visual means for determining these limits over the range of measurements. In a data example that included five readers’ measurements of 40 lung lesions, the intra-class correlation (ICC) was 0.84 indicating readers can reliably measure the lesions. However, the estimated limits of agreement with the mean were −1.1 to 1.1 cm implying that the readers’ measurements can plausibly differ from the mean estimated tumour size by more than 1 cm. This is a clinically significant difference according to the study authors. In addition, a plot of the limits of agreement with the mean by mean tumour size shows heterogeneous agreement presumably due to the varying degrees of definition at the edge of the lesions.

Conclusions
The proposed graphical method of assessing agreement can be used alongside other measures such as ICC for reporting on reproducibility in studies of multiple observers making continuous measurements.
Keyword Accuracy
Precision
Reliability
Reproducibility
Agreement
Q-Index Code C1
Q-Index Status Confirmed Code
Institutional Status UQ
Additional Notes Article number dyr10 / First published online: 6 July 2011.

Document type: Journal Article
Sub-type: Article
Collections: Official 2012 Collection
School of Population Health Publications
 
Versions
Version Filter Type
Citation counts: TR Web of Science Citation Count  Cited 0 times in Thomson Reuters Web of Science Article
Scopus Citation Count Cited 0 times in Scopus Article
Google Scholar Search Google Scholar
Access Statistics: 48 Abstract Views  -  Detailed Statistics
Created: Tue, 06 Sep 2011, 13:22:47 EST by Geraldine Fitzgerald on behalf of School of Population Health