The Work-ability Support Scale: Evaluation of Scoring Accuracy and Rater Reliability

Turner-Stokes, Lynne, Fadyl, Joanna, Rose, Hilary, Williams, Heather, Schluter, Philip J. and McPherson, Kathryn (2013) The Work-ability Support Scale: Evaluation of Scoring Accuracy and Rater Reliability. Journal of Occupational Rehabilitation, 24 3: 511-524. doi:10.1007/s10926-013-9486-1

Author Turner-Stokes, Lynne
Fadyl, Joanna
Rose, Hilary
Williams, Heather
Schluter, Philip J.
McPherson, Kathryn
Title The Work-ability Support Scale: Evaluation of Scoring Accuracy and Rater Reliability
Journal name Journal of Occupational Rehabilitation   Check publisher's open access policy
ISSN 1053-0487
Publication date 2013-12-12
Year available 2013
Sub-type Article (original research)
DOI 10.1007/s10926-013-9486-1
Open Access Status Not Open Access
Volume 24
Issue 3
Start page 511
End page 524
Total pages 14
Place of publication New York, NY United States
Publisher Springer New York LLC
Language eng
Formatted abstract
Purpose The Work-ability Support Scale (WSS) is a new tool designed to assess vocational ability and support needs following onset of acquired disability, to assist decision-making in vocational rehabilitation. In this article, we report an iterative process of development through evaluation of inter- and intra-rater reliability and scoring accuracy, using vignettes. The impact of different methodological approaches to analysis of reliability is highlighted.

Methods Following preliminary evaluation using case-histories, six occupational therapists scored vignettes, first individually and then together in two teams. Scoring was repeated blind after 1 month. Scoring accuracy was tested against agreed ‘reference standard’ vignette scores using intraclass correlation coefficients (ICCs) for total scores and linear-weighted kappas (kw) for individual items. Item-by-item inter- and intra-rater reliability was evaluated for both individual and team scores, using two different statistical methods.

ICCs for scoring accuracy ranged from 0.95 (95 % CI 0.78–0.98) to 0.96 (0.89–0.99) for Part A, and from 0.78 (95 % CI 0.67–0.85) to 0.84 (0.69–0.92) for Part B. Item by item analysis of scoring accuracy, inter- and intra-rater reliability all showed ‘substantial’ to ‘almost perfect’ agreement (kw ≥ 0.60) for all Part-A and 8/12 Part-B items, although multi-rater kappa (Fleiss) produced more conservative results (mK = 0.34–0.79). Team rating produced marginal improvements for Part-A but not Part-B. Four problematic contextual items were identified, leading to adjustment of the scoring manual.

Conclusion This vignette-based study demonstrates generally acceptable levels of scoring accuracy and reliability for the WSS. Further testing in real-life situations is now warranted.
Keyword Rehabilitation
Needs assessment
Reliability and validity
Q-Index Code C1
Q-Index Status Confirmed Code
Institutional Status UQ

Document type: Journal Article
Sub-type: Article (original research)
Collections: Official 2014 Collection
School of Nursing, Midwifery and Social Work Publications
Version Filter Type
Citation counts: TR Web of Science Citation Count  Cited 1 times in Thomson Reuters Web of Science Article | Citations
Scopus Citation Count Cited 0 times in Scopus Article
Google Scholar Search Google Scholar
Created: Thu, 09 Jan 2014, 01:06:06 EST by Vicki Percival on behalf of School of Nursing, Midwifery and Social Work