Abstract

In many literacy studies, it is important to establish the reliability of independent observers' judgments. Reliability most commonly is measured either by the percentage of agreement or the correlation between the observers' judgments. This article argues that the percentage of agreement measure is more difficult to interpret than are correlation measures because of the following: (a) the effects of chance agreement are not accounted for automatically by the percentage of agreement measure; and (b) rates of chance agreement are strongly influenced by the variability of the data, by “ceiling” and “floor” effects, and by the scoring of near agreement as perfect agreement. For these reasons, the authors recommend that the field of literacy research adopt correlation as the standard method for estimating the reliability of observers' judgments.

Journal
Written Communication
Published
1999-07-01
DOI
10.1177/0741088399016003004
CompPile
Search in CompPile ↗
Open Access
Closed
Topics
Export

Citation Context

Cited by in this index (1)

  1. Written Communication

References (33) · 18 in this index

  1. Assessing Writing
  2. 10.2307/747620
  3. Written Communication
  4. 10.2307/748198
  5. Written Communication
Show all 33 →
  1. Written Communication
  2. 10.2307/748030
  3. 10.1598/RRQ.31.1.4
  4. Written Communication
  5. Assessing Writing
  6. Statistical methods in education and psychology
  7. 10.1598/RRQ.31.3.5
  8. 10.2307/747631
  9. Assessing Writing
  10. Written Communication
  11. Written Communication
  12. Written Communication
  13. 10.1207/s15326977ea0101_4
  14. 10.1598/RRQ.31.4.4
  15. 10.1598/RRQ.31.3.4
  16. Research in the Teaching of English
  17. 10.37514/JBW-J.1997.16.1.06
    Journal of Basic Writing  
  18. Research in the Teaching of English
  19. Research in the Teaching of English
  20. 10.1598/RRQ.31.4.5
  21. Research in the Teaching of English
  22. 10.1207/s15326977ea0402_2
  23. 10.2307/747624
  24. Written Communication
  25. 10.1207/s15326977ea0202_1
  26. Research in the Teaching of English
  27. Research in the Teaching of English
  28. Written Communication