Written Communication Jul 1999

Issues in Measuring Reliability: Correlation Versus Percentage of Agreement

JOHN R. HAYES ; JILL A. HATCH

Abstract

In many literacy studies, it is important to establish the reliability of independent observers' judgments. Reliability most commonly is measured either by the percentage of agreement or the correlation between the observers' judgments. This article argues that the percentage of agreement measure is more difficult to interpret than are correlation measures because of the following: (a) the effects of chance agreement are not accounted for automatically by the percentage of agreement measure; and (b) rates of chance agreement are strongly influenced by the variability of the data, by “ceiling” and “floor” effects, and by the scoring of near agreement as perfect agreement. For these reasons, the authors recommend that the field of literacy research adopt correlation as the standard method for estimating the reliability of observers' judgments.

Journal
Written Communication
Published
1999-07-01
DOI
10.1177/0741088399016003004
CompPile
Search in CompPile ↗
Open Access
Closed
Topics
literacy studies
Export
BibTeX RIS

Citation Context

Cited by in this index (1)

  1. JOHNSON et al. (2001)
    Score Resolution and the Interrater Reliability of Holistic Scores in Rating Essays
    Written Communication

References (33) · 18 in this index

  1. Allen (1995)
    Valuing differences: Portnet's first year
    Assessing Writing
  2. 10.2307/747620
     ↗
  3. BOICE (1997)
    Which is more Productive, Writing in Binge Patterns of Creative Illness or in M…
    Written Communication
  4. 10.2307/748198
     ↗
  5. CHAPMAN (1994)
    The Emergence of Genres: Some Findings from an Examination of First-Grade Writing
    Written Communication
Show all 33 →
  1. CRAMMOND (1998)
    The Uses and Complexity of Argument Structures in Expert and Student Persuasive…
    Written Communication
  2. 10.2307/748030
     ↗
  3. 10.1598/RRQ.31.1.4
     ↗
  4. FABER (1996)
    Rhetoric in Competition: The Formation of Organizational Discourse in Conferenc…
    Written Communication
  5. Gearhart et al. (1995)
    Toward the instructional utility of large-scale writing assessment: Validation …
    Assessing Writing
  6. Glass, G. V.
    Statistical methods in education and psychology
  7. 10.1598/RRQ.31.3.5
     ↗
  8. 10.2307/747631
     ↗
  9. Haswell et al. (1996)
    Gender bias and critique of student writing
    Assessing Writing
  10. JOHNSON et al. (1998)
    Students' Thinking and Writing in the Context of Probability
    Written Communication
  11. MATHISON (1996)
    Writing the Critique, a Text about a Text
    Written Communication
  12. NEWELL (1994)
    The Effects of Written Between-Draft Responses on Students' Writing and Reasoni…
    Written Communication
  13. 10.1207/s15326977ea0101_4
     ↗
  14. 10.1598/RRQ.31.4.4
     ↗
  15. 10.1598/RRQ.31.3.4
     ↗
  16. Smolkin (1997)
    Dealing with Dialogue: Fifth Graders’ Responses to Reading a Play
    Research in the Teaching of English
  17. Soliday, M. (1997)
    10.37514/JBW-J.1997.16.1.06
    Journal of Basic Writing  ↗
  18. Sperling (1994)
    Constructing the Perspective of Teacher-as-Reader: A Framework for Studying Res…
    Research in the Teaching of English
  19. Sperling et al. (1997)
    Two Classrooms, Two Writing Communities: Urban and Suburban Tenth Graders Learn…
    Research in the Teaching of English
  20. 10.1598/RRQ.31.4.5
     ↗
  21. Stolarek (1994)
    Prose Modeling and Metacognition: The Effect of Modeling on Developing a Metaco…
    Research in the Teaching of English
  22. 10.1207/s15326977ea0402_2
     ↗
  23. 10.2307/747624
     ↗
  24. GELDEREN (1997)
    Elementary Students' Skills in Revising: Integrating Quantitative and Qualitati…
    Written Communication
  25. 10.1207/s15326977ea0202_1
     ↗
  26. Wollman-Bonilla (1994)
    Why Don’t They “Just Speak?” Attempting Literature Discussion with More and Les…
    Research in the Teaching of English
  27. Yaqelski (1995)
    The Role of Classroom Context in the Revision Strategies of Student Writers
    Research in the Teaching of English
  28. Zhu (1995)
    Effects of Training for Peer Response on Students' Comments and Interaction
    Written Communication
CrossRef global citation count: 43 View in citation network → Build reading path →

Related Articles

  1. Computers and Composition Jun 2026
    Conversing computers, composition, and culture: Lessons on multilingualism and multiliteracies from Indigenous Voices in Digital Spaces ↗
    Analeigh E. Horton; Cindy Tekobbe
    digital rhetoric literacy studies
  2. Computers and Composition Jun 2026
    Beyond paying attention: Praxis for critical digital literacy ↗
    Amber M. Buck
    digital rhetoric literacy studies
  3. Written Communication May 2026
    Genres Fall Apart: Recuperating the Centrifugal in Rhetorical Genre Analysis ↗
    Rebecca Lorimer Leonard; Angela Rounsaville
    rhetorical criticism genre theory multilingual writers literacy studies community literacy
  4. Assessing Writing Apr 2026
    Developing students’ feedback literacy in disciplinary academic writing through generative artificial intelligence ↗
    Jianda Liu; Zihao Shi; Wanqing Li
    artificial intelligence literacy studies
  5. IEEE Transactions on Professional Communication Mar 2026
    Surveys as UXR: Using Design Thinking to Shape a Survey-Based UX Assessment for Rural Audiences ↗
    Kaitlin J. Coyle; Derek G. Ross
    assessment technical communication qualitative research literacy studies