Issues in Measuring Reliability

Written Communication Jul 1999

JOHN R. HAYES ; JILL A. HATCH

Abstract

In many literacy studies, it is important to establish the reliability of independent observers' judgments. Reliability most commonly is measured either by the percentage of agreement or the correlation between the observers' judgments. This article argues that the percentage of agreement measure is more difficult to interpret than are correlation measures because of the following: (a) the effects of chance agreement are not accounted for automatically by the percentage of agreement measure; and (b) rates of chance agreement are strongly influenced by the variability of the data, by “ceiling” and “floor” effects, and by the scoring of near agreement as perfect agreement. For these reasons, the authors recommend that the field of literacy research adopt correlation as the standard method for estimating the reliability of observers' judgments.

Journal: Written Communication
Published: 1999-07-01
DOI: 10.1177/0741088399016003004
Open Access: Closed
Topics: literacy studies

Citation Context

Cited by in this index (1)

JOHNSON et al. (2001)

Score Resolution and the Interrater Reliability of Holistic Scores in Rating Essays

Written Communication

Cites in this index (15)

CRAMMOND (1998)

The Uses and Complexity of Argument Structures in Expert and Student Persuasive…

Written Communication
JOHNSON et al. (1998)

Students' Thinking and Writing in the Context of Probability

Written Communication
BOICE (1997)

Which is more Productive, Writing in Binge Patterns of Creative Illness or in M…

Written Communication
GELDEREN (1997)

Elementary Students' Skills in Revising

Written Communication
Smolkin (1997)

Dealing with Dialogue: Fifth Graders’ Responses to Reading a Play

Research in the Teaching of English

Show all 15 →

Sperling et al. (1997)

Two Classrooms, Two Writing Communities: Urban and Suburban Tenth Graders Learn…

Research in the Teaching of English
FABER (1996)

Rhetoric in Competition

Written Communication
MATHISON (1996)

Writing the Critique, a Text about a Text

Written Communication
Zhu (1995)

Effects of Training for Peer Response on Students' Comments and Interaction

Written Communication
Yaqelski (1995)

The Role of Classroom Context in the Revision Strategies of Student Writers

Research in the Teaching of English
Wollman-Bonilla (1994)

Why Don’t They “Just Speak?” Attempting Literature Discussion with More and Les…

Research in the Teaching of English
CHAPMAN (1994)

The Emergence of Genres

Written Communication
NEWELL (1994)

The Effects of Written Between-Draft Responses on Students' Writing and Reasoni…

Written Communication
Sperling (1994)

Constructing the Perspective of Teacher-as-Reader: A Framework for Studying Res…

Research in the Teaching of English
Stolarek (1994)

Prose Modeling and Metacognition: The Effect of Modeling on Developing a Metaco…

Research in the Teaching of English

Also cites 17 works outside this index ↓

10.1016/1075-2935(95)90005-5

↗
10.2307/747620

↗
10.2307/748198

↗
10.2307/748030

↗
10.1598/RRQ.31.1.4

↗
10.1016/1075-2935(95)90013-6

↗
10.1598/RRQ.31.3.5

↗
10.2307/747631

↗
10.1016/S1075-2935(96)90004-5

↗
10.1207/s15326977ea0101_4

↗
10.1598/RRQ.31.4.4

↗
10.1598/RRQ.31.3.4

↗
Soliday, M. (1997)

10.37514/JBW-J.1997.16.1.06

Journal of Basic Writing ↗
10.1598/RRQ.31.4.5

↗
10.1207/s15326977ea0402_2

↗
10.2307/747624

↗
10.1207/s15326977ea0202_1

↗

CrossRef global citation count: 43 View in citation network →