Abstract
This study addressed the question, “How consistently do students perform on holistically scored writing assignments?” Instructors from 13 introductory writing classes at two colleges were asked to provide essay sets written by their students in response to the three to five most important writing assignments in their classes. In all, 796 essays were collected from 241 students. The study drew on a pool of 15 experienced judges to evaluate the essays. Each essay set was scored holistically and independently by 6 of the judges who either ranked or graded the essays in the set. All papers written by a particular student were scored by the same judges. Pairwise correlations of the scores assigned to each essay set were computed for each judge and then averaged across judges. The average of these correlations was 0.16, indicating very low consistency of holistically scored student performance from essay to essay. This result suggests that drawing conclusions from one or even a few writing samples is problematic.