Assessing Writing Oct 2011

The accuracy of performance task scores after resolution of rater disagreement: A Monte Carlo study

James A. Penny ; Robert L. Johnson University of South Carolina
Journal
Assessing Writing
Published
2011-10-01
DOI
10.1016/j.asw.2011.06.001
CompPile
Search in CompPile ↗
Open Access
Closed
Export
BibTeX RIS

Citation Context

Cited by in this index (0)

No articles in this index cite this work.

References (53) · 12 in this index

  1. Baker (1999)
    Dimensions of children's motivation for reading and their relations to reading activity a…
    Reading Research Quarterly  ↗
  2. Bond (1998)
    Culturally responsive pedagogy and the assessment of accomplished teaching
    Journal of Negro Education  ↗
  3. Brennan (1996)
    Generalizability of performance assessments
    Technical issues in large-scale performance assessment
  4. Brown et al. (2004)
    Accuracy in the scoring of writing: Studies of reliability and validity using a…
    Assessing Writing
  5. Cherry (1993)
    Reliability issues in holistic assessment
    Validating holistic scoring for writing assessment: Theoretical and empirical foundations
Show all 53 →
  1. Clauser (1999)
    Components of rater error in a complex performance assessment
    Journal of Educational Measurement  ↗
  2. Cohen (1977)
    Statistical power analysis for the behavioral sciences
  3. College Board. (2011). AP: Art History. Retrieved June 12, 2011, from http://www.collegeboard.com/student/tes…
  4. Crawford et al. (2008)
    When a “sloppy copy” is good enough: Results of a state writing assessment
    Assessing Writing
  5. Cronbach (1997)
    Generalizability analysis for performance assessments of student achievement or school ef…
    Educational and Psychological Measurement  ↗
  6. Dappen et al. (2008)
    A statewide writing assessment model: Student proficiency and future implications
    Assessing Writing
  7. Diedrich (1974)
    Measuring growth in English
  8. East (2009)
    Evaluating the reliability of a detailed analytic scoring rubric for foreign la…
    Assessing Writing
  9. Educational Testing Service. (2004). GRE for educators. Retrieved July 11, 2004, from http://www.gre.org/writ…
  10. Gordon, B. (2000, April). A practitioner's perspective on scoring resolution methods: In defense of differenc…
  11. Greenwald, E., Persky, H., Campbell, J., & Mazzeo, J. (1999). The NAEP 1998 writing report card for the natio…
  12. Hao (2008)
    The differential impact of resolution methods on the operational scores of gender and eth…
    Journal of Applied Measurement
  13. HAYES et al. (2000)
    Does Holistic Assessment Predict Writing Performance?: Estimating the Consisten…
    Written Communication
  14. Hieronymous (1987)
    Handbook for focused holistic scoring
  15. Hogan (1980)
    Relationships between essay tests and objective tests of language skills for elementary s…
    Journal of Educational Measurement  ↗
  16. Huang (2008)
    How accurate are ESL students’ holistic writing scores on large-scale assessmen…
    Assessing Writing
  17. Huot (1990)
    The literature of direct writing assessment: Major concerns and prevailing trends
    Review of Educational Research  ↗
  18. Johnson (2003)
    Score resolution: An investigation of the reliability and validity of resolved scores
    Applied Measurement in Education  ↗
  19. Johnson (2000)
    The relationship between score resolution methods and interrater reliability: An empirica…
    Applied Measurement in Education  ↗
  20. JOHNSON et al. (2001)
    Score Resolution and the Interrater Reliability of Holistic Scores in Rating Essays
    Written Communication
  21. Johnson (2009)
    Assessing performance: Developing, scoring, and validating performance tasks
  22. Johnson (2005)
    Resolving score differences in the rating of writing samples: Does discussion improve the…
    Language Assessment Quarterly  ↗
  23. Johnson, R., Penny, J., & Johnson, C. (2000, April). A conceptual framework for score resolution in the ratin…
  24. Johnson, R., Willeke, M., Bergman, T., & Steiner, D. (1996). Collaboration to create a family literacy portfo…
  25. Lane (1996)
    Generalizability and validity of a mathematics performance assessment
    Journal of Educational Measurement  ↗
  26. LeMahieu (1995)
    Portfolios in large-scale assessment: Difficult but not impossible
    Educational Measurement: Issues and Practice  ↗
  27. Livingston, S. (1998, April). Results of the pilot test of the School Leaders’ Licensure Assessment. Paper pr…
  28. Masters, J. (1992). A study of arbitrations in Pennsylvania's writing assessment. Paper presented at the annu…
  29. McCormick, C., & Geisinger, K. (2009). The potential for interrater reliability inflation with tertium quid r…
  30. Moss, P., Schutz, A., & Collins, K. (1997). Developing coherence between assessment and reform: An integrativ…
  31. Myers (1980)
    A procedure for writing assessment and holistic scoring
  32. Myford (2002)
    When raters disagree, then what: Examining a third-rating discrepancy resolution procedur…
    Journal of Applied Measurement
  33. National Board for Professional Teaching Standards. (2001). Early and middle childhood art: Scoring guide. Wa…
  34. National Board for Professional Teaching Standards. (2010). Early and middle childhood art: Scoring guide. Wa…
  35. Penny (2003)
    Reading high stakes writing samples: My life as a reader
    Assessing Writing
  36. Penny et al. (2000)
    The effect of rating augmentation on inter-rater reliability
    Assessing Writing
  37. Penny (2000)
    Using rating augmentation to expand the scale of an analytic rubric
    Journal of Experimental Education  ↗
  38. Rezaei et al. (2010)
    Reliability and validity of rubrics for assessment through writing
    Assessing Writing
  39. Shumate (2007)
    The effects of the number of scale points and nonnormality on the generalizability coeffi…
    Applied Measurement in Education  ↗
  40. Test Scoring and Reporting Services. (2000). Test Scoring and Reporting Services: 1999–2000 report. Retrieved…
  41. Taylor (1998)
    An investigation of scoring methods for mathematics performance-based assessments
    Educational Assessment  ↗
  42. Welch, C., & Martinovich-Barhite, D. (1997, April). Reliability issues and possible solutions. Paper presente…
  43. White (1993)
    Holistic scoring: Past triumphs, future challenges
    Validating holistic scoring for writing assessment: Theoretical and empirical foundations
  44. White (1994)
    Teaching and assessing writing: Recent advances in understanding, evaluating, and improving student performance
  45. Whithaus et al. (2008)
    Keyboarding compared with handwriting on a high-stakes writing assessment: Stud…
    Assessing Writing
  46. Wiley (1996)
    Extended assessment tasks: Purposes, definitions, scoring, and accuracy
    Implementing performance assessment: Promises, problems, and challenges
  47. Willard-Traub et al. (1999)
    The development of large-scale portfolio placement assessment at the University…
    Assessing Writing
  48. Wolcott, W. (with Legg, S.) (1998). An overview of writing assessment: Theory, research, and practice. Urbana…
CrossRef global citation count: 7 View in citation network → Build reading path →