Assessing Writing Apr 2026 Open Access

Evaluating the consistency between human raters and three AI systems on the scoring of argumentative essays

SeongYeup Kim Pennsylvania State University ; Matthew T. McCrudden Pennsylvania State University
Journal
Assessing Writing
Published
2026-04-01
DOI
10.1016/j.asw.2026.101031
CompPile
Search in CompPile ↗
Open Access
OA PDF Green
Export
BibTeX RIS

Citation Context

Cited by in this index (0)

No articles in this index cite this work.

References (80) · 2 in this index

  1. Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D.R., Smith, N.A., & Tsvetkov, Y. (2023). Do all langua…
     ↗
  2. Ajay H.B., Tillett P.I., Page E.B. (1973) Analysis of essays by computer (AEC-II) (Report No. 8-0102). Washin…
  3. Attali (2006)
    Automated essay scoring with e-rater® V. 2
    The Journal of Technology, Learning and Assessment
  4. Attali (2013)
    Scoring with the computer: Alternative procedures for improving the reliability of holist…
    Language Testing  ↗
  5. Baek (2024)
    ChatGPT seems too good to be true”: College students’ use and perceptions of generative AI
    Computers and Education: Artificial Intelligence
Show all 80 →
  1. Banihashem (2024)
    Feedback sources in essay writing: peer-generated or AI-generated feedback?
    International Journal of Educational Technology in Higher Education  ↗
  2. Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., Arx, S. von, Bernstein, M.S., Bohg, J., Bossel…
  3. Bouziane (2024)
    AI versus human effectiveness in essay evaluation
    Discover Education  ↗
  4. Broad, B. (2003). What we really value: Beyond rubrics in Teaching and Assessing Writing All USU Press Public…
     ↗
  5. Bucol (2025)
    Exploring ChatGPT as a writing assessment tool
    Innovations in Education and Teaching International  ↗
  6. Bui (2024)
    ChatGPT as an automated essay scoring tool in the writing classrooms: how it compares wit…
    Education and Information Technologies  ↗
  7. Burke, R.A. (2003). The influence of rubrics on high school students' creative writing skills [Master's thesi…
  8. Coe, M., Hanita, M., Nishioka, V., & Smiley, R. (2011). An investigation of the impact of the 6+1 trait writi…
  9. Correnti (2022)
    Building a validity argument for an automated writing evaluation system (eRevise) as a fo…
    Computers and Education Open  ↗
  10. Crusan (2010)
    Assessment in the second language writing classroom
  11. Culham, R. (2003). 6+ 1 traits of writing: The complete guide grades 3 and up. Scholastic Inc.
  12. Deane (2019)
    The case for scenario-based assessment of written argumentation
    Reading and Writing  ↗
  13. Deane (2013)
    Automated essay scoring in innovative assessments of writing from sources
    Journal of Writing Assessment
  14. DeJarnette, N.K. (2008). Effect of the 6+ 1 trait writing model on student writing achievement. (Publication …
  15. Delgado (2024)
    Relationships between middle-school teachers' perceptions and application of automated wr…
    The Routledge international handbook of automated essay evaluation
  16. Diederich (1974)
    Measuring growth in English
  17. Dikli, S. (2006). An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment…
  18. Doewes, A., Kurdhi, N., & Saxena, A. (2023). Evaluating quadratic weighted kappa as the standard performance …
  19. Etxaniz, J., Azkune, G., Soroa, A., de Lacalle, O.L., & Artetxe, M. (2023). Do multilingual language models t…
     ↗
  20. Fang (2023)
    Artificial intelligence-based assessment in education
    Handbook of artificial intelligence in education
  21. Freedman (2016)
    Teaching writing in a digital and global age: Toward access, learning, and development for all
    Handbook of research on teaching
  22. Graham (2007)
    Writing next-effective strategies to improve writing of adolescents in middle and high schools
    Alliance For Excellent Education
  23. Gupta, V., Chowdhury, S.P., Zouhar, V., Rooein, D., & Sachan, M. (2025). Multilingual performance biases of l…
  24. Huawei (2023)
    A systematic review of automated writing evaluation systems
    Education and Information Technologies  ↗
  25. Ifenthaler (2022)
    Automated essay scoring systems
    Handbook of open, distance and digital education
  26. Kim (2024)
    ChatGPT for writing evaluation: Examining the accuracy and reliability of AI-generated sc…
    Exploring artificial intelligence in applied linguistics
  27. Klebanov, B.B., & Madnani, N. (2020). Automated evaluation of writing – 50 years and counting. In Proceedings…
     ↗
  28. Li (2025)
    LLM-based automated grading with human-in-the-loop
    arXiv Preprint arXiv
  29. Li (2014)
    The role of automated writing evaluation holistic scores in the ESL classroom
    System  ↗
  30. Li (2024)
    Quantifying multilingual performance of large language models across languages
    arXiv Preprint arXiv
  31. Link (2023)
    Automated scoring of writing
    Digital writing technologies in higher education: Theory, research, and practice
  32. Liu (2025)
    Enhancing GPT-based automated essay scoring: the impact of fine-tuning and linguistic com…
    Computer Assisted Language Learning
  33. Lu (2024)
    Can ChatGPT effectively complement teacher assessment of undergraduate students’ academic…
    Assessment Evaluation in Higher Education  ↗
  34. Manning (2025)
    Human versus machine: The effectiveness of ChatGPT in automated essay scoring
    Innovations in Education and Teaching International
  35. Mansour (2024)
    Can large language models automatically score proficiency of written essays?
    arXiv Preprint
  36. Messick (1995)
    Validity of psychological assessment: Validation of inferences from persons' responses an…
    American Psychologist  ↗
  37. Mirzaeian (2025)
    A comparative evaluation of artificial intelligence scoring versus human scoring of EFL s…
    Teaching English as a Second Language Quarterly
  38. Mizumoto (2023)
    Exploring the potential of using an AI language model for automated essay scoring
    Research Methods in Applied Linguistics  ↗
  39. Olson (1976)
    On choosing a test statistic in multivariate analysis of variance
    Psychological Bulletin  ↗
  40. Pack (2024)
    Large language models and automated essay scoring of English language learner writing: In…
    Computers and Education: Artificial Intelligence
  41. Page (1967)
    The imminence of… grading essays by computer
    The Phi Delta Kappan
  42. Palermo (2024)
    Automated essay evaluation at scale: Hybrid automated scoring/hand scoring in the summati…
    The Routledge international handbook of automated essay evaluation
  43. Palermo et al. (2020)
    Implementing Automated Writing Evaluation in Different Instructional Contexts: …
    Journal of Writing Research
  44. Perelman (2014)
    When “the state of the art” is counting words
    Assessing Writing
  45. Petrov (2023)
    Language model tokenizers introduce unfairness between languages
    Advances in Neural Information Processing Systems
  46. Porath (2010)
    6-Traits writing rubric: Things that make us smart can also make us dumb
    The Wisconsin English Journal
  47. Ramesh (2022)
    An automated essay scoring systems: a systematic literature review
    Artificial Intelligence Review  ↗
  48. Rudner (2002)
    Automated essay scoring using Bayes' theorem
    The Journal of Technology, Learning and Assessment
  49. Saglam (2025)
    Comparative evaluation of artificial intelligence models GPT-4 and GPT-3.5 in clinical de…
    BMC Medical Informatics and Decision Making  ↗
  50. Schleppegrell (2004)
    The language of schooling: A functional linguistics perspective
  51. Seßler, K., Fürstenberg, M., Bühler, B., & Kasneci, E. (2025). Can AI grade your essays? A comparative analys…
     ↗
  52. Shermis (2013)
    Contrasting state-of-the-art automated scoring of essays
    Handbook of automated essay evaluation
  53. Shermis (2001)
    On-line grading of student essays: PEG goes on the World Wide Web
    Assessment Evaluation in Higher Education  ↗
  54. Shermis (2024)
    Introduction to automated essay evaluation
    The Routledge international handbook of automated essay evaluation
  55. Steiss (2024)
    Comparing the quality of human and ChatGPT feedback of students’ writing
    Learning and Instruction  ↗
  56. (2006)
    Automated scoring of complex tasks in computer-based testing
  57. Tate (2024)
    Can AI provide useful holistic essay scoring?
    Computers and Education: Artificial Intelligence
  58. Valenti (2003)
    An overview of current research on automated essay grading
    Journal of Information Technology Education: Research
  59. Wang (2018)
    Automatic essay scoring incorporating rating schema via reinforcement learning
    Proceedings of the 2018 conference on empirical methods in natural language processing
  60. Warschauer (2008)
    Automated writing assessment in the classroom
    Pedagogies: An International Journal  ↗
  61. Warschauer (2006)
    Automated writing evaluation: Defining the classroom research agenda
    Language Teaching Research  ↗
  62. Weigle (2002)
    Assessing writing
  63. Wetzler (2024)
    Grading the graders: comparing generative AI and human assessment in essay evaluation
    Teaching of Psychology  ↗
  64. (2013)
    Handbook of automated essay evaluation
  65. Williamson (2012)
    A framework for evaluation and use of automated scoring
    Educational Measurement: Issues and Practice  ↗
  66. Wilson (2024)
    Recommendations for integrating automated writing evaluation with evidence-based instruct…
    International Journal of Changes in Education  ↗
  67. Wilson (2016)
    Automated essay evaluation software in english language arts classrooms: Effects on teach…
    Computers Education  ↗
  68. Wilson (2021)
    Automated feedback and automated scoring in the elementary grades: Usage, attitudes, and …
    International Journal of Artificial Intelligence in Education  ↗
  69. Wilson (2020)
    Automated writing evaluation and feedback: Multiple metrics of efficacy
    Journal of Educational Computing Research  ↗
  70. Wilson (2024)
    A latent dirichlet allocation approach to understanding students’ perceptions of automate…
    Computers and Education Open  ↗
  71. Wu, T.I., Lo, T.H., Chao, F.A., Sung, Y.T., & Chen, B. (2023). Effective neural modeling leveraging readabili…
     ↗
  72. Xiao, C., Ma, W., Song, Q., Xu, S.X., Zhang, K., Wang, Y., & Fu, Q. (2025). Human-AI collaborative essay scor…
     ↗
  73. Yang (2024)
    Learner interaction with, and response to, AI-programmed automated writing evaluation fee…
    Education and Information Technologies  ↗
  74. Zhai (2024)
    Multi-prompts learning with cross-modal alignment for attribute-based person re-identification
    Proceedings of the AAAI Conference on Artificial Intelligence  ↗
  75. Zupanc (2017)
    Automated essay evaluation with semantic analysis
    Knowledge-Based Systems  ↗
CrossRef global citation count: 0 View in citation network → Build reading path →