Assessing Writing Jul 2025 Open Access

A large-scale corpus for assessing source-based writing quality: ASAP 2.0

Scott A. Crossley Vanderbilt University ; Perpetual Baffour Texas Education Agency ; L. Burleigh ; Jules King

Abstract

This paper introduces ASAP 2.0, a dataset of ∼25,000 source-based argumentative essays from U.S. secondary students. The corpus addresses the shortcomings of the original ASAP corpus by including demographic data, consistent scoring rubrics, and source texts. ASAP 2.0 aims to support the development of unbiased, sophisticated Automatic Essay Scoring (AES) systems that can foster improved educational practices by providing summative to students. The corpus is designed for broad accessibility with the hope of facilitating research into writing quality and AES system biases. • We introduce the ASAP 2.0 corpus. • The corpus contains over 25,000 source-based essays. • Each essay is scored for overall writing quality. • The corpus can be used to computationally and quantitatively model source-based writing quality.

Journal
Assessing Writing
Published
2025-07-01
DOI
10.1016/j.asw.2025.100954
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Export
BibTeX RIS

Citation Context

Cited by in this index (0)

No articles in this index cite this work.

References (35) · 5 in this index

  1. Baffour (2023)
    Analyzing bias in large language model solutions for assisted writing feedback tools: les…
    In Proceedings of the 2023 BEA 18th Workshop on Innovative Use of NLP for Building Educational Applications
  2. Blanchard (2013)
    TOEFL11: A corpus of non-native English
    ETS Research Report Series  ↗
  3. Condon (2013)
    Large-scale assessment, locally-developed measures, and automated scoring of es…
    Assessing Writing
  4. Crossley (2014)
    A multidimensional analysis of essay writing: What linguistic features tell us about situ…
    Multi-Dimensional Analysis, 25 years on: A Tribute to Douglas Biber
  5. Crossley (2023)
    Measuring second language proficiency using the English Language Learner Insight, Profici…
    International Journal of Learner Corpus Research  ↗
Show all 35 →
  1. Deane (2013)
    On the relation between automated essay scoring and modern views of the writing…
    Assessing Writing
  2. Dikli, S. (2006). An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment, 5.
  3. Elbow (1986)
    Using portfolios to judge writing proficiency at SUNY Stony Brook
    New directions in college writing programs
  4. Feng (2019)
    The roles of handwriting and keyboarding in writing: A meta-analytic review
  5. Granger (2009)
    International Corpus of Learner English (Version 2)
  6. Haswell (2006)
    Automatons and automated scoring: Drudges, black boxes, and dei ex machina
    Machine scoring of student essays: Truth and consequences
  7. (2006)
    Machine scoring of student essays: Truth and consequences
  8. Herrington (2001)
    What happens when machines read our students’ writing?
    College English
  9. Holmes, L., Morris, W., Crossley, S.A., & Choi, J.S. (in press). Assessing the Reliability and Concurrent Val…
  10. Huot (1996)
    Toward a New Theory of Writing Assessment
    College Composition and Communication
  11. Hunter (1996)
    The use of holistic versus analytic scoring for large-scale assessment of writing
    The Canadian Journal of Program Evaluation  ↗
  12. Li, S., & Ng, V. (2024). Automated essay scoring: Recent successes and future directions. In Proceedings of t…
     ↗
  13. Ling (2014)
    A study on the impact of fatigue on human raters when scoring speaking responses
    Language Testing  ↗
  14. Mayfield, E., & Black, A.W. (2020, July). Should you fine-tune BERT for automated essay scoring?. In Proceedi…
     ↗
  15. Page, E.B.(2003). Project Essay Grade: PEG. In M. D. Shermis & J. Burstein (Eds.), (pp. 43-54). Mahwah, NJ: L…
  16. Perelman, L. (2012). Construct validity, length, score, and time in holistically graded writing assessments: …
     ↗
  17. Schneider, M., & Garg, K. (2021). We need a SpaceX for assessment. The Hechinger Report. 〈https://hechingerre…
  18. Shermis (2014)
    State-of-the-art automated essay scoring: Competition, results, and future dire…
    Assessing Writing
  19. Shermis, M.D. (2024). Using ChatGPT to Score Essays and Short-Form Constructed Responses. arXiv preprint arXi…
  20. Shermis (2002)
  21. Shermis (2003)
    Automated essay scoring: A cross-disciplinary perspective
  22. Shermis (2006)
    Applications of computers in assessment and analysis of writing
    Handbook of writing research
  23. Shermis (2013)
    Contrasting state-of-the-art automated scoring of essays
    In Handbook of automated essay evaluation
  24. Tenney, I., Das, D., & Pavlick, E. (2019). BERT Rediscovers the Classical NLP Pipeline. In Proceedings of the…
     ↗
  25. Wardle et al. (2012)
    Addressing the complexity of writing development: Toward an ecological model of…
    Assessing Writing
  26. Warner, J. (2018). Why they can’t write: Killing the five-paragraph essay and other necessities. JHU Press.
  27. Williamson (1999)
    Mental model comparison of automated and human scoring
    Journal of Educational Measurement  ↗
  28. (2020)
    Handbook of automated scoring: Theory into practice
  29. Yang (2020)
    Enhancing automated essay scoring performance via fine-tuning pre-trained language models…
    Computational Linguistics: EMNLP 2020
  30. Yannakoudakis, H., Briscoe, T., & Medlock, B. (2011). A new dataset and method for automatically grading ESOL…
CrossRef global citation count: 2 View in citation network → Build reading path →