Assessing Writing Oct 2024 Open Access

The impact of task duration on the scoring of independent writing responses of adult L2-English writers

Ben Naismith Advanced Cooling Technologies (United States) ; Yigal Attali Advanced Cooling Technologies (United States) ; Geoffrey T. LaFlair Advanced Cooling Technologies (United States)

Abstract

In writing assessment, there is inherently a tension between authenticity and practicality: tasks with longer durations may more closely reflect real-life writing processes but are less feasible to administer and score. What is more, given total testing time, there is necessarily a trade-off between task duration and number of tasks. Traditionally, high-stakes assessments have managed this trade-off by administering one or two writing tasks each test, allowing 20–40 minutes per task. However, research on second language (L2) English writing has not found longer task durations to significantly improve score validity or reliability. Importantly, very few studies have compared much shorter durations for writing tasks to more traditional allotments. To explore this issue, we asked adult L2-English test takers to respond to two writing prompts with either 5-minute or 20-minute time limits. Responses were then evaluated by expert human raters and an automated writing evaluation tool. Regardless of scoring method, short duration scores evidenced equally high test-retest reliability and criterion validity as long duration scores. As expected, longer task duration yielded higher scores, but regardless of duration, test takers demonstrated the entire spectrum of writing proficiency. Implications for writing assessment are discussed in relation to scoring practices and task design. • Longer writing tasks do not have higher test-retest reliability than shorter ones. • Longer writing tasks do not have higher criterion validity than shorter ones. • The impact of task duration is not mediated by scoring method (human or machine).

Journal
Assessing Writing
Published
2024-10-01
DOI
10.1016/j.asw.2024.100895
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
assessment artificial intelligence
Export
BibTeX RIS

Citation Context

Cited by in this index (0)

No articles in this index cite this work.

References (92) · 12 in this index

  1. Tabari et al. (2023)
    Exploring new insights into the role of cohesive devices in written academic genres
    Assessing Writing
  2. Ackerman (2009)
    Test length and cognitive fatigue: An empirical examination of effects on performance and…
    Journal of Experimental Psychology: Applied
  3. Ackerman (2010)
    Cognitive fatigue during testing: An examination of trait, time-on-task, and strategy inf…
    Human Performance  ↗
  4. American Educational Research Association (2014)
    Standards for educational and psychological testing
  5. Applebee (1990)
    Learning to Write in Our Nation’s Schools: Instruction and Achievement in 1988 at Grades 4, 8, and 12 NAEP (AMP-90-3480, NAEP-19-W-02; p. 122)
Show all 92 →
  1. Attali, Y. (2012, April). Factor structure of the e-rater automated essay scoring system [Paper presentation]…
  2. Attali (2013)
    Validity and reliability of automated essay scoring
    Handbook of automated essay evaluation
  3. Attali (2016)
    A comparison of newly-trained and experienced raters on a standardized writing assessment
    Language Testing  ↗
  4. Bachman (2010)
    Language assessment in practice: Developing language assessments and justifying their use in the real world
  5. Banerjee (2007)
    Documenting features of written language production typical at different IELTS band score…
    IELTS Research Reports
  6. Barkaoui (2011)
    Think-aloud protocols in research on essay rating: An empirical study of their veridicali…
    Language Testing  ↗
  7. Ben-Simon (2007)
    Toward More Substantively Meaningful Automated Essay Scoring
    The Journal of Technology Learning and Assessment
  8. Biber (2017)
    Exploring the relationship between TOEFL iBT scores and disciplinary writing performance
    TESOL Quarterly  ↗
  9. Biola (1982)
    Time Limits and Topic Assignments for Essay Tests
    Research in the Teaching of English
  10. Braddock (1963)
    Research in written composition
  11. Breland (1987)
    Assessing Writing Skill. Research Monograph No. 11
  12. Breland (1999)
    Writing assessment in admission to higher education: Review and framework
    ETS Research Report Series  ↗
  13. Brennan (1995)
    Generalizability analyses of WorkKeys listening and writing tests
    Educational and Psychological Measurement  ↗
  14. Bridgeman (2020)
    Relationship between testing time and testing outcomes
    Integrating Timing Considerations to Improve Testing Practices
  15. British Council. (2023). IELTS Practice Academic Writing test 2 - Task 2. In Take IELTS. 〈https://takeielts.b…
  16. Brown (1984)
    A categorical instrument for scoring second language writing skills
    Language Learning  ↗
  17. Burnstein (2022)
    A theoretical assessment ecosystem for a digital-first assessment—The Duolingo English Test
    Duolingo Research Reports (DRR-22-01)
  18. Burstein (2013)
    The e-rater® automated essay scoring system
    Handbook of automated essay evaluation: Current applications and new directions
  19. Cambridge University Press (2009)
    Cambridge IELTS 7: Examination Papers from University of Cambridge ESOL Examinations: English for Speakers of Other Languages
  20. Cambridge University Press (2011)
    Cambridge IELTS 8: Examination Papers from University of Cambridge ESOL Examinations: English for Speakers of Other Languages
  21. Cardwell (2023)
    Duolingo English Test: Technical Manual
    Duolingo Research Reports
  22. Creswell (2011)
    Designing and conducting mixed methods research
    SAGE Publications
  23. Caudery (1990)
    The validity of timed essay tests in the assessment of writing skills
    ELT Journal  ↗
  24. Chen (2022)
    Examining human and automated ratings of elementary students’ writing quality: A multivar…
    American Educational Research Journal  ↗
  25. Cho (2003)
    Assessing writing: Are we bound by only one method?
    Assessing Writing
  26. Cohen (1988)
    Statistical power analysis for the behavioral sciences
  27. Cooper (1984)
    The assessment of writing ability: A review of research
    ETS Research Report Series  ↗
  28. Crawford (2004)
    Writing performance assessments: How important is extended time?
    Journal of Learning Disabilities  ↗
  29. Crossley (2014)
    Linguistic microfeatures to predict L2 writing proficiency: A case study in automated wri…
    Journal of Writing Assessment
  30. Cumming et al. (2005)
    Differences in written discourse in independent and integrated prototype tasks …
    Assessing Writing
  31. Cumming (2002)
    Decision making while rating ESL/EFL writing tasks: A descriptive framework
    The Modern Language Journal  ↗
  32. Cushing (2022)
    Comparison of IELTS Academic and Duolingo English Test
    IELTS Research Reports
  33. Davis (2023)
    A comparison of two TOEFL® writing tasks
    ETS Research Memorandum Series (ETS RM–23-06)
  34. Dunn (1969)
    Correlation coefficients measured on the same individuals
    Journal of American Statistical Association  ↗
  35. Deane (2013)
    On the relation between automated essay scoring and modern views of the writing…
    Assessing Writing
  36. Eckes (2012)
    Operational rater types in writing assessment: Linking rater cognition to rater behavior
    Language Assessment Quarterly  ↗
  37. Educational Testing Service (2020)
    The official guide to the TOEFL iBT Test
  38. Enright (2010)
    Complementing human judgment of essays written by English language learners with e-rater(…
    Language Testing  ↗
  39. Fleiss (2003)
    Statistical Methods for Rates and Proportions
  40. Gebril (2009)
    Score generalizability of writing tasks: Does one test method fit it all?
    Language Testing  ↗
  41. Gebril (2010)
    Bringing reading-to-write and writing-only assessment tasks together: A general…
    Assessing Writing
  42. Goh (2018)
    Teacher-examiners’ explicit and enacted beliefs about proficiency indicators in national …
    Teacher Involvement in High-Stakes Language Testing
  43. Guo et al. (2013)
    Predicting human judgments of essay quality in both integrated and independent …
    Assessing Writing
  44. Hale (1992)
    Effects of amount of time allowed on the test of written English
    ETS Research Report Series  ↗
  45. Hamp-Lyons (1990)
    Second language writing: Assessment issues
    Second language writing
  46. Huang (2012)
    Using generalizability theory to examine the accuracy and validity of large-sca…
    Assessing Writing
  47. In'nami (2016)
    Task and rater effects in L2 speaking and writing: A synthesis of generalizability studies
    Language Testing
  48. Jurich (2020)
    A history of test speededness: Tracing the evolution of theory and practice
    Integrating Timing Considerations to Improve Testing Practices
  49. Kane (1982)
    A sampling model for validity
    Applied Psychological Measurement  ↗
  50. Kane (2020)
    The impact of time limits and timing information on validity
    Integrating Timing Considerations to Improve Testing Practices
  51. Kenworthy (2006)
    Timed versus at-home assessment tests: Does time affect the quality of second language le…
    TESL-Ejemplo
  52. Khuder et al. (2015)
    Writing in Test and Non-test Situations: Process and Product
    Journal of Writing Research
  53. Klein (1981)
    The effects of time limits, item sequence, and question format on applicant performance on the California Bar Examination (No. 81-7)
  54. Knoch (2010)
    Validity and fairness implications of varying time conditions on a diagnostic test of aca…
    System  ↗
  55. Kroll (1990)
    What does time buy? ESL student performance on home versus class compositions
    Second Language Writing (Cambridge Applied Linguistics): Research Insights for the Classroom
  56. Landis (1977)
    The measurement of observer agreement for categorical data
    Biometrics  ↗
  57. (2014)
    The Oxford Handbook of Qualitative Research
  58. Lee, J., & Son, M. (2023). The effects of varied time constraints on writing performance. 〈https://www.aaal.o…
  59. Lee (2021)
    The effect of additional time on the quality of argumentation in L2 writing assessment: A…
    Language Assessment Quarterly  ↗
  60. Lee (2005)
    Dependability of new writing task scores: Evaluating prototype tasks and alternative rati…
    TOEFL Monograph Series, 31
  61. Livingston (1987)
    The effects of time limits on the quality of student-written essays
    Paper presented at the Annual Meeting of the American Educational Research Association
  62. Llosa (2019)
    Comparability of students’ writing performance on TOEFL iBT and in required university wr…
    Language Testing  ↗
  63. Lovett (2010)
    Effects of response mode and time allotment on college students’ writing
    Journal of College Reading and Learning  ↗
  64. Margolis (2020)
    Timing considerations for performance assessments
    Integrating Timing Considerations to Improve Testing Practices
  65. Meng (1992)
    Comparing correlated correlation coefficients
    Psychological Bulletin  ↗
  66. Moore (1999)
    Authenticity in the IELTS Academic Module Writing Test: A comparative study of Task 2 ite…
    British Council/IDP Australia Research Reports
  67. Page (1966)
    The imminence of grading essays by computer
    Phi Delta Kappan
  68. Pearson. (2023). Pearson shortens PTE Academic test and launches online version Pearson PTE. Pearson PTE. 〈ht…
  69. Polio (1998)
    If I only had more time:” ESL learners’ changes in linguistic accuracy on essay revisions
    Journal of Second Language Writing  ↗
  70. Polio (2016)
    Thematic analysis. In
    Understanding, Evaluating, and Conducting Second Language Writing Research
  71. Polio (1996)
    ESL writing assessment prompts: How students choose
    Journal of Second Language Writing  ↗
  72. Powers (1996)
    Effects of applying different time limits to a proposed GRE writing test
    Journal of Educational Measurement  ↗
  73. Qian (2023)
    Use of lexical features in high-stakes tests: Evidence from the perspectives of…
    Assessing Writing
  74. Ramineni (2012)
    Evaluation of e-rater® for the GRE® issue and argument prompts
    ETS Research Report Series  ↗
  75. Révésv (2017)
    Investigating IELTS Academic Writing Task 2: Relationships between cognitive writing proc…
    IELTS Research Reports
  76. Schoonen (2005)
    Generalizability of writing scores: An application of structural equation modeling
    Language Testing  ↗
  77. Shavelson (1993)
    Sampling variability of performance assessments
    Journal of Educational Measurement  ↗
  78. Shin (2011)
    The effects of time allocation on Korean college students’ performance of drafted and tim…
    English Teaching  ↗
  79. Vaughan (1991)
    Holistic assessment: What goes on in the rater’s mind?
    Assessing second language writing in academic contexts
  80. Weigle (2002)
    Assessing writing
  81. Weigle (2010)
    Validation of automated scores of TOEFL iBT tasks against non-test indicators of writing …
    Language Testing  ↗
  82. White (1995)
    An Apologia for the Timed Impromptu Essay Test
    College Composition and Communication
  83. Wu (2016)
    The effect of timing on the quantity and quality of test-takers’ writing
    New Zealand Studies in Applied Linguistics
  84. Yang et al. (2023)
    Predicting EFL expository writing quality with measures of lexical richness
    Assessing Writing
  85. (1921)
  86. Younkin (1986)
    Speededness as a source of test bias for non-native English speakers on the college level academic skills test [PhD thesis]
  87. Zhang (2014)
    Investigating native and non-native English-speaking teacher raters’ judgements of oral p…
    Assessment in Education: Principles, Policy Practice
CrossRef global citation count: 2 View in citation network → Build reading path →

Related Articles

  1. Assessing Writing Jul 2026
    LAWE-CL2: Multi-agent LLM-based automated writing evaluation system integrating linguistic features with fine-tuning for Chinese L2 writing assessment ↗
    Xuelin Wang; Qihao Yang; Yuxin Hao; Zhijun Wang; Sijia Guo
    assessment artificial intelligence multilingual writers
  2. Assessing Writing Jul 2026
    Investigating the impact of ChatGPT-assisted self-assessment on college students' writing development: Insights from diverse linguistic backgrounds ↗
    Hamidreza Moeiniasl
    assessment artificial intelligence
  3. Assessing Writing Apr 2026
    Assessing fairness in finetuned scoring models with demographically restricted training data ↗
    Langdon Holmes; Wesley Morris; Scott Crossley; Joon Suh Choi
    assessment artificial intelligence race and writing
  4. Assessing Writing Apr 2026
    Generative artificial intelligence for automated writing evaluation: A systematic review of trends, efficacy, and challenges ↗
    Shadi I. Abudalfa; Jessie S. Barrot
    assessment artificial intelligence book reviews
  5. Journal of Technical Writing and Communication Mar 2026
    Canon to Code: Rhetorical Rulemaking for Generative AI Content Audits and Governance ↗
    Miriam F. Williams
    classical rhetoric modern rhetorical theory rhetorical criticism assessment technical communication artificial intelligence