Assessing Writing Oct 2025 Open Access

Judgment accuracy in primary school EFL writing assessment: Do text characteristics matter?

Ruth Trüb FHNW University of Applied Sciences and Arts ; Jens Möller ; Julian Lohmann ; Thorben Jansen Leibniz Institute for Science and Mathematics Education ; Stefan D. Keller

Abstract

Assessing the writing competence of pupils learning English as a foreign language (EFL) at primary school is challenging. This study aimed at examining a largely unexplored topic, namely the role of text characteristics in writing assessment, and analysed judgment accuracy differentiated by nine aspects of text quality (communicative effect, level of detail, coherence, cohesion, complexity of syntax and grammar, correctness of syntax and grammar, vocabulary, orthography and punctuation). Two hundred pre-service teachers assessed four randomly assigned texts from learners in grade six. Their assessment was compared to the existing ratings of two experts from a previous study. We found a relative judgment accuracy between r = .34 and .60 for the nine assessment criteria, with vocabulary being assessed significantly more accurately than almost all other criteria. Orthography, complexity and correctness of syntax and grammar and punctuation were rated with significantly more accuracy than cohesion, level of detail, communicative effect and coherence. The pre-service teachers assessed most criteria more strictly and with higher variability than the experts. The results suggest that teacher education should offer pre-service teachers concrete opportunities to practise writing assessment, implement activities to strengthen the assessment of content- and structure-related criteria, and help them adjust their assessment rigour. • Judgment accuracy in the assessment of primary school EFL learners’ texts. • Relative judgment accuracy between r = .34 and .60 for the different criteria. • Significant differences in relative judgment accuracy between assessment criteria. • Linguistic text qualities are assessed with more accuracy than content- and structure-related aspects. • Pre-service teachers are more rigorous and heterogeneous in rating than experts.

Journal
Assessing Writing
Published
2025-10-01
DOI
10.1016/j.asw.2025.100957
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
teacher development assessment multilingual writers grammar and mechanics
Export
BibTeX RIS

Citation Context

Cited by in this index (1)

  1. Trüb et al. (2026)
    Pursuing fair writing assessment: Halo effects in primary school foreign langua…
    Assessing Writing

References (65) · 3 in this index

  1. Augst (2007)
    Text – Sorten – Kompetenz. Eine echte Longitudinalstudie zur Entwicklung der Textkompetenz im Grundschulalter [Text - genre - competence. A true longitudinal study on the development of text competence at primary school age]
  2. Behrmann (2013)
    The relation between teachers’ diagnostic sensitivity, their instructional activities, an…
    Zeitschrift für Pädagogische Psychologie  ↗
  3. Brinker (2018)
    Linguistische Textanalyse: Eine Einführung in Grundbegriffe und Methoden [Linguistic text analysis: An introduction to basic concepts and methods]
  4. Brock (2015)
    Handbuch writing (2014 und 2015). Kommentierte Schreibperformanzen 2014 und 2015: Gesamtb…
    Annotated writing performances 2014 and 2015: Overall assessments. Guidelines for the assessment of writing performances (6th and 7th grade)]
  5. Chernikova (2020)
    Facilitating diagnostic competences in higher education – a meta-analysis in medical and …
    Educational Psychology Review  ↗
Show all 65 →
  1. Cizek (2007)
    Standard setting: a guide to establishing and evaluating performance standards on tests
  2. Cohen (1988)
    Statistical power analysis for the behavioral sciences
  3. Council of Europe (2001)
    Common European framework of reference for languages: learning, teaching, assessment
  4. DeLuca (2016)
    Approaches to classroom assessment inventory: a new instrument to support teacher assessm…
    Educational Assessment  ↗
  5. Departement Bildung Kultur und Sport des Kantons Aargau. (2014). Lehrplan für die Volksschule des Kantons Aar…
  6. Departement Bildung Kultur und Sport des Kantons Aargau. (2018). Aargauer Lehrplan Volksschule. Gesamtausgabe…
  7. Deutschschweizer Erziehungsdirektoren-Konferenz. (2015). Lehrplan 21 [Curriculum 21]. 〈https://v-fe.lehrplan.…
  8. Doyé (1977)
    Diagnostischer Leistungstest Englisch 5–6 [Diagnostic achievement test English 5–6]
  9. Dunn (1969)
    Correlation coefficients measured on the same individuals
    Journal of the American Statistical Association  ↗
  10. Eckes (2015)
    Introduction to many-facet rasch measurement: analyzing and evaluating rater-mediated assessments
  11. Edelenbos (2004)
    Teacher assessment: the concept of ‘diagnostic competence
    Language Testing  ↗
  12. Elder (2007)
    Evaluating rater responses to an online training program for L2 writing assessment
    Language Testing  ↗
  13. Fleckenstein (2020)
    Is a long essay always a good essay? The effect of text length on writing assessment
    Frontiers in Psychology  ↗
  14. Gass (2000)
    Stimulated recall methodology in second language research
  15. Grausam, N.C. (2018). Diagnosekompetenz von Lehrpersonen als Voraussetzung individueller Förderung im Bereich…
  16. Griva (2009)
    A study of FL composing process and writing strategies employed by young learners
    Early learning of modern foreign languages. Processes and outcomes
  17. Grotjahn (2017)
    Feedback zu schriftlichen Lernerproduktionen [Feedback on written learner productions]
    Schreibkompetenzen in der Fremdsprache
  18. Grotjahn (2017)
    Kriteriale Evaluation von Schreibkompetenzen [Criterion-based Evaluation of writing compe…
    Schreibkompetenzen in der Fremdsprache. Aufgabengestaltung, kriterienorientierte Bewertung und Feedback
  19. Harsch (2012)
    Adapting CEF-descriptors for rating purposes: validation by a combined rater training and…
  20. Hasselgreen (2012)
    Assessment of young learner literacy linked to the common european framework of reference for languages
  21. Helmke (1987)
    Interactional effects of instructional quality and teacher judgement accuracy on achievement
    Teaching and Teacher Education  ↗
  22. Hemmerich, W. (2017). StatistikGuru: Korrelationen statistisch vergleichen [StatistikGuru: Statistically comp…
  23. Herppich (2018)
    Teachers' assessment competence: integrating knowledge-, process-, and product-oriented a…
    Teaching and Teacher Education  ↗
  24. Hyland (2004)
    Genre and second language writing
  25. Jansen (2024)
    Feedback on teachers’ text assessment: Does it foster assessment accuracy and motivation?
    Zeitschrift für Pädagogische Psychologie  ↗
  26. Jansen (2021)
    Judgment accuracy in experienced versus student teachers: Assessing essays in English as …
    Teaching and Teacher Education  ↗
  27. Jansen (2021)
    Don’t just judge the spelling! The influence of spelling on assessing second-language stu…
    Frontline Learning Research  ↗
  28. Karst, K. (2017). Akkurate Urteile – die Ansätze von Schrader (1989) und McElvany et al. (2009) [Accurate jud…
  29. Keller (2019)
    Can an instructional video increase the quality of English teachers’ assessment of learne…
    Research in Subject-matter Teaching and Learning (RISTAL)
  30. Konrad (2018)
    Assessing writing at lower levels: research findings, task development locally and intern…
    British Council
  31. Kvasova (2014)
    The assessment competence of university foreign language teachers: a Ukrainian perspective
    Language Learning in Higher Education  ↗
  32. Lasagabaster (2003)
    Maturational constraints on foreign language written production. In M. D. p
    Age and the acquisition of English as a foreign language
  33. Lenhard, W., & Lenhard, A. (2016). Berechnung von Effektstärken [Calculation of effect sizes]. Available at 〈…
  34. Lindgren (2013)
    Interactional resources in the letters of young writers in Swedish and English
    Journal of Second Language Writing  ↗
  35. Llach (2007)
    Children's characteristics in vocabulary acquisition and use in the written production
    Revista Espanola Délelőtt Linguistica Aplicada
  36. Lohmann (2024)
    Neural networks or linguistic features? – Comparing different machine-learning approaches…
    International Journal of Artificial Intelligence in Education
  37. Möller (2022)
    Judgment accuracy of German student texts: Do teacher experience and content knowledge matter?
    Teaching and Teacher Education  ↗
  38. Oxford University Press. (2024). Oxford Placement Test 〈https://elt.oup.com/feature/global/oxford-online-plac…
  39. Parr et al. (2010)
    Feedback to writing, assessment for teaching and learning and student progress
    Assessing Writing
  40. Pastore (2023)
    Teacher assessment literacy: a systematic review
    Frontiers in Education  ↗
  41. Patekar (2021)
    A look into the practices and challenges of assessing young EFL learners’ writing in Croatia
    Language Testing  ↗
  42. Praetorius (2017)
    Eine Einführung in das Thema der diagnostischen Kompetenz von Lehrkräften [An introductio…
    Diagnostische Kompetenz von Lehrkräften
  43. Raaen (2012)
    Basic skills: orthography and sentence complexity in written English from grades 7 to 10
    The young language learner. Research-based insights into teaching and learning
  44. Retelsdorf (2016)
    Diagnostik und Leistungsbeurteilung [Diagnostics and performance assessment]
    Basiswissen Lehrerbildung: Schule und Unterricht – Lehren und Lernen
  45. Seker (2018)
    Intervention in teachers’ differential scoring judgments in assessing L2 writing through …
    Studies in Educational Evaluation  ↗
  46. Silver (2004)
    Testing dependent correlations with nonoverlapping variables: a Monte Carlo simulation
    The Journal of Experimental Education  ↗
  47. Spinath (2005)
    Akkuratheit der Einschätzung von Schülermerkmalen durch Lehrer und das Konstrukt der diag…
    Zeitschrift für Pädagogische Psychologie  ↗
  48. Südkamp (2012)
    Accuracy of teachers’ judgments of students’ academic achievement: A meta-analysis
    Journal of Educational Psychology  ↗
  49. Südkamp (2008)
    Der simulierte Klassenraum. Eine experimentelle Untersuchung zur diagnostischen Kompetenz…
    Zeitschrift für Pädagogische Psychologie  ↗
  50. Szabo (2016)
    Collated representative samples of descriptors of language competences developed for young learners aged 11–15 years
  51. Trüb (2022)
    An empirical study of EFL writing at primary school
  52. Trüb (2021)
    Conceptualising and measuring writing in English as a foreign language at primary school
    Current research into young foreign language learnres' literacy skills
  53. UNESCO Institute for Statistics (2012)
    International standard classification of education
    ISCED 2011
  54. Urhahne (2021)
    A review on the accuracy of teacher judgments
    Educational Research Review  ↗
  55. Van Orman (2024)
    Preparing teacher candidates to assess for learning: a systematic review
    Review of Educational Research
  56. Vögelin et al. (2019)
    The influence of lexical features on teacher judgements of ESL argumentative essays
    Assessing Writing
  57. Vogt (2014)
    Assessment literacy of foreign language teachers: findings of a European study
    Language Assessment Quarterly  ↗
  58. Weigle (2002)
    Assessing writing
  59. Wolfe et al. (2016)
    Features of difficult-to-score essays
    Assessing Writing
  60. Zhu (2015)
    Teachers’ judgements of students’ foreign-language achievement
    European Journal of Psychology of Education  ↗
CrossRef global citation count: 1 View in citation network → Build reading path →

Related Articles

  1. Assessing Writing Jan 2026
    How reliable and valid is peer evaluation in adolescents’ L2 argumentative writing? ↗
    Albert W. Li; Steve Graham
    writing pedagogy teacher development argument assessment multilingual writers grammar and mechanics literacy studies
  2. Assessing Writing Jan 2019
    The influence of lexical features on teacher judgements of ESL argumentative essays ↗
    Cristina Vögelin; Thorben Jansen; Stefan D. Keller; Nils Machts; Jens Möller
    teacher development assessment multilingual writers grammar and mechanics race and writing
  3. Journal of Writing Research Jun 2018
    Assessment of L2 Student Writing: Does Teacher Disciplinary Background Matter? ↗
    G. Eckstein; R. Casper; J. Chan; L. Blackwell
    rhetorical criticism teacher development assessment multilingual writers grammar and mechanics
  4. Writing and Pedagogy Jul 2015
    The Role of Information Management in the Assessment of Grammar in L2 Academic Writing ↗
    Heike Neumann
    discourse analysis writing pedagogy teacher development assessment qualitative research multilingual writers grammar and mechanics
  5. Assessing Writing Apr 2026
    How do L2 writing subskills interact hierarchically? Insights from diagnostic classification models ↗
    Farshad Effatpanah; Hamdollah Ravand; Mahmoud Abdi Tabari; Yi-Hsin Chen; Olga Kunina-Habenicht
    assessment multilingual writers grammar and mechanics