A Direct Functional Measure of Text Quality: Did the Reader Understand?

Joachim Grabowski Leibniz University Hannover ; Moti Mathiebe Leibniz University Hannover

Abstract

Assessing text quality as an indication of underlying skills still remains challenging; irrespective of the approach, many studies struggle with reliability or validity problems. If writing is considered problem-solving, a report must make the reader understand the described situation and call for its mental reconstruction. Therefore, text quality may not only comprise linguistic aspects but also the cognitive-functional power of a text. The presented study aims at exploring the functionality of students’ reporting texts in relation to general text-quality measures, using a corpus of accident reports written by German fifth- and ninth-graders (n = 277) prompted by a pictorial stimulus of a bike accident scenario. An online tool was developed in which 277 university students graphically reenacted the situation from one respective text according to the existence, position, and color of the involved elements. Thereafter, the match of the resulting spatial reconstructions with the original situation was assessed by two raters. While most subscales showed sufficiently high interrater reliabilities, the aggregated functionality score (α = .74) had medium-high correlations with other text-quality ratings and was comparably dependent on grade, education level, and linguistic family background. However, the correlational pattern, regression analysis, and factor analysis showed that the functionality score also contributed unique portions of variance to the assessment of writing skill that were not represented by rating measures. Moreover, the direct indication of whether a text allows for the reader’s adequate cognitive representation is evident. Altogether, the approach of indicating text functionality through practical understanding offers a sound, though empirically laborious, alternative for text-quality measurement. Results are discussed with regard to the didactical strategy according to which students can improve their writing when they observe whether others can make use of their texts.

Journal
Written Communication
Published
2024-04-01
DOI
10.1177/07410883231222952
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
Export

Citation Context

Cited by in this index (1)

  1. Written Communication

References (44) · 6 in this index

  1. 10.2753/EUE1056-4934370406
  2. Bildung in deutschland 2016. Ein indikatorengestützter Bericht mit einer analyse zu Bildu…
  3. Bachmann T., Becker-Mrotzek M. (2010). Schreibaufgaben situieren und profilieren. In Pohl T., Steinhoff T. (E…
  4. 10.1111/1467-9817.00175
  5. Assessing Writing
Show all 44 →
  1. CTAP: A web-based tool supporting automatic complexity analysis
  2. 10.1177/001440298204800417
  3. 10.1002/j.2333-8504.1961.tb00286.x
  4. Text—Textsorten—Semantik
  5. 10.3389/fpsyg.2020.562462
  6. Methods in writing process research
  7. Journal of Writing Research
  8. 10.1111/j.1467-9817.2009.01431.x
  9. 10.1037/0022-0663.89.1.170
  10. 10.1177/1948550610368434
  11. Research in the Teaching of English
  12. Cognitive processes in writing
  13. Assessing Writing
  14. 10.1111/jedm.12000
  15. 10.1016/j.lindif.2023.102274
  16. Ke Z., Ng V. (2019). Automated essay scoring: A survey of the state of the art. Proceedings of the 28th Inter…
  17. Sprachförderung und Sprachdiagnostik
  18. 10.2307/2529310
  19. Wortschatz und Schreibkompetenz. Bildungssprachliche Mittel in Schülertexten der Sekundar…
  20. National Assessment Governing Board. (2010). Writing framework for the 2011 national assessment of educationa…
  21. 10.1163/9789004248489_004
  22. Sprachfähigkeiten—besser als ihr Ruf und nötiger denn je!
  23. 10.1007/s11145-008-9124-z
  24. 10.1007/s11145-012-9392-5
  25. The nation’s report card: writing 2002
  26. Journal for Educational Research Online
  27. Didaktik Deutsch
  28. 10.1007/s10462-021-10068-2
  29. 10.4135/9780857021069.n31
  30. Journal of Writing Research
  31. 10.1080/09500690500336932
  32. 10.17239/L1ESLL-2016.16.01.03
  33. 10.1163/9789004248489_002
  34. Plain language: Principles and practice
  35. Utrecht Studies in Language and Communication
  36. 10.1111/jedm.12336
  37. PARE
  38. 10.1163/9789004248489
  39. Assessing Writing