Assessing Writing Oct 2024 Open Access

Validating an integrated reading-into-writing scale with trained university students

Claudia Harsch University of Bremen ; Valeriia Koval University of Bremen ; Paraskevi (Voula) Kanistra ; Ximena Delgado-Osorio DIPF | Leibniz Institute for Research and Information in Education

Abstract

Integrated tasks are often used in higher education (HE) for diagnostic purposes, with increasing popularity in lingua franca contexts, such as German HE, where English-medium courses are gaining ground. In this context, we report the validation of a new rating scale for assessing reading-into-writing tasks. To examine scoring validity, we employed Weir’s (2005) socio-cognitive framework in an explanatory mixed-methods design. We collected 679 integrated performances in four summary and opinion tasks, which were rated by six trained student raters. They are to become writing tutors for first-year students. We utilized a many-facet Rasch model to investigate rater severity, reliability, consistency, and scale functioning. Using thematic analysis, we analyzed think-aloud protocols, retrospective and focus group interviews with the raters. Findings showed that the rating scale overall functions as intended and is perceived by the raters as valid operationalization of the integrated construct. FACETS analyses revealed reasonable reliabilities, yet exposed local issues with certain criteria and band levels. This is corroborated by the challenges reported by the raters, which they mainly attributed to the complexities inherent in such an assessment. Applying Weir’s (2005) framework in a mixed-methods approach facilitated the interpretation of the quantitative findings and yielded insights into potential validity threads. • FACET analyses show reasonable reliabilities and scale functioning. • Mixed-methods approach facilitates interpreting the quantitative findings. • Raters perceive rating scale as valid operationalization of integrated construct. • Applying Weir’s socio-cognitive framework reveals potential validity threads. • Raters attribute challenges to the complexities inherent in integrated writing.

Journal
Assessing Writing
Published
2024-10-01
DOI
10.1016/j.asw.2024.100894
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
assessment peer tutoring qualitative research
Export
BibTeX RIS

Citation Context

Cited by in this index (1)

  1. Shackleton et al. (2026)
    Developing a rating scale for written intralinguistic mediation in a local context
    Assessing Writing

References (59) · 10 in this index

  1. Attali (2015)
    A comparison of newly-trained and experienced raters on a standardized writing assessment
    Language Testing  ↗
  2. Barkaoui (2010)
    Do ESL essay raters' evaluation criteria change with experience? A mixed-methods, cross-s…
    TESOL Quarterly  ↗
  3. Barkaoui (2010)
    Variability in ESL essay rating processes: The role of the rating scale and rater experience
    Language Assessment Quarterly  ↗
  4. Barkaoui (2011)
    Think-aloud protocols in research on essay rating: An empirical study of their veridicali…
    Language Testing  ↗
  5. Braun (2006)
    Using thematic analysis in psychology
    Qualitative Research in Psychology  ↗
Show all 59 →
  1. Chan et al. (2015)
    Developing rubrics to assess the reading-into-writing skills: A case study
    Assessing Writing
  2. Chan (2023)
    Towards more valid scoring criteria for integrated reading-writing and listening-writing …
    Language Testing  ↗
  3. Charters (2003)
    The use of think-aloud methods in qualitative research. An introduction to think-aloud methods
    Brock Education
  4. Cheong (2019)
    Effects of intertextual processing on L2 integrated writing
    Journal of Second Language Writing  ↗
  5. Collins (2014)
    Cognitive interviewing practice
  6. Cohen (2011)
    Research Methods in Education
  7. Council of Europe (2001). Common European Framework of Reference for Languages: Learning, teaching, assessmen…
  8. Creswell (2018)
    Designing and conducting mixed methods research
  9. Crossley et al. (2011)
    The Development of Writing Proficiency as a Function of Grade Level: A Linguist…
    Written Communication
  10. Cumming (2013)
    Assessing Integrated Writing Tasks for Academic Purposes: Promises and Perils
    Language Assessment Quarterly  ↗
  11. Cumming (2002)
    Decision making while rating ESL/EFL writing tasks: A descriptive framework
    Modern Language Journal  ↗
  12. Cumming (2016)
    Students’ writing from sources for academic purposes: A synthesis of recent research
    Journal of English for Academic Purposes  ↗
  13. Deygers (2015)
    Determining the scoring validity of a co-constructed CEFR-based rating scale
    Language Testing  ↗
  14. Eckes (2012)
    Operational rater types in writing assessment: Linking rater cognition to rater behavior
    Language Assessment Quarterly  ↗
  15. Eckes (2015)
    Introduction to Many-facet Rasch measurement: Analysing and evaluating rater-mediated assessments (2nd Revised and updated edition
  16. Frey (1991)
    The group interview in social research
    The Social Science Journal  ↗
  17. Gebril et al. (2014)
    Assembling validity evidence for assessing academic writing: Rater reactions to…
    Assessing Writing
  18. Harsch, C., Koval, V., Delgado-Osorio, X. & Hartig, J. (2024). Usability of CEFR Companion Volume scales for …
     ↗
  19. Harsch et al. (2012)
    Adapting CEF-descriptors for rating purposes: Validation by a combined rater tr…
    Assessing Writing
  20. Jia (2023)
    Rater cognitive processes in integrated writing tasks: From the perspective of problem-solving
    Lang Test Asia  ↗
  21. KMK, Ed. (2014). Bildungsstandards für die fortgeführte Fremdsprache (Englisch/Französisch) für die Allgemein…
  22. Keck (2006)
    The use of paraphrase in summary writing: A comparison of L1 and L2 writers
    Journal of Second Language Writing  ↗
  23. Knoch (2011)
    Rating scales for diagnostic assessment of writing: What should they look like …
    Assessing Writing
  24. Knoch (2018)
    Validation of rating processes within an argument-based framework
    Language Testing  ↗
  25. Knoch (2021)
    Revisiting rating scale development for rater-mediated language performance assessments: …
    Language Testing  ↗
  26. Knoch et al. (2013)
    A closer look at integrated writing tasks: Towards a more focussed definition f…
    Assessing Writing
  27. Kukartz (2019)
    Analyzing Qualitative Data with MAXQDA
  28. Lestari (2023)
    Operationalizing the reading-into-writing construct in analytic rating scales: Effects of…
    Language Testing  ↗
  29. Li (2014)
    The role of reading and writing in summarization as an integrated task
    Language Testing in Asia  ↗
  30. Li (2021)
    Development and validation of a rating scale for summarization as an integrated task
    Asian-Pacific Journal of Second and Foreign Language Education  ↗
  31. Lim (2011)
    The development and maintenance of rating quality in performance writing assessment: A lo…
    Language Testing  ↗
  32. Linacre (2004)
    Optimizing rating scale category effectiveness
    Introduction to Rasch measurement. Theory, models, and applications
  33. Linacre (2023)
    A user's guide to FACETS Rasch-Model computer programs
    Program Manual
  34. Lumley (2005)
    Assessing Second Language Writing: The Rater's Perspective
  35. Motz (2005)
    Englisch oder Deutsch in Internationalen Studiengängen? [English or German in international degree programs?
  36. Murray (2016)
    Standards of English in higher education: issues, challenges and strategies
  37. Myford (2012)
    Rater cognition research: Some possible directions for the future
    Educational Measurement: Issues and Practice  ↗
  38. Myford (2004)
    Detecting and measuring rater effects using Many-Facet Rasch Measurement: Part I. In
    Introduction to Rasch measurement: Theory, models, and applications
  39. Myford (2004)
    Detecting and measuring rater effects using many-facet Rasch measurement: Part II. In
    Introduction to Rasch Measurement: Theory, models, and applications
  40. Ono (2019)
    Holistic and analytic assessments of the TOEFL iBT® integrated writing task.
    JLTA (Japan Language Testing Association) Journal
  41. Pearson Education (2015). Global Scale of English Learning Objectives for Academic English. Pearson Education…
  42. Plakans (2015)
    Assessment myths: Applying second language research to classroom teaching
  43. Ringwald (2018)
    Und dann kommt das große Erwachen an der Uni“ – Eine explorative Bedarfsanalyse
    Fremdsprachen und Hochschule
  44. Rölke (2012)
    The ItemBuilder: A graphical authoring system for complex item development
    Proceedings of E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education
  45. Rupp, A.A., Vock, M., Harsch, C., & Köller, O. (2008). Developing standards-based assessment tasks for Englis…
  46. Shi (2004)
    Textual Borrowing in Second-Language Writing
    Written Communication
  47. Shin (2015)
    What accounts for integrated reading-to-write task scores?
    Language Testing  ↗
  48. Sormunen, E., Heinstrom, J., Romu, L. & Turunen, R. (2012). A Method for the Analysis of Information Use in S…
  49. Spivey (1989)
    Readers as writers composing from sources
    Reading Research Quarterly  ↗
  50. Uludag et al. (2022)
    Validating a rubric for assessing integrated writing in an EAP context
    Assessing Writing
  51. Wang et al. (2017)
    Evaluating rater accuracy and perception for integrated writing assessments usi…
    Assessing Writing
  52. Weir (2005)
    Language testing and validation: an evidence-based approach
  53. Xie (2023)
    Assessing source use: Summary vs. reading-to-write argumentative essay
    Assessing Writing
  54. V.E.R.B.I. Software 2021, MAXQDA 2022, computer program, VERBI Software, Berlin.
CrossRef global citation count: 2 View in citation network → Build reading path →

Related Articles

  1. Journal of Academic Writing Feb 2025
    Generative AI for Academic Writing: Case Studies Beyond Simple Chatbot Interactions ↗
    Thomas Lancaster
    teacher development assessment peer tutoring qualitative research artificial intelligence
  2. Writing Center Journal 2014
    Questioning in Writing Center Conferences ↗
    Isabelle Thompson; Jo Mackiewicz
    rhetorical criticism discourse analysis teacher development argument collaborative writing assessment writing centers peer tutoring qualitative research
  3. Composition Forum 2012
    The Peer-Interactive Writing Center at the University of New Mexico ↗
    Daniel Sanford
    collaborative writing assessment writing centers peer tutoring qualitative research
  4. IEEE Transactions on Professional Communication Mar 2026
    US Hospital Educators' Technology Needs: A Qualitative Study for Developing Action-Oriented Technology ↗
    Margaret Webb; Sweta Baniya; Alexa Smith; Nadra Rasberry; Ihudiya Ogbonnaya-Ogburu
    modern rhetorical theory teacher development assessment professional writing qualitative research
  5. IEEE Transactions on Professional Communication Mar 2026
    Surveys as UXR: Using Design Thinking to Shape a Survey-Based UX Assessment for Rural Audiences ↗
    Kaitlin J. Coyle; Derek G. Ross
    assessment technical communication qualitative research literacy studies