Assessing Writing Apr 2025 Open Access

Designing a rating scale for an integrated reading-writing test: A needs-oriented approach

Aynur Ismayilli Karakoҫ Cambridge University Press ; Peter Gu ; Rachael Ruegg Victoria University of Wellington

Abstract

To meet the current trends in higher education, there is accountability on EAP programmes to prepare and assess students’ access to higher education. Thus, multimodal tasks including integrated writing (IW) assessments have seen a resurgence because they arguably closely mirror academic writing. However, test practicality constraints and variability in the use and format of these assessments mean rating scales often fall short in substantiating the central claims of IW assessment. We developed an integrated reading-writing scale taking into account reading-writing requirements and empirical research on IW tests designed to assess readiness for first-year humanities and social science courses. We approached test development as part of the ongoing validation efforts, detailing the considerations involved in the scale development process. We argue that alignment with academic writing requirements should guide the development of IW tests, thereby acknowledging and comprehending nuances of academic writing. The paper demonstrates considerations and decisions in scale design as the validation process from the start, which is a reminder that assessment is not just a quantitative exercise but a multifaceted process. • The design of a rating scale for first-year undergraduate academic writing is detailed. • Emphasis is placed on the role of reading in integrated writing scales. • Academic argumentation, rather than solely source-use mechanics, is considered. • Implications for construct operationalisation in academic evaluations are offered.

Journal
Assessing Writing
Published
2025-04-01
DOI
10.1016/j.asw.2025.100918
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
first-year composition argument assessment empirical research multimodality literacy studies
Export
BibTeX RIS

Citation Context

Cited by in this index (1)

  1. Shackleton et al. (2026)
    Developing a rating scale for written intralinguistic mediation in a local context
    Assessing Writing

References (57) · 13 in this index

  1. Bachman (1991)
    What does language testing have to offer?
    TESOL Quarterly  ↗
  2. Barkaoui (2010)
    Do ESL essay raters' evaluation criteria change with experience? A mixed-methods, cross-s…
    TESOL Quarterly  ↗
  3. Bruce (2015)
    Opposing tensions of local and international standards for EAP writing programmes: Who ar…
    Journal of English for Academic Purposes  ↗
  4. Chan et al. (2015)
    Developing rubrics to assess the reading-into-writing skills: A case study
    Assessing Writing
  5. Chan (2023)
    Towards more valid scoring criteria for integrated reading-writing and listening-writing …
    Language Testing  ↗
Show all 57 →
  1. Chan et al. (2022)
    Integrated writing and its correlates: A meta-analysis
    Assessing Writing
  2. Crusan et al. (2016)
    Writing assessment literacy: Surveying second language teachers’ knowledge, bel…
    Assessing Writing
  3. Cumming (2021)
    Assessing academic writing
    Assessing English for Higher Education Admissions
  4. Cumming et al. (2005)
    Differences in written discourse in independent and integrated prototype tasks …
    Assessing Writing
  5. Delaney (2008)
    Investigating the reading-to-write construct
    Journal of English for academic purposes  ↗
  6. Diagnostic English Language Needs Assessment (2023). Handbook. University of Auckland. 〈https://www.auckland.…
  7. Dumitru (2019)
    Creating meaning. The importance of Arts, Humanities and Culture for critical thinking de…
    Studies in Higher Education  ↗
  8. Fulcher (2007)
    Language testing and assessment: An advance resource book
  9. Gebril (2009)
    Investigating source use, discourse features, and process in integrated writing tests
    Spaan Fellow Working Papers in Second Oregon Foreign Language Assessment
  10. Gebril (2013)
    Toward a transparent construct of reading-to-write tasks: The interface between discourse…
    Language Assessment Quarterly  ↗
  11. Gebril et al. (2014)
    Assembling validity evidence for assessing academic writing: Rater reactions to…
    Assessing Writing
  12. Hamp-Lyons (1991)
    Assessing second language writing in academic contexts
  13. Homayounzadeh et al. (2019)
    Investigating the effect of source characteristics on task comparability in int…
    Assessing Writing
  14. Hubley (2011)
    Validity and the consequences of test interpretation and use
    Social Indicators Research  ↗
  15. Ismayilli Karakoç (2022)
    Beyond comprehension: Reading requirements in first-year undergraduate courses
    Journal of English for Academic Purposes  ↗
  16. Ismayilli Karakoҫ (2022)
    First-year writing in the humanities and social sciences: Requirements, expectations, and…
    Language Teaching Research
  17. Johns (2021)
    Situated argumentation as a rhetorical act
    Argumentative writing in a second language: Perspectives on research and pedagogy
  18. Jonsson (2007)
    The use of scoring rubrics: Reliability, validity and educational consequences
    Educational Research Review  ↗
  19. Kane (2006)
    Validation
    Educational measurement
  20. Keck (2014)
    Copying, paraphrasing, and academic writing development: A re-examination of L1 and L2 su…
    Journal of Second Language Writing  ↗
  21. Knoch (2009)
    Diagnostic writing assessment: The development and validation of a rating scale
  22. Knoch (2021)
    Scoring second language spoken and written performance: Issues, options and directions
  23. Knoch (2019)
    Assessing English for professional purposes
  24. Knoch et al. (2013)
    A closer look at integrated writing tasks: Towards a more focussed definition f…
    Assessing Writing
  25. Leki (1997)
    Completely different worlds: EAP and the writing experiences of ESL students in universit…
    TESOL Quarterly  ↗
  26. Lestari (2023)
    Operationalizing the reading-into-writing construct in analytic rating scales: Effects of…
    Language Testing  ↗
  27. Li (2012)
    Two first-year students’ strategies for writing from sources: Patchwriting or plagiarism?
    Journal of Second Language Writing  ↗
  28. Linacre, M. (2012). Many-facet Rasch measurement: FACETS tutorial. 〈www.winsteps.com/facetman/tutorialpdfs.htm〉.
  29. Lumley (2002)
    Assessment criteria in a large-scale writing test: What do they really mean to the raters?
    Language Testing  ↗
  30. Marsden (2024)
    Changing the narrative: Valuing Arts and Humanities degrees
    The Learned Society of Wales
  31. Moore (2005)
    Dimensions of difference: a comparison of university writing and IELTS writing
    Journal of English for Academic Purposes  ↗
  32. Ohta et al. (2018)
    Integrated writing scores based on holistic and multi-trait scales: A generaliz…
    Assessing Writing
  33. Payant (2019)
    Predicting integrated writing task performance: Source comprehension, prewriting planning…
    Journal of English for Academic Purposes  ↗
  34. Plakans (2009)
    Discourse synthesis in integrated second language writing assessment
    Language Testing  ↗
  35. Plakans (2021)
    Writing integrated tasks
    The Routledge handbook of language testing
  36. Plakans et al. (2012)
    A close investigation into source use in integrated second language writing tasks
    Assessing Writing
  37. Plakans (2013)
    Using multiple texts in an IW assessment: Source text use as a predictor of score
    Journal of Second Language Writing  ↗
  38. Plakans (2015)
    Assessment myths: Applying second language research to classroom teaching
  39. Plakans et al. (2017)
    Exploring the relationship of organization and connection with scores in integr…
    Assessing Writing
  40. Plakans (2019)
    Shaping a score: Complexity, accuracy, and fluency in integrated writing performances
    Language Testing  ↗
  41. Shaw (2013)
    Source use in academic writing: An introduction to the special issue
    Journal of English for Academic Purposes  ↗
  42. Shin (2015)
    What accounts for integrated reading-to-write task scores?
    Language Testing  ↗
  43. Spivey (1997)
    The constructive metaphor
  44. Uludag et al. (2022)
    Validating a rubric for assessing integrated writing in an EAP context
    Assessing Writing
  45. Watanabe, Y. (2001). Read-to-write tasks for the assessment of second language academic writing skills: Inves…
  46. Weigle (2004)
    Integrating reading and writing in a competency test for non-native speakers of…
    Assessing Writing
  47. Weigle (2012)
    Source text borrowing in an integrated reading/writing assessment
    Journal of Second Language Writing  ↗
  48. Weigle (2013)
    Exploring Reading Processes in an Academic Reading Test Using Short-Answer Questions
    Language Assessment Quarterly  ↗
  49. Weir (2005)
    Language testing and validation: An evidence-based approach
  50. Xie (2023)
    Assessing source use: Summary vs. reading-to-write argumentative essay
    Assessing Writing
  51. Yang (2012)
    Second language writers’ strategy use and performance on an integrated reading‐listening‐…
    TESOL Quarterly  ↗
  52. Yu (2013)
    From integrative to integrated language assessment: Are we there yet?
    Language Assessment Quarterly  ↗
CrossRef global citation count: 1 View in citation network → Build reading path →

Related Articles

  1. Pedagogy Oct 2024
    Contributors ↗
    rhetorical criticism first-year composition writing pedagogy basic writing writing across the curriculum graduate education two-year college service learning teacher development revision argument collaborative writing assessment writing program administration multimodality multilingual writers literacy studies race and writing disability studies community literacy editorial matter
  2. Research in the Teaching of English May 2025
    Argument as Architecture: Constructing an Alternative K–12 Writing Paradigm for Collective Civic Futures ↗
    Nicole Mirra; Kia Turner; Antero Garcia
    discourse analysis writing pedagogy argument assessment multimodality literacy studies
  3. Pedagogy Apr 2024
    To Reimagine . . . To Start Again ↗
    Zachary C. Beare; Jessica Masterson; Shari J. Stenberg
    rhetorical criticism first-year composition writing pedagogy graduate education teacher development argument assessment writing program administration professional writing labor and working conditions multimodality race and writing gender and writing affect and writing editorial matter
  4. Double Helix Jan 2024
    Challenging Assessment Practices, and the Need for Multimodal Applications to Service Learning in First-Year Composition ↗
    Kimberly A. Bain
    first-year composition writing pedagogy service learning revision argument assessment multimodality artificial intelligence editorial matter
  5. Pedagogy Oct 2023
    Contributors ↗
    rhetorical criticism first-year composition writing pedagogy writing across the curriculum two-year college teacher development collaborative writing assessment writing centers qualitative research multimodality literacy studies race and writing gender and writing disability studies affect and writing literary studies book reviews editorial matter