Assessing Writing Jul 2023 Open Access

Beyond literacy and competency – The effects of raters’ perceived uncertainty on assessment of writing

Mari Honko University of Jyväskylä ; Reeta Neittaanmäki University of Jyväskylä ; Scott Jarvis Northern Arizona University ; Ari Huhta University of Jyväskylä

Abstract

This study investigated how common raters’ experiences of uncertainty in high-stakes testing are before, during, and after the rating of writing performances, what these feelings of uncertainty are, and what reasons might underlie such feelings. We also examined if uncertainty was related to raters’ rating experience or to the quality of their ratings. The data were gathered from the writing raters (n = 23) in the Finnish National Certificates of Proficiency, a standardized Finnish high-stakes language examination. The data comprise 12,118 ratings as well as raters’ survey responses and notes during rating sessions. The responses were analyzed by using thematic content analysis and the ratings by descriptive statistics and Many-Facets Rasch analyses. The results show that uncertainty is variable and individual, and that even highly experienced raters can feel unsure about (some of) their ratings. However, uncertainty was not related to rating quality (consistency or severity/leniency). Nor did uncertainty diminish with growing experience. Uncertainty during actual ratings was typically associated with the characteristics of the rated performances but also with other, more general and rater-related or situational factors. Other reasons external to the rating session were also identified for uncertainty, such as those related to the raters themselves. An analysis of the double-rated performances shows that although similar performance-related reasons seemed to cause uncertainty for different raters, their uncertainty was largely associated with different test-takers’ performances. While uncertainty can be seen as a natural part of holistic ratings in high-stakes tests, the study shows that even if uncertainty is not associated with the quality of ratings, we should constantly seek ways to address uncertainty in language testing, for example by developing rating scales and rater training. This may make raters’ work easier and less burdensome.

Journal
Assessing Writing
Published
2023-07-01
DOI
10.1016/j.asw.2023.100768
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
assessment literacy studies
Export
BibTeX RIS

Citation Context

Cited by in this index (0)

No articles in this index cite this work.

References (46)

  1. Ahola (2022)
    Rimaa hipoen selviää tilanteesta: yleisten kielitutkintojen suomen kielen arvioijien käsityksiä kielitaidon arvioinnista ja suullisesta kielitaidosta
  2. Ahola (2016)
    Puhetta arvioinnista: yleisten kielitutkintojen arvioijien käsityksiä arvioinnista
    Kielitaidon arviointitutkimus 2000-luvun Suomessa
  3. Alderson (1995)
    Language test construction and evaluation
  4. Anderson (2019)
    The relationship between uncertainty and affect
    Frontiers in Psychology  ↗
  5. Aryadoust (2021)
    A comprehensive review of Rasch measurement in language assessment: Recommendations and g…
    Language Testing  ↗
Show all 46 →
  1. Barkaoui (2011)
    Think-aloud protocols in research on essay rating: An empirical study of their veridicali…
    Language Testing  ↗
  2. Bejar (2012)
    Rater cognition: Implications for validity
    Educational Measurement: Issues and Practice  ↗
  3. Bhise (2018)
    Defining and measuring diagnostic uncertainty in medicine: A systematic review
    Journal of General Internal Medicine  ↗
  4. Bogorevich (2018)
    Native and non-native raters of L2 speaking performance: Accent familiarity and cognitive processes. (Ph.D. dissertation)
  5. Borders (2021)
    Objective and subjective clinical swallowing outcomes via telehealth: Reliability in outp…
    American Journal of Speech - Language Pathology  ↗
  6. Bosshardt (2016)
    Measuring stuttering in preschool-aged children across different languages: An internatio…
    Folia Phoniatrica Et Logopaedica  ↗
  7. Braun (2006)
    Using thematic analysis in psychology
    Qualitative Research in Psychology  ↗
  8. Canino (2001)
    Posed emotional expression in brain-damaged patients across three channels of communication. (Ph.D. dissertation)
  9. Council of Europe (2001)
    The common european framework of reference for languages: Learning, teaching, assessment
  10. Cumming (2002)
    Decision making while rating ESL/EFL writing tasks: A descriptive framework
    Modern Language Journal  ↗
  11. Davies (1999)
    Dictionary of language testing
  12. Eckes (2005)
    Examining rater effects in TestDaF writing and speaking performance assessments: A many-f…
    Language Assessment Quarterly  ↗
  13. Eckes (2008)
    Rater types in writing performance assessments: A classification approach to rater variability
    Language Testing  ↗
  14. Eckes (2011)
    Introduction to many-facets Rasch measurement: Analyzing and evaluating rater-mediated assessments
  15. Eckes (2017)
    Rater effects: Advances in item response modeling of human ratings—Part I [Guest editorial]
    Psychological Test and Assessment Modeling
  16. Elder (2005)
    Individual feedback to enhance rater training: Does it work
    Language Assessment Quarterly  ↗
  17. Faraji-Rad (2013)
    Uncertainty increases the reliance on affect in decisions
    Journal of Consumer Research
  18. Fitzpatrick (2020)
    Word association research and the L2 lexicon
    Language Teaching  ↗
  19. FNAE (2011)
    Yleisten kielitutkintojen perusteet
  20. Gorsuch (2018)
    Second language testing for student evaluation and classroom research
  21. Honko (2023)
    Instrument to study rater behaviour: Rater uncertainty and its impact on the quality of a…
    University of Jyväskylä
  22. Hsu (2012)
    The impact of world englishes on language assessment: Perception, rating behavior and challenges. (Unpublished doctoral dissertation)
  23. Huhta (2014)
    Assessing learners’ writing skills in a SLA study: Validating the rating process across t…
    Language Testing  ↗
  24. Katz (2002)
    The silent world of doctor and patient
  25. Kuiken (2014)
    Rating written performance: What do raters do and why?
    Language Testing  ↗
  26. Linacre (1994)
    Many-facet Rasch measurement.
  27. Linacre (2021)
    Facets computer program for many-facet Rasch measurement, version 3.83.5
  28. Lumley (2005)
    Assessing second language writing: The rater’s perspective
  29. Milburn (1976)
    Decision-making perspectives from psychology: Dealing with risk and uncertainty
    American Behavioral Scientist  ↗
  30. Neittaanmäki (2014)
    Yleisten kielitutkintojen osallistujat taustatietojen valossa
    Yleiset kielitutkinnot 20 vuotta
  31. Pill (2021)
    Raters. Behavior and training
    The Routledge Handbook of second language acquisition and language testing, Chapter 13.New
  32. Posner (2008)
    How judges think
  33. Reed (2001)
    Revisiting raters and rating in oral language assessment
  34. Schaefer (2008)
    Rater bias patterns in an EFL writing assessment
  35. Tarnanen (2002)
    Arvioija valokeilassa – Suomi toisena kielenä -kirjoittamisen arviointia
    Jyväskylän yliopisto: Centre for Applied Language Studies
  36. Tarnanen (2014)
    Arvioija taidon arvottajana
    Yleiset Kielitutkinnot 20 vuotta.
  37. Van Moere (2013)
    Raters and ratings
    The companion to language assessment
  38. Vaughan (1991)
    Holistic assessment: What goes on in the rater’s mind?
    Assessing second language writing in academic contexts
  39. Wind (2019)
    Exploring patterns of principal judgments in teacher evaluation related to reported gende…
    Studies in Educational Evaluation  ↗
  40. Wind (2021)
    Principals’ severity affects teacher evaluation: statistical adjustments mitigate effects
    School Effectiveness and School Improvement  ↗
  41. Youn (2018)
    Rater variability across examinees and rating criteria in paired speaking assessment
    Papers in Language Testing and Assessment
CrossRef global citation count: 2 View in citation network → Build reading path →

Related Articles

  1. IEEE Transactions on Professional Communication Mar 2026
    Surveys as UXR: Using Design Thinking to Shape a Survey-Based UX Assessment for Rural Audiences ↗
    Kaitlin J. Coyle; Derek G. Ross
    assessment technical communication qualitative research literacy studies
  2. Journal of Writing Analytics Jan 2026
    A Tribute to Robert J. Mislevy Part 1: Mapping the Skills of Tomorrow--Principled Assessment of Literacy and Numeracy Skills Embedded in U.S. Workplace Contexts ↗
    Maria Elena Oliveri; Aria Immanuel
    assessment literacy studies
  3. Assessing Writing Jan 2026
    Generative artificial intelligence for automated essay scoring: Exploring teacher agency through an ecological perspective ↗
    Jessie S. Barrot
    writing pedagogy teacher development assessment artificial intelligence literacy studies
  4. Assessing Writing Jan 2026
    How reliable and valid is peer evaluation in adolescents’ L2 argumentative writing? ↗
    Albert W. Li; Steve Graham
    writing pedagogy teacher development argument assessment multilingual writers grammar and mechanics literacy studies
  5. Business and Professional Communication Quarterly Dec 2025
    Rethinking Teacher-Student Communication in the AI Era ↗
    Huu Phuoc Tran
    writing pedagogy teacher development assessment digital rhetoric artificial intelligence literacy studies