Assessing Writing Oct 2020 Open Access

TOEIC® Writing test scores as indicators of the functional adequacy of writing in the international workplace: Evaluation by linguistic laypersons

Jonathan Schmidgall Educational Testing Service ; Donald E. Powers Educational Testing Service

Abstract

This study examines the extent to which TOEIC Writing test scores relate to an external criterion: evaluations by linguistic laypersons of the functional adequacy of writing in the international workplace. Test-taker responses to two representative tasks from the TOEIC Writing test (e-mail requests, opinion surveys) were adapted for workplace role-play scenarios that laypersons read and evaluated in an online survey. After reading each role-play scenario, laypersons evaluated the text produced by their imagined interlocutor using functional adequacy scale items (comprehensibility, content adequacy, effectiveness, support and coherence). Overall functional adequacy evaluations were obtained by averaging the ratings for each of the two tasks. Layperson ratings of functional adequacy were strongly correlated with TOEIC Writing test scores (r = 0.76). Results suggested that test-takers’ writing performance is likely to be perceived as functionally adequate for test scores at which important decisions are typically made. Study results are discussed in terms of their implications for claims about the generalizability of TOEIC Writing test score interpretations with respect to those made in the international workplace, as well as the potential benefits, challenges, and limitations involved in this approach to validation.

Journal
Assessing Writing
Published
2020-10-01
DOI
10.1016/j.asw.2020.100492
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
assessment digital rhetoric
Export
BibTeX RIS

Citation Context

Cited by in this index (0)

No articles in this index cite this work.

References (46) · 1 in this index

  1. American Educational Research Association (2014)
    Standards for educational and psychological testing
  2. Bachman (2007)
    What is the construct? The dialectic of abilities and contexts in defining constructs in …
    Language testing reconsidered
  3. Bachman (2010)
    Language assessment in practice
  4. Barnwell (1989)
    ‘Naïve’ native speakers and judgments of oral proficiency in Spanish
    Language Testing  ↗
  5. Bayliss (2004)
    The link between academic success and L2 proficiency in the context of two professional p…
    The Canadian Modern Language Review  ↗
Show all 46 →
  1. Brennan (2001)
    Generalizability theory
  2. Bridgeman (2016)
    Predicting grades from an English language assessment: The importance of peeling the onion
    Language Testing  ↗
  3. Bridgeman (2011)
    TOEFL iBT speaking test scores as indicators of oral communicative language proficiency
    Language Testing  ↗
  4. Cardinet (2010)
    Applying generalizability theory using Edu-G
  5. (2008)
    Building a validity argument for the Test of English as a Foreign Language
  6. Davies (2005)
    Validity and validation in language testing
    Handbook of research in second language teaching and learning
  7. de Jong (2012)
    Facets of speaking proficiency
    Studies in Second Language Acquisition  ↗
  8. DeNisi (2017)
    Performance appraisal and performance management: 100 years of progress?
    The Journal of Applied Psychology  ↗
  9. Educational Testing Service (2019)
    Score user guide: TOEIC Speaking and Writing tests
  10. Elder (2016)
    The hunt for “indigenous criteria” in assessing communication in the physiotherapy workplace
    Language Testing  ↗
  11. Elder (2007)
    The role of language proficiency in academic success: Perspectives from a New Zealand uni…
    Melbourne Papers in Language Testing
  12. Elder (2017)
    Interrogating the construct of communicative competence in language assessment contexts: …
    Language & Communication  ↗
  13. Everson (2010)
    How ETS scores the TOEIC Speaking and Writing test responses
    TOEIC compendium
  14. Fayer (1987)
    Native and nonnative judgments of intelligibility and irritation
    Language Learning  ↗
  15. Hines (2010)
    Evidence-centered design: The TOEIC Speaking and Writing tests
    TOEIC compendium
  16. Hymes (1972)
    On communicative competence
    Sociolinguistics
  17. Im (2019)
    Critical review of validation models and practices in language testing: Their limitations…
    Language Testing in Asia
  18. Jenkins (2014)
    English as a lingua franca
    The companion to language assessment
  19. Kane (2013)
    Validating the interpretation and use of test scores
    Journal of Educational Measurement  ↗
  20. Kankaanranta (2010)
    “English? – Oh, it’s just work!”: A study of ELF users’ perceptions
    English for Specific Purposes  ↗
  21. Knoch (2018)
    Validation of rating processes within an argument-based framework
    Language Testing  ↗
  22. Knoch (2016)
    Language assessment for the workplace
    Handbook of second language assessment
  23. Knoch (2020)
    Assessing English for professional purposes
  24. Kuiken (2017)
    Functional adequacy in L2 writing: Towards a new rating scale
    Language Testing  ↗
  25. Kuiken (2018)
    Assessing functional adequacy of L2 performance in a task-based approach
    Task-based approaches to teaching and assessing pragmatics
  26. McNamara (2004)
    Language testing
    The handbook of applied linguistics
  27. Mislevy (2002)
    A brief introduction to evidence-centered design. (Research Report No. RR-03-16)
  28. Munro (1998)
    The effects of speaking rate on listener evaluations of native and foreign-accented speech
    Language Learning  ↗
  29. Powers (1994)
    Direct assessment, direct validation? An example from the assessment of writing
    Educational Assessment  ↗
  30. Powers (1999)
    Validating the revised Test of Spoken English against a criterion of communicative success
    Language Testing  ↗
  31. Sato (2014)
    Linguistic laypersons’ perspective on second language oral communication ability (unpublished doctoral dissertation)
  32. Sato (2018)
    The gap between communicative ability measurements: General-purpose English speaking test…
    Papers in Language Testing and Assessment
  33. Sato (2018)
    What counts in second language oral communication ability? The perspective of linguistic …
    Applied Linguistics
  34. Schmidgall (2013)
    Modeling speaker proficiency, comprehensibility, and perceived competence in a language use domain (Unpublished doctoral dissertation)
  35. Schmidgall (2020)
    Predicting communicative effectiveness in the international workplace: Support for TOEIC …
    Language Testing
  36. Schmidgall (2020)
    Validation of language assessments
    The concise encyclopedia of applied linguistics
  37. Takala (2012)
    The landscape of language testing and assessment in Europe: Developments and challenges
    Research Papers in Language Teaching and Learning
  38. Tannenbaum (2013)
    Mapping TOEIC and TOEIC Bridge® test scores to the Common European framework of reference
    TOEIC compendium
  39. White (2019)
    (Re)visiting twenty-five years of writing assessment
    Assessing Writing
  40. Willingham (1980)
    The criterion problem: What measure of success in graduate education?
    Applied Psychological Measurement  ↗
  41. Yan (2018)
    Listeners and raters: Similarities and differences in evaluation of accented speech
    Assessment in second language pronunciation
CrossRef global citation count: 7 View in citation network → Build reading path →

Related Articles

  1. Written Communication Mar 2026
    Time, Space, and Tools: A Materio-cognitive Model of Digital Writing Process Development ↗
    Matthew Overstreet; Silvia Vaccino-Salvadore; Diana Akhmedjanova
    writing pedagogy assessment digital rhetoric
  2. Assessing Writing Jan 2026
    The effects of online resource use on L2 learners’ computer-mediated writing processes and written products ↗
    Honglan Wang; Jookyoung Jung
    revision argument assessment digital rhetoric multilingual writers grammar and mechanics affect and writing
  3. Business and Professional Communication Quarterly Dec 2025
    Rethinking Teacher-Student Communication in the AI Era ↗
    Huu Phuoc Tran
    writing pedagogy teacher development assessment digital rhetoric artificial intelligence literacy studies
  4. Res Rhetorica Mar 2025
    Persuasiveness of inclusive language in organizational communication ↗
    Jacek Wasilewski; Agata Kostrzewa
    collaborative writing assessment professional writing digital rhetoric book reviews
  5. Business and Professional Communication Quarterly Sep 2022
    Making Meaningful Connections and Learning Workplace-Like Business Writing Through LinkedIn and Blogging ↗
    Daneshwar Sharma
    genre theory assessment professional writing digital rhetoric