Assessing Writing Jan 2020 Open Access

Linking TOEFL iBT® writing rubrics to CEFR levels: Cut scores and validity evidence from a standard setting study

Johanna Fleckenstein FHNW University of Applied Sciences and Arts ; Stefan Keller FHNW University of Applied Sciences and Arts ; Maleika Krüger FHNW University of Applied Sciences and Arts ; Richard J. Tannenbaum Educational Testing Service ; Olaf Köller Leibniz Institute for Science and Mathematics Education

Abstract

English writing is a key competence for higher education success. However, research on the assessment of writing skills in English as a foreign language in European upper secondary education (i.e. beyond year 9) remains scarce. The Common European Framework of Reference (CEFR) describes language proficiency on a scale of six ascending levels (A1-C2). For writing skills at the end of secondary education in Europe, the common standard is vantage level B2. In this study, experts from Germany and Switzerland linked upper secondary students’ writing profiles elicited in a constructed response test (integrated and independent essays from the TOEFL iBT®) to CEFR levels. Standard setting methodology (a modified examinee paper selection/performance profile approach) was used to establish the linkages. The study reports the methodology and procedure of the standard setting process and discusses the procedural and internal validity of resulting cut scores. It also applies the cut scores to a large sample of upper secondary students in Germany and Switzerland to gain evidence for external and consequential validity.

Journal
Assessing Writing
Published
2020-01-01
DOI
10.1016/j.asw.2019.100420
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
writing pedagogy assessment multilingual writers
Export
BibTeX RIS

Citation Context

Cited by in this index (2)

  1. Siekmann et al. (2022)
    Structure and coherence as challenges in composition: A study of assessing less…
    Assessing Writing
  2. Barrot et al. (2021)
    Complexity, accuracy, and fluency as indices of college-level L2 writers’ proficiency
    Assessing Writing

References (50)

  1. Alderson (2005)
    Diagnosing foreign language proficiency: The interface between learning and assessment
  2. Alderson (2005)
    The development of a suite of computer-based diagnostic tests based on the Common Europea…
    Language Testing  ↗
  3. Attali (2007)
    Construct validity of E-rater in scoring TOEFL iBT essays (ETS Research Rep. No RR-07-21).
  4. Bayer (2016)
    Evaluation der Englischkompetenzen im Kanton Aargau. Englischkompetenzen auf der Primarst…
    Evaluation of English competences in the Canton Aargau. English competences on primary and lower secondary level]
  5. Bundesamt für Statistik (2018)
    Maturitätsquote – Daten der Grafiken
    Graduation rates – data of figures]
Show all 50 →
  1. Burstein (2013)
    The E-rater® automated essay scoring system
    Handbook of automated essay evaluation: Current applications and new directions
  2. Carlsen (2014)
    The B2 level and its applicability in university entrance tests
    Paper Presented at the 5th ALTE International Conference
  3. (2012)
    Setting performance standards: Foundations, methods, and innovations
  4. Cizek (2007)
    Standard setting: A guide to establishing and evaluating performance standards on tests
  5. Council of Europe (2001)
    Common European Framework of Reference for languages- Learning, teaching and assessment
  6. Council of Europe (2009)
    Relating language examinations to the Common European Framework of Reference for languages: Learning, teaching, assessment (CEFR). A manual
  7. Deygers (2017)
    One framework to unite them all? Use of the CEFR in European university entrance policies
    Language Assessment Quarterly
  8. EDB Educational Department of Berne (2017)
    Educational Department of Berne [EDB] (2017). Sprachniveau an der Maturität gemäss Europä…
    Language proficiency at Matura level according to the European Language Portfolio ELP]
  9. EDBS Educational Department of Basel-Stadt (2017)
    Lehrplan Gymnasium [Curriculum for Gymnasium schools]
  10. [EDK] Swiss Conference of Cantonal Ministers of Education (1994)
    Rahmenlehrplan für die Maturitätsschulen
    General curriculum for upper secondary schools]
  11. European Commission (2017)
    Key data on teaching languages at school in Europe – 2017 edition. Eurydice report
  12. Fleckenstein (2018)
    Teachers’ judgment accuracy concerning CEFR levels of prospective university students
    Language Assessment Quarterly
  13. Fleckenstein (2016)
    Proficient beyond borders: Assessing non-native speakers in a native speakers’ framework
    Large-scale Assessments in Education  ↗
  14. Freunberger (2013)
    Linking the Austrian standards-based test for English to the CEFR: The standard setting process
    Technical Documentation – Standards-Based Proficency Test in English, 8th Grade
  15. Hambleton (2000)
    Setting performance standards on complex educational assessments
    Applied Psychological Measurement  ↗
  16. Hambleton (2006)
    Setting performance standards
    Educational measurement
  17. Ingenkamp (1976)
    Die Fragwürdigkeit der Zensurengebung. Texte und Untersuchungsberichte
    The dubiousness of grading. Texts and research reports]
  18. Kaftandjieva (2010)
    Methods for setting cut scores in criterion-referenced achievement tests: A comparative analysis of six methods with an application to tests of reading in EFL
  19. Kaftandjieva (2002)
    Council of Europe scales of language proficiency: A validation study
    Common European Framework of Reference for languages: Learning, teaching, assessment. Case studies
  20. Kane (2008)
    Terminology, emphasis, and utility in validation
    Educational Researcher  ↗
  21. Keller (2013)
    Integrative Schreibdidaktik Englisch für die Sekundarstufe. Theorie, Prozessgestaltung, Empirie
    Integrative writing instruction in English at secondary level. Theory, process design, empirical evidence]
  22. Keller (2016)
    Measuring writing at secondary level (MEWS). Eine binationale Studie
    Babylonia
  23. [KMK] The Standing Conference of the Ministers of Education and Cultural Affairs of the Länder in the Federal Republic of Germany (2014)
    Bildungsstandards für die fortgeführte Fremdsprache (Englisch/Französisch) für die Allgem…
    (Beschluss der Kultusministerkonferenz vom 18.10.2012) [Educational standards for the continued foreign language (English/French) at the end of upper secondary school]
  24. [KMK] The Standing Conference of the Ministers of Education and Cultural Affairs of the Länder in the Federal Republic of Germany (2018)
    Dokumentation 215: Schüler, Klassen, Lehrer und Absolventen der Schulen 2007-2016
    Documentation 2015: Students, classes, teachers, and graduates in schools 2007-2016]
  25. Morgan (2004)
    The performance profile method (PPM): A unique standard setting method as applied to a un…
    Paper Presented at the Annual Meeting of the Council of Chief State School Officers
  26. Milanovic (2010)
    Series editors’ note
    Relating language examinations to the Common European Framework of Reference for Languages: Case studies and reflections on the use of the Council of Europes Draft Manual (pp. Viii−Xx)
  27. Mills (1998)
    Creating description of desired student achievement when setting performance standards
    Meeting the requirements of title I: Handbook for the development of performance standards (pp. 73–85)
  28. Nichols (2010)
    Standard-setting methods as measurement processes
    Educational Measurement Issues and Practice  ↗
  29. Pant (2009)
    Validity issues in standard-setting studies
    Studies in Educational Evaluation  ↗
  30. Perie (2012)
    Setting achievement standards on assessments for students with disabilities
    Setting performance standards: Foundations, methods, and innovations
  31. Pitoniak (2003)
    Standard-setting methods for complex licensure examinations. Unpublished doctoral dissertation
  32. Plakans (2010)
    Independent vs. Integrated writing tasks: A comparison of task representation
    TESOL Quarterly  ↗
  33. Porsch (2010)
    Standardbasiertes Testen im Fach Englisch
    Standard-based assessment in English].
  34. Ramineni (2012)
    Evaluation of the E-rater scoring engine for the TOEFL iBT independent and integrated prompts
    ETS research report RR-12-06
  35. Rupp (2019)
    Automated essay scoring at scale: A case study in Switzerland and Germany
  36. SH, 2014 Schleswig-Holstein, Ministerium für Bildung Wissenschaft Fachanforderungen English. Sekundarstufe II…
  37. (2016)
    IQB Bildungstrend 2015. Sprachliche Kompetenzen am Ende der 9. Jahrgangsstufe im zweiten Ländervergleich [IQB education trend 2015. Language competences at the end of 9th grade in the second German National Assessment]
  38. Südkamp (2012)
    Accuracy of teachers’ judgments of students’ academic achievement: A meta-analysis
    Journal of Educational Psychology  ↗
  39. Tannenbaum (2011)
    Mapping TOEFL iBT® ITP scores onto the Common European Framework of Reference (RM 11–33)
  40. Tannenbaum (2014)
    Critical factors to consider in evaluating standard- setting studies to map language test…
    Language Assessment Quarterly  ↗
  41. Tannenbaum (2008)
    Linking English language test scores onto the Common European Framework of Reference: An application of standard setting methodology
  42. Van Ek (2001)
    Vantage
  43. van Ockenburg (2016)
    Learning to write synthesis texts: A review of intervention studies
    Journal of Writing Research
  44. Zieky (2001)
    So much has changed: How the setting of cutscores has evolved since the 1980s
    Setting performance standards: Concepts, methods, and perspectives
  45. Zieky (2008)
    Cutscores: A manual for setting standards of performance on educational and occupational tests
CrossRef global citation count: 29 View in citation network → Build reading path →

Related Articles

  1. Assessing Writing Jan 2026
    How reliable and valid is peer evaluation in adolescents’ L2 argumentative writing? ↗
    Albert W. Li; Steve Graham
    writing pedagogy teacher development argument assessment multilingual writers grammar and mechanics literacy studies
  2. Assessing Writing Jan 2026
    Assessing the effects of explicit coherence instruction on EFL students’ integrated writing performance ↗
    Xi Li; Mo Chen
    discourse analysis writing pedagogy graduate education teacher development assessment empirical research multilingual writers
  3. Writing and Pedagogy Apr 2025
    A Model of Self- and Socially Regulated Multilingual Writing ↗
    Diana Akhmedjanova; Heidi L. Andrade
    writing pedagogy assessment empirical research multilingual writers affect and writing
  4. Pedagogy Oct 2024
    Contributors ↗
    rhetorical criticism first-year composition writing pedagogy basic writing writing across the curriculum graduate education two-year college service learning teacher development revision argument collaborative writing assessment writing program administration multimodality multilingual writers literacy studies race and writing disability studies community literacy editorial matter
  5. Assessing Writing Apr 2024
    Is the variation in syntactic complexity features observed in argumentative essays produced by B1 level EFL learners in Finland and Pakistan attributable exclusively to their L1? ↗
    Ghulam Abbas Khushik
    writing pedagogy assessment multilingual writers grammar and mechanics