The influence of lexical features on teacher judgements of ESL argumentative essays

Cristina Vögelin University of Basel ; Thorben Jansen Christian-Albrechts-Universität zu Kiel ; Stefan D. Keller ; Nils Machts Christian-Albrechts-Universität zu Kiel ; Jens Möller

Abstract

Numerous studies have examined the relationship between lexical features of students’ compositions and judgements of text quality. However, the degree to which teachers’ judgements are influenced by the quality of vocabulary in students’ essays with regard to their assessment of other textual characteristics is relatively unexplored. This experimental study investigates the influence of lexical features on teachers’ judgements of English as a second language (ESL) argumentative essays. Using analytic and holistic rating scales, English pre-service teachers (N = 37) in Switzerland assessed four essays of different proficiency levels in which the levels of lexical diversity and sophistication had been experimentally varied. Coh-Metrix software was used to manipulate the level of lexical diversity, as measured by MTLD and D, and the Tool for the Automatic Analysis of Lexical Sophistication (TAALES) software was used to obtain differing levels of lexical sophistication, as measured by word range. The results suggested that texts with greater lexical diversity and sophistication were assessed more positively concerning their overall quality as well as the analytic criteria ‘grammar’ and ‘frame of essay’. The implications of this study for classroom practice and teacher education are discussed.

Journal
Assessing Writing
Published
2019-01-01
DOI
10.1016/j.asw.2018.12.003
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
Export

Citation Context

Cited by in this index (13)

  1. Assessing Writing
  2. Assessing Writing
  3. Assessing Writing
  4. Assessing Writing
  5. Assessing Writing
Show all 13 →
  1. Assessing Writing
  2. Assessing Writing
  3. Assessing Writing
  4. Assessing Writing
  5. Assessing Writing
  6. Assessing Writing
  7. Assessing Writing
  8. Assessing Writing

References (107) · 10 in this index

  1. Towards a theory of diagnosis in second and foreign language assessment: Insights from pr…
    Applied Linguistics  
  2. Stichwort: Professionelle Kompetenz von Lehrkräften
    Zeitschrift für Erziehungswissenschaften  
  3. Support for assessment practice: Developing the assessment design decisions framework
    Teaching in Higher Education  
  4. Detecting halo effects in performance-based examinations
    Applied Psychological Measurement  
  5. The psychology of written composition
Show all 107 →
  1. How concordant are teachers’ essay scorings? A replication of Rudolf Weiss’ studies
    Psychologie in Erziehung und Unterricht
  2. Assessment and classroom learning
    Assessment in Education Principles Policy and Practice  
  3. ’In praise of educational research’: Formative assessment
    British Educational Research Journal  
  4. Handbook on the formative and summative evaluation of student learning
  5. Writing framework for the 2011 national assessment of education progress
  6. Perception and the representative design of psychological experiments
  7. Englisch
    Hochschulreife und Studierfähigkeit - Zürcher Analysen und Empfehlungen zur Schnittstelle
  8. An investigation of the impact of the 6+1 trait writing model on grade 5 student writing achievement - final report
  9. Common European framework of reference for languages: Learning, teaching, assessment
  10. Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second la…
    International Journal of Continuing Engineering Education and Life-Long Learning  
  11. Predicting second language writing proficiency: The roles of cohesion and linguistic soph…
    Journal of Research in Reading  
  12. Predicting the proficiency level of language learners using lexical indices
    Language Testing  
  13. Comparing count-based and band-based incides of word frequency: Implications for active v…
    System  
  14. Linguistic microfeatures to predict L2 writing proficiency: A case study in automated wri…
    Journal of Writing Assessment
  15. Validating lexical measures using human scores of lexical proficiency
    Vocabulary knowledge - Human ratings and automated measures
  16. Assessing lexical proficiency using analytic ratings: A case for collocation accuracy
    Applied Linguistics
  17. Assessment in the second language writing classroom
  18. Assessing Writing
  19. 6 + 1 traits of writing: The complete guide
  20. Assessing Writing
  21. Decision making while rating ESL/EFL writing tasks: A descriptive framework
    Modern Language Journal  
  22. Modelling and assessing vocabulary knowledge
  23. Developmental trends in lexical diversity
    Applied Linguistics  
  24. Rater types in writing performance assessments: A classification approach to rater variability
    Language Testing  
  25. Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated ass…
    Frankfurt am Main: Peter Lang.
  26. Assessing writing
    Handbook of second language assessment
  27. Rahmenlehrplan für die Maturiätsschulen
  28. Reglement über die Anerkennung von gymnasialen Maturiätsausweisen
  29. The relationship of lexical proficiency to the quality of ESL compositions
    Journal of Second Language Writing  
  30. Invariant Measurement with Raters And Rating Scales: Rasch Models for Rater-Mediated Assessments
  31. Teacher commentary on student writing: Descriptions & implications
    Journal of Second Language Writing  
  32. A cognitive process theory of writing
    National Council of Teachers of English
  33. Coh-Metrix: Analysis of text on cohesion and language
    Behavior Research Methods Instruments & Computers  
  34. Using computer-tagged linguistic features to describe L2 writing differences
    Journal of Second Language Writing  
  35. Assessing Writing
  36. Writing teachers as assessors of writing
    Exploring the dynamics of second language writing
  37. Purposes of assessment
    Handbook of second language assessment
  38. Assessing Writing
  39. A genre description of the argumentative essay
    RELC Journal  
  40. Second language writing
  41. Lehrbuch der Pädagogischen Diagnostik [Handbook of pedagogical diagnostics]
  42. Short texts, best fitting curves, and new measures of lexical diversity
    Language Testing  
  43. Defining and measuring lexical diversity
    Vocabulary knowledge - Human ratings and automated measures
  44. Introduction
    Vocabulary knowledge - Human ratings and automated measures
  45. Exploring multiple profiles of highly rated learner compositions
    Journal of Second Language Writing  
  46. Das Schülerinventar: Welche Schülermerkmale die Leistungsurteile von Lehrkräften beeinflussen
    Zeitschrift für Erziehungswissenschaften  
  47. The effects of student characteristics on teachers’ judgment accuracy: Disentangling ethn…
    Journal of Educational Psychology  
  48. Integrative Schreibdidaktik Englisch für die Sekundarstufe - Theorie, Prozessgestaltung, Empirie [an integrative approach of teaching English writing at secondary level - theory, process development, empiricism]
  49. Bildungsstandards für die fortgeführte Fremdsprache (Englisch / Französisch) für die Allgemeine Hochschulreife
  50. Diagnostic assessment of writing: A comparison of two rating scales
    Language Testing  
  51. Teilkomponenten von Schreibkompetenz untersuchen: Bericht aus einem interdisziplinären em…
    Formate Fachdidaktischer Forschung - Empirische Projekte - historische Analysen - theoretische Grundlegungen
  52. Die systematische Zufälligkeit des Bildungserfolgs [The systematic continguency of educational achievement]
  53. Automatically assessing lexical sophistication: Indices, tools, findings, and application
    TESOL Quarterly  
  54. The relationship between lexical sophistication and independent and source-based writing
    Journal of Second Language Writing  
  55. Vocabulary size and use: Lexical richness in L2 written production
    Applied Linguistics  
  56. The lexical approach: The state of ELT and a way forward
  57. Lexis in composition: A performance analysis of swedish learners’ written english
  58. The relationship of lexical richness to the quality of ESL learners’ oral narratives
    Modern Language Journal  
  59. Lexical diversity and language development: Quantification and assessment
  60. Composition errors and essay examination grades re-examined
    American Educational Research Journal  
  61. An assessment of the range and usefulness of lexical diversity measures and the potential of the Measure of Textual, Lexical Diversity (MTLD). Unpublished doctoral dissertation
  62. Vocd: A theoretical and empirical evaluation
    Language Testing  
  63. MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversi…
    Behavior Research Methods  
  64. From intrinsic to extrinsic issues of lexical diversity assessment - An ecological valida…
    Vocabulary knowledge - Human ratings and automated measures
  65. Written Communication
  66. Automated evaluation of text and discourse with coh-metrix
  67. The effect of marker background and training on the quality of marking in GCSE English
  68. Principled rubric adoption and adaptation: One multi-method case study
    Contemporary second language assessment
  69. Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning
  70. Researching and analyzing vocabulary
  71. The relationship between vocabulary and writing quality in three genres
    Reading and Writing  
  72. Assessing Writing
  73. Schreibkompetenzvermittlung im Englischunterricht in der Sekundarstufe I - Empirische Analysen zu Leistungen, Einstellungen, Unterrichtsmethoden und Zusammenhängen von Leistungen in der Mutter- und Fremdsprache
  74. Standardbasiertes Testen von Schreibkompetenzen im Fach Englisch
    Schulische Lerngelegenheiten und Kompetenzentwicklung
  75. Assessing Writing
  76. Subjektive Einschätzungen des Kompetenzerwerbs in der Lehramtsausbildung
    Forschung zur Lehrerbildung - Kompetenzentwicklung und Programmevaluation
  77. Assessing vocabulary
  78. Assessing Writing
  79. Is Teaching Experience Necessary for Reliable Scoring of Extended English Questions?
    Educational Measurement Issues and Practice  
  80. Formative assessment and the design of instructional systems
    Instructional Science  
  81. The effect of selected composition errors on grades assigned to essay examinations
    American Educational Research Journal  
  82. Is it really robust? Reinvestigating the robustness of ANOVA against violations of the no…
    Methodology  
  83. Diagnostische Kompetenz von Lehrpersonen
    Beiträge zur Lehrerbildung  
  84. The methodology of evaluation
  85. The effect of raters’ background and training on the reliability of direct writing tests
    Modern Language Journal  
  86. Intraclass correlations: Uses in assessing rater reliability
    Psychological Bulletin  
  87. Knowledge and teaching: Foundations of the new reform
    Harvard Educational Review  
  88. Understanding first-year L2 writing: A lexico-grammatical analysis across L1s, genres, an…
    Journal of Second Language Writing  
  89. Accuracy of teachers’ judgments of students academic achievement: A meta-analysis
    Journal of Educational Psychology  
  90. From oxford university computing services on behalf of the BNC consortium
  91. A constant error in psychological ratings
    The Journal of Applied Psychology
  92. Measuring lexical diversity among L2 learners of French - An exploration of the validity …
    Vocabulary knowledge - Human ratings and automated measures
  93. Assessing writing
  94. Teaching writing teachers about assessment
    Journal of Second Language Writing  
  95. The specification, realization and validation of an English language proficiency test
    Testing English for university study. ELT documents 127
  96. Are you assessment literate? Some fundamental questions regarding effective classroom-bas…
    OnCUE Journal
  97. Assessing Writing
  98. Assessing Writing
  99. Lexical diversity in writing and speaking task performances
    Applied Linguistics  
  100. Writers at work - the essay
  101. Performing argumentative writing in english: Difficulties, processes, and strategies
    TESL Canada Journal