Examining the direct and indirect impacts of verbatim source use on linguistic complexity in integrated argumentative writing assessment

Huiying Cai University of Illinois Urbana-Champaign ; Xun Yan University of Illinois Urbana-Champaign

Abstract

Verbatim source use (VSU) in integrated argumentative writing tasks may enhance linguistic complexity of writing performance. This assistance might present an unequal advantage for test-takers across levels of writing proficiency, engendering validity and fairness concerns. While previous research has mostly examined the relationships between source use characteristics and proficiency levels, the relationship between VSU and linguistic complexity remains underexplored. To further unpack these relationships, this study examined both the direct impact of VSU on linguistic complexity of writing performances and its indirect impact through interaction with writing proficiency. Using natural language processing tools and techniques, we examined 34 linguistic complexity features and three VSU features of 3250 argumentative writing performances on a university-level English Placement Test (EPT). We performed exploratory factor analysis to identify linguistic complexity dimensions and applied mixed-effect models to examine how VSU features and proficiency level impacted these dimensions. Post-hoc analyses suggested weak direct impacts of different VSU features on linguistic complexity, which might reflect different essay writing strategies. However, no meaningful indirect impact was found. The findings help unravel the impact of VSU on argumentative writing and provide empirical evidence for validity arguments for integrated writing assessments.

Journal
Assessing Writing
Published
2024-07-01
DOI
10.1016/j.asw.2024.100868
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
Export

Citation Context

Cited by in this index (1)

  1. Assessing Writing

References (52) · 6 in this index

  1. Recurrent word combinations in EAP test-taker writing: Differences between high- and low-…
    Language Assessment Quarterly  
  2. Challenging stereotypes about academic writing: Complexity, elaboration, explicitness
    Journal of English for Academic Purposes  
  3. Investigating grammatical complexity in L2 English writing research: Linguistic descripti…
    Journal of English for Academic Purposes  
  4. Merging corpus linguistic and discourse analytic research goals: Discourse units in biolo…
    Corpus Linguistics and Linguistic Theory  
  5. Conceptualizing and measuring short-term changes in L2 writing complexity
    Journal of Second Language Writing  
Show all 52 →
  1. Assessing Writing
  2. Building a validity argument for the Test of English as a Foreign Language
  3. Statistical Power Analysis for the Behavioral Sciences
  4. A First Course in Factor Analysis
  5. Common European Framework of Reference for Languages: Learning, teaching, assessment
  6. Cutting the Gordian knot: The moving-average type–token ratio (MATTR)
    Journal of Quantitative Linguistics  
  7. A new academic word list
    TESOL Quarterly  
  8. The academic word list 10 years on: Research and teaching implications
    Tesol Quarterly  
  9. Journal of Writing Research
  10. Predicting the proficiency level of language learners using lexical indices
    Language Testing  
  11. Analysis of Discourse Features and Verification of Scoring Levels for Independent and Int…
    ETS Research Report Series  
  12. Beginning and intermediate L2 writer’s use of ngrams: An association measures study
    International Review of Applied Linguistics in Language Teaching  
  13. Gebril, A., & Plakans, L. (2009). Investigating source use, discourse features, and process in integrated wri…
  14. Source-based tasks in academic writing assessment: Lexical diversity, textual borrowing a…
    Journal of English for Academic Purposes  
  15. The use of collocations by intermediate vs. advanced non-native writers: A bigram-based study
    International Review of Applied Linguistics  
  16. Analyzing lexical density and lexical diversity in university students’ written discourse
    Procedia-Social and Behavioral Sciences  
  17. Problèmes et méthodes de la statistique linguistique
  18. Assessing Writing
  19. Studies in language behavior: I. A program of research
    Psychological Monographs  
  20. Analysis of factorial simplicity
    Psychometrika  
  21. An argument-based approach to validity
    Psychological Bulletin  
  22. Validating the Interpretations and Uses of Test Scores
    Journal of Educational Measurement  
  23. Predicting L2 Writing Proficiency Using Linguistic Complexity Measures: A Corpus-Based Study
    English Teaching  
  24. Kyle, K. (2016). Measuring syntactic development in L2 writing: Fine grained indices of syntactic complexity …
  25. Assessing Writing
  26. The relationship between lexical sophistication and independent and source-based writing
    Journal of Second Language Writing  
  27. Measuring syntactic complexity in L2 writing using fine-grained clausal and phrasal indices
    Modern Language Journal  
  28. The tool for the automatic analysis of lexical sophistication (TAALES): version 2.0
    Behavior Research Methods  
  29. Assessing the validity of lexical diversity indices using direct judgements
    Language Assessment Quarterly  
  30. Completely Different Worlds”: EAP and the Writing Experiences of ESL Students in Universi…
    TESOL Quarterly  
  31. Automatic analysis of syntactic complexity in second language writing
    International Journal of Corpus Linguistics  
  32. The relationship of lexical richness to the quality of ESL learners’ oral narratives
    Modern Language Journal  
  33. MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversi…
    Behavior Research Methods  
  34. Assessing Writing
  35. Borrowing Others’ Words: Text, Ownership, Memory, and Plagiarism
    TESOL Quarterly  
  36. Discourse synthesis in integrated second language writing assessment
    Language Testing  
  37. Assessing Writing
  38. Shaping a score: Complexity, accuracy, and fluency in integrated writing performances
    Language Testing  
  39. Using multiple texts in an integrated writing assessment: Source text use as a predictor …
    Journal of Second Language Writing  
  40. Revelle, W. (2016). Psych: Procedures for psychological, psychometric, and personality research (Version 1.6.…
  41. Multilevel analysis: An introduction to basic and advanced multilevel modeling
  42. The constructivist metaphor: Reading, writing, and making of meaning
  43. Lexical density and register differentiation
    Applications of Linguistics
  44. Source text borrowing in an integrated reading/writing assessment
    Journal of Second Language Writing  
  45. Second language development in writing: Measures of fluency accuracy and complexity (No. 17)
  46. Handbook of regression methods
  47. The relationship between lexical use and L2 writing quality: A case of two genres
    International Journal of Applied Linguistics