Abstract

This study examined whether two integrated reading-to-write tasks could broaden the construct representation of the writing component of Duolingo English Test (DET). It also verified whether they could enhance DET’s predictive power of English academic writing in universities. The tasks were (1) writing a summary based on two source texts and (2) writing a reading-to-write essay based on five texts. Both were given to a sample (N = 204) of undergraduates from Hong Kong. Each participant also submitted an academic assignment written for the assessment of a disciplinary course. Three professional raters double-marked all writing samples against detailed analytical rubrics. Raw scores were first processed using Multi-Faceted Rasch Measurement to estimate inter- and intra-rater consistency and generate adjusted (fair) measures. Based on these measures, descriptive analyses, sequential multiple regression, and Structural Equation Modeling were conducted (in that order). The analyses verified the writing tasks’ underlying component constructs and assessed their relative contributions to the overall integrated writing scores. Both tasks were found to contribute to DET’s construct representation and add moderate predictive power to the domain performance. The findings, along with their practical implications, are discussed, especially regarding the complex relations between construct representation and predictive validity. • studied the concepts of construct representation (CR) and predictive validity (PV). • within the context of an AI-facilitated language test (Duolingo English Test). • Revealed the complex relations between CR and PV.

Journal
Assessing Writing
Published
2024-07-01
DOI
10.1016/j.asw.2024.100846
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
Export

Citation Context

Cited by in this index (1)

  1. Assessing Writing

References (61) · 8 in this index

  1. Abunawas, M. (2014). A meta-analytic investigation of the predictive validity of the Test of English as a For…
  2. What does language testing have to offer?
    TESOL Quarterly  
  3. What is the construct? The dialectic of abilities and contexts in defining constructs in …
    Language testing reconsidered
  4. Brenzel, J. & Settles, B. (2017). The Duolingo English Test-Design, Validity, and Value. DET Whitepaper(Short…
  5. Predicting grades from an English language assessment: The importance of peeling the onion
    Language Testing  
Show all 61 →
  1. Opposing tensions of local and international standards for EAP writing programmes: Who ar…
    Journal of English for Academic Purposes  
  2. Assessing Writing
  3. Assessing Writing
  4. Relationship of TOEFL iBT scores to academic performance: Some evidence from American uni…
    Language Testing  
  5. Assessing Writing
  6. Students' writing from sources for academic purposes: A synthesis of recent research
    Journal of English for Academic Purposes  
  7. Construct validity: Construct representation versus nomothetic span
    Psychological Bulletin  
  8. Mixed Rasch models for measurement in cognitive psychology
    Multivariate and mixture distribution Rasch models: Extensions and applications
  9. Language testing and assessment: An advance resource book
  10. The role of cross-disciplinary academic language skills in disciplinary, source-based wri…
    Reading and Writing  
  11. A classification of genre families in university student writing
    Applied Linguistics  
  12. Investigating source use, discourse features, and process in integrated writing tests
    Spaan Fellow Working Papers in Second Oregon Foreign Language Assessment
  13. Interpreting the relationships between TOEFL iBT scores and GPA: Language proficiency, po…
    Language Testing  
  14. Using computer-tagged linguistic features to describe L2 writing differences
    Journal of Second Language Writing  
  15. Construct-irrelevant variance in high-stakes testing
    Educational Measurement: Issues and Practice  
  16. Investigating Chinese university students’ knowledge of and attitudes towards plagiarism …
    Language Learning  
  17. A meta-analysis on the predictive validity of English language proficiency assessments fo…
    Language Testing  
  18. Examining the predictive validity of the Duolingo English Test: Evidence from a major UK …
    Language Testing  
  19. The Duolingo English Test and academic English (Duolingo Research Report DRR-16-01)
    Duolingo English Test
  20. Across the Disciplines
  21. Opening the black box: Exploring automated speaking evaluation
    Challenges in Language Testing Around the World: Insights for language test users
  22. Assessing Writing
  23. LaFlair, G.T. (2020). Duolingo English Test: Subscores (DRR-20–03). 〈https://englishtest.duolingo.com/research〉.
  24. LaFlair, G.T. & Settles, B. (2020). Duolingo English Test: Technical Manual. Duolingo. 〈https://englishtest.d…
  25. Comparability of students’ writing performance on TOEFL iBT and in required university wr…
    Language Testing  
  26. Facets computer program for many-facet Rasch measurement (Version 3.83.3)
  27. Validity
    Educational measurement
  28. Validity of psychological assessment: Validation of inferences from persons’ responses an…
    American Psychologist  
  29. Validity and washback in language testing
    Language Testing  
  30. Discourse synthesis in integrated second language writing assessment
    Language Testing  
  31. Using multiple texts in an integrated writing assessment: Source text use as a predictor …
    Journal of Second Language Writing  
  32. A register analysis of the GEPT advanced level examinees’ written production (LTTC-GEPT Research Report No. RG-04)
  33. The generalizability of summarization and free recall ratings in L2 reading assessment
    JLTA Journal  
  34. Understanding learner strengths and weaknesses: Assessing performance on an integrated wr…
    Language Assessment Quarterly  
  35. Machine learning driven language assessment
    Transactions of the Association for Computational Linguistics  
  36. Innovative item formats in computer-based testing: In pursuit of improved construct repre…
    Handbook of test development
  37. Soper, D.S. (2020). A-priori Sample Size Calculator for Structural Equation Models [Software]. 〈http://www.da…
  38. Personnel selection; test and measurement techniques
  39. The role of language skills in mid-adolescents’ science summaries
    Journal of Literacy Research  
  40. Examining the criterion-related validity of GEPT advanced reading and writing tests: Comparing GEPT with IELTS and real-life academic performance (LTTC-GEPT Research Report No. RG-01)
  41. Source text use by undergraduate post-novice L2 writers in disciplinary assignments: Prog…
    Journal of Second Language Writing  
  42. Wongtrirat, R. (2010). English language proficiency and academic achievement of international students: A met…
  43. Does test preparation work? Implications for score validity
    Language Assessment Quarterly  
  44. Do component weighting and test method affect time management and approaches to test prep…
    System  
  45. Assessing Writing
  46. Diagnosing linguistic problems in English academic writing of university students: An ite…
    Language Assessment Quarterly  
  47. Assessing Writing
  48. Do test design and uses influence test preparation: Testing a model of washback with Stru…
    Language Testing  
  49. Toward a model of strategies and summary writing performance
    Language Assessment Quarterly  
  50. Second language writers’ strategy use and performance on an integrated reading‐listening‐…
    TESOL Quarterly  
  51. Evidence-centered design in language testing
    The Routledge Handbook of Language Testing
  52. Assessing Writing
  53. Effects of instruction on ESL students’ synthesis writing
    Journal of Second Language Writing  
  54. Exploring the relationships between independent listening and listening-reading-writing t…
    Language Assessment Quarterly  
  55. Secondary school students’ discourse synthesis performance on Chinese (L1) and English (L…
    Reading and Writing