Writing Quality Predictive Modeling: Integrating Register-Related Factors

Written Communication Oct 2023

Writing Quality Predictive Modeling: Integrating Register-Related Factors

Heqiao Wang Michigan State University ; Gary A. Troia Michigan State University

Abstract

The primary purpose of this study is to investigate the degree to which register knowledge, register-specific motivation, and diverse linguistic features are predictive of human judgment of writing quality in three registers—narrative, informative, and opinion. The secondary purpose is to compare the evaluation metrics of register-partitioned automated writing evaluation models in three conditions: (1) register-related factors alone, (2) linguistic features alone, and (3) the combination of these two. A total of 1006 essays ( n = 327, 342, and 337 for informative, narrative, and opinion, respectively) written by 92 fourth- and fifth-graders were examined. A series of hierarchical linear regression analyses controlling for the effects of demographics were conducted to select the most useful features to capture text quality, scored by humans, in the three registers. These features were in turn entered into automated writing evaluation predictive models with tuning of the parameters in a tenfold cross-validation procedure. The average validity coefficients (i.e., quadratic-weighed kappa, Pearson correlation r, standardized mean score difference, score deviation analysis) were computed. The results demonstrate that (1) diverse feature sets are utilized to predict quality in the three registers, and (2) the combination of register-related factors and linguistic features increases the accuracy and validity of all human and automated scoring models, especially for the registers of informative and opinion writing. The findings from this study suggest that students’ register knowledge and register-specific motivation add additional predictive information when evaluating writing quality across registers beyond that afforded by linguistic features of the paper itself, whether using human scoring or automated evaluation. These findings have practical implications for educational practitioners and scholars in that they can help strengthen consideration of register-specific writing skills and cognitive and motivational forces that are essential components of effective writing instruction and assessment.

Journal: Written Communication
Published: 2023-10-01
DOI: 10.1177/07410883231185287
Open Access: Closed
Topics: writing pedagogy multimodality artificial intelligence

Citation Context

Cited by in this index (2)

Hwang (2025)

The Impact of Subordination Type and Finiteness on Second Language Development …

Written Communication
Qin et al. (2024)

Pragmatic Competence in an Email Writing Task: Influences of Situation, L1 Back…

Written Communication

Cites in this index (4)

Aull (2019)

Linguistic Markers of Stance and Genre in Upper-Level Student Writing

Written Communication
Troia et al. (2019)

Multidimensional Levels of Language Writing Measures in Grades Four to Six

Written Communication
Crossley et al. (2011)

The Development of Writing Proficiency as a Function of Grade Level: A Linguist…

Written Communication
McNamara et al. (2010)

Linguistic Features of Writing Quality

Written Communication

Also cites 69 works outside this index ↓

10.17239/jowr-2018.10.01.04

↗
10.1521/jscp.1986.4.3.359

↗
10.1016/j.asw.2007.07.001

↗
10.1007/s11145-013-9481-0

↗
10.1007/s11145-007-9107-5

↗
10.1080/03004430.2012.711590

↗
10.1177/0265532214542994

↗
10.1007/s10648-020-09530-4

↗
10.1016/1075-2935(95)90011-X

↗
10.1007/s11145-021-10221-x

↗
10.1002/j.1545-7249.2008.tb00142.x

↗
10.1177/2515245919898466

↗
10.1016/j.asw.2012.10.002

↗
10.1177/001440290407000306

↗
10.1371/journal.pone.0209749

↗
10.1007/s11145-018-9918-6

↗
10.1111/medu.12517

↗
10.1086/669938

↗
10.1111/j.1756-8765.2010.01081.x

↗
10.1086/678293

↗
10.3758/BF03195564

↗
10.1080/00461520.2018.1481406

↗
10.1111/1467-9817.12245

↗
10.1086/693009

↗
10.1044/2015_LSHSS-14-0043

↗
10.17265/2159-5313/2016.09.003

↗
10.1007/s11145-020-10057-x

↗
10.1093/elt/52.4.308

↗
10.5395/rde.2013.38.1.52

↗
10.1007/s11145-017-9724-6

↗
10.1080/00220671.1995.9941199

↗
10.7820/vli.v01.1.koizumi

↗
10.1044/0161-1461(2012/11-0018)

↗
10.1080/10573560701277542

↗
10.37514/ATD-J.2011.8.4.23

↗
10.1007/s11145-018-9853-6

↗
10.2307/357865

↗
10.1177/0265532207080767

↗
10.3758/BRM.42.2.381

↗
10.3758/s13428-012-0258-1

↗
10.4018/978-1-60960-741-8.ch011

↗
10.1017/CBO9780511894664

↗
10.1080/01638530902959943

↗
10.1002/trtr.1227

↗
10.1037/a0037549

↗
10.1007/s11145-012-9392-5

↗
10.1080/10573560308222

↗
10.1037/0022-0663.91.1.50

↗
10.1006/ceps.2000.1069

↗
10.1016/j.asw.2013.04.001

↗
10.1016/j.asw.2021.100574

↗
Troia G., Lawrence F., Brehmer J., Glause K., Reichmuth H. (2023). Efficient measurement of writing knowledge…

↗
10.3390/educsci10110297

↗
10.1007/s11145-012-9379-2

↗
10.1080/10573569.2012.632729

↗
10.1016/j.cedpsych.2022.102100

↗
10.1016/j.jsp.2022.07.002

↗
10.1177/0022466919887150

↗
10.1007/s11145-019-09938-7

↗
10.1371/journal.pone.0224365

↗
10.1016/j.asw.2021.100567

↗
10.1016/j.jslw.2012.03.004

↗
10.1016/0273-2297(92)90011-P

↗
10.1111/j.1745-3992.2011.00223.x

↗
10.1016/j.asw.2017.08.002

↗
Wright K. L., Hodges T. S., Enright E., Abbott J. (2021). The relationship between middle and high school stu…

↗
10.1080/09588221.2014.881384

↗
10.1016/j.stueduc.2020.100911

↗
10.1006/ceps.1997.0919

↗

CrossRef global citation count: 11 View in citation network →