Utility-Value Score: A Case Study in System Generalization for Writing Analytics

Beata Beigman Klebanov; Stacy Priniski; Jill Burstein; Binod Gyawali; Judith Harackiewicz; Dustin Thoman

doi:10.37514/jwa-j.2018.2.1.13

Journal of Writing Analytics Jan 2018 Open Access

Utility-Value Score: A Case Study in System Generalization for Writing Analytics

Beata Beigman Klebanov Educational Testing Service ; Stacy Priniski ; Jill Burstein Educational Testing Service ; Binod Gyawali Educational Testing Service ; Judith Harackiewicz ; Dustin Thoman

Abstract

Collection and analysis of students' writing samples on a large scale is a part of the research agenda of the emerging writing analytics community that promises to deliver an unprecedented insight into characteristics of student writing. Yet with a large scale often comes variability of contexts in which the samples were produced-different institutions, different purposes of writing, different author demographics, to name just a few possible dimensions of variation. What are the implications of such variation for the ability of automated methods to create indices/features based on the writing samples that would be valid and meaningful? This paper presents a case study in system generalization. Building on a system developed to assess the expression of utility value (a social-psychology-based construct) in essays written by first-year biology students at one postsecondary institution, we vary data parameters and observe system performance. From the point of view of social psychology, all these variants represent the same underlying construct (i.e., utility value), and it is thus very tempting to think that an automatically produced utility-value score could provide a meaningful analytic, consistently, on a large collection of essays. However, findings from this research show that there are challenges: Some variations are easier to deal with than others, and some components of the automated system generalize better than others. The findings are then discussed both in the context of the case study and more generally.

Journal: Journal of Writing Analytics
Published: 2018-01-01
DOI: 10.37514/jwa-j.2018.2.1.13
CompPile: Search in CompPile ↗
Open Access: OA PDF Gold
Topics: qualitative research
Export: BibTeX RIS

Citation Context

Cited by in this index (0)

No articles in this index cite this work.

References (0)

No references on file for this article.

CrossRef global citation count: 4 View in citation network → Build reading path →

Utility-Value Score: A Case Study in System Generalization for Writing Analytics

Abstract

Citation Context

Cited by in this index (0)

References (0)

Related Articles