Visualizing formative feedback in statistics writing: An exploratory study of student motivation using DocuScope Write & Audit

Michael Laudenbach Carnegie Mellon University ; David West Brown Carnegie Mellon University ; Zhiyu Guo ; Suguru Ishizaki Carnegie Mellon University ; Alex Reinhart Carnegie Mellon University ; Gordon Weinberg Carnegie Mellon University

Abstract

Recently, formative feedback in writing instruction has been supported by technologies generally referred to as Automated Writing Evaluation tools. However, such tools are limited in their capacity to explore specific disciplinary genres, and they have shown mixed results in student writing improvement. We explore how technology-enhanced writing interventions can positively affect student attitudes toward and beliefs about writing, both reinforcing content knowledge and increasing student motivation. Using a student-facing text-visualization tool called Write & Audit, we hosted revision workshops for students (n = 30) in an introductory-level statistics course at a large North American University. The tool is designed to be flexible: instructors of various courses can create expectations and predefine topics that are genre-specific. In this way, students are offered non-evaluative formative feedback which redirects them to field-specific strategies. To gauge the usefulness of Write & Audit, we used a previously validated survey instrument designed to measure the construct model of student motivation (Ling et al. 2021). Our results show significant increases in student self-efficacy and beliefs about the importance of content in successful writing. We contextualize these findings with data from three student think-aloud interviews, which demonstrate metacognitive awareness while using the tool. Ultimately, this exploratory study is non-experimental, but it contributes a novel approach to automated formative feedback and confirms the promising potential of Write & Audit.

Journal
Assessing Writing
Published
2024-04-01
DOI
10.1016/j.asw.2024.100830
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
Export

Citation Context

Cited by in this index (0)

No articles in this index cite this work.

References (95) · 22 in this index

  1. Computer-based writing instruction
    Handbook of writing research
  2. Design-based research: A decade of progress in education research?
    Educational Researcher  
  3. Journal of Writing Analytics
  4. Readability: Text and context
  5. Discursively structured activities
    Mind, Culture and Activity  
Show all 95 →
  1. Knowledge telling and knowledge transforming in written composition
    Advances in Applied Psycholinguistics
  2. The think-aloud controversy in second language research
  3. Research in the Teaching of English
  4. Corpora and Rhetorically Informed Text Analysis: The Diverse Applications of DocuScope. J…
    Series in Corpus Linguistics
  5. Examining dimensions of self-efficacy for writing
    Journal of Educational Psychology  
  6. Self-efficacy beliefs and motivation in writing development
    Handbook of writing research
  7. Expanding automated writing evaluation
    Handbook of automated scoring
  8. Strategy uptake in Writing Pal: Adaptive feedback and instruction
    Journal of Educational Computing Research  
  9. The evolving relationship between composition and cognitive studies: Gaining some histori…
    Contemporary perspectives on cognition and, writing  
  10. Assessing Writing
  11. Journal of Writing Analytics
  12. Understanding graduate writers’ interaction with and impact of the Research Writing Tutor…
    Research
  13. The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of loc…
    Behavior Research Methods  
  14. One instance of Prague School methodology: Functional analysis of utterance and text
    Method and Theory in Linguistics  
  15. Analytical techniques for developing argumentative writing in STEM: A pilot study
    IEEE Transactions on Education  
  16. Assessing Writing
  17. Toward a theory of adaptive transfer: Expanding disciplinary discussions of “transfer” in…
    Journal of Second Language Writing  
  18. Statistics: A data science for the 21st century
    Journal of the Royal Statistical Society: Series A (Statistics in Society)  
  19. Formative and summative assessment in the classroom
    Theory into Practice  
  20. Academic Ableism: Disability and Higher Education
  21. 50 Years of Data Science
    Journal of Computational and Graphical Statistics  
  22. Beyond knowledge and skills: Writing transfer and the role of student dispositions
    Composition Forum
  23. A conceptual history of the achievement goal construct
    Handbook of motivation and competence
  24. A hierarchical model of approach and avoidance achievement motivation
    Journal of Personality and Social Psychology  
  25. Automated essay evaluation and the teaching of writing
    Handbook of automated essay evaluation: Current applications and new directions
  26. The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results
  27. College English
  28. The construction of negotiated meaning
  29. Written Communication
  30. Research in the Teaching of English
  31. College Composition and Communication
  32. Writing goals in U.S. undergraduate data science course outlines: A textual analysis
    Teaching Statistics  
  33. The science of scientific writing
    American Scientist
  34. Graham, S., Harris, K., & Hebert, M. (2011). Informing Writing: The Benefits of Formative Assessment. A Repor…
  35. Research-based writing practices and the common core: Meta-analysis and meta-synthesis
    The Elementary School Journal  
  36. Knowledge of writing and the composing process, attitude towards writing, and self-effica…
    Journal of Learning Disabilities  
  37. Data science in statistics curricula: Preparing students to “Think with Data”
    The American Statistician  
  38. Journal of Technical Writing and Communication
  39. Studying the mediated action of composing with time-use diaries
    Digital writing research: Technologies, methodologies and ethical issues
  40. Using writing to improve student learning of statistics
    Writing Across the Curriculum
  41. Written Communication
  42. Formative automated writing evaluation: A standpoint theory of action
    Journal of Response to Writing
  43. Assessing Writing
  44. Exploring the Use of Mobile Technologies and Process Logs in Writing Research
    International Journal of Qualitative Methods  
  45. Curriculum guidelines for undergraduate programs in statistical science
    American Statistical Association
  46. Writing without conviction? Hedging in science research articles
    Applied Linguistics  
  47. Boosting, hedging and the negotiation of academic knowledge
    Text & Talking
  48. Directives: Argument and engagement in academic writing
    Applied Linguistics  
  49. Ishizaki, S. & Kaufer, D. (2011). Computer-aided Rhetorical Analysis. In P. McCarthy & C. Boonthum (Ed.), App…
  50. Journal of Writing Research
  51. Kratochwill, T.R., Hitchcock, J., Horner, R.H., Levin, J.R., Odom, S.L., Rindskopf, D.M. & Shadish, W.R. (201…
  52. Enhancing the scientific credibility of single-case intervention research: Randomization …
    Psychological Methods  
  53. What is the problem to be solved? The purpose for using think-aloud and cognitive laborat…
    Using Think-Aloud Interviews and Cognitive Labs in Educational Research
  54. Written Communication
  55. Topical themes and thematic progression: The “picture” of research articles
    Text & Talking
  56. Thematic density of research-article abstracts: A systemic-functional account
    Word  
  57. Assessing Writing
  58. The actor-oriented transfer perspective and its contributions to educational research and…
    Educational Psychologist  
  59. Self-regulated strategy instruction in developmental writing: A design research project
    Community College Review  
  60. A multicomponent measure of writing motivation with basic college writers
    Learning Disability Quarterly  
  61. Writing to reduce anxiety and improve outcomes in introduction to statistics for psycholo…
    Psychology Teaching Review  
  62. National Academies of Sciences, Engineering, and Medicine. (2017). Supporting students’ college success: The …
  63. The promise of portfolios: Training modern data scientists
    Harvard Data Science Review
  64. Effectiveness of automated writing evaluation systems in school settings: A systematic re…
    Journal of Computer Assisted Learning  
  65. Self-efficacy beliefs, motivation, and achieve- ment in writing: A review of the literature
    Reading & Writing Quarterly  
  66. Writing self-efficacy and its relation to gender, writing motivation and writing competen…
    Writing and Motivation  
  67. Knowledge to go: A motivational and dispositional view of transfer
    Educational Psychologist  
  68. Philippakos, Z.A., & MacArthur, C.A. (2015). Writing motivation: Validation of a measure for college writers.…
  69. Think-aloud interviews: A tool for exploring student statistical reasoning
    Journal of Statistics and Data Science Education  
  70. Written Communication
  71. Fundamental research statistics for the behavioral sciences
  72. Computers and Composition
  73. Writing pal: Feasibility of an intelligent writing strategy tutor in the high school classroom
    Journal of Educational Psychology  
  74. Automated detection of essay revising patterns: Application for intelligent feedback in a…
    Cognition, and Learning
  75. Developing pedagogically-guided algorithms for intelligent writing feedback
    International Journal of Learning Technology  
  76. Constructing automated revision graphs: A novel visualization technique to study student …
    Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part II
  77. Digital support for academic writing: A review of technologies and pedagogies
    Computers & Education  
  78. Genre analysis: English in academic and research settings
  79. Developing strategic learners: Supporting self-efficacy through goal setting and reflection
    The Language and Literacy Spectrum
  80. Writing motivation of college students in basic writing and first-year composition classe…
    Journal of Learning Disabilities  
  81. Written Communication
  82. Assessing Writing
  83. Very like a whale: The assessment of writing programs
  84. Implicit writing beliefs and their relation to writing quality
    Contemporary Educational Psychology  
  85. Journal of Writing Research
  86. Topical structure and revision: An exploratory study
    College Composition and Communication
  87. College Composition and Communication
  88. Writing assignments to assess statistical thinking
    Journal of Statistics Education  
  89. Journal of Writing Analytics
  90. The Effectiveness of Automated Writing Evaluation on Writing Quality: A Meta-Analysis
    Journal of Educational Computing Research