What do writers do with the feedback they receive? While the answer will vary depending on the writer’s experience and the rhetorical situation, understanding what writers do can provide important information for course redesign and professional development of tutors and instructors. In this first of two manuscripts, the authors examine how first-semester, first-year writing students use responses provided via asynchronous online tutoring (AOT) in revising their assignments. Our primary research question was: What was happening in—and after—those tutorials? We addressed this question by a process of narrowing and refining of data analysis toward increasingly precise inferences as we progressed from automated to coded analysis, which culminated in examining the drafts submitted for tutoring, tutor feedback, and the subsequent assignments submitted for evaluation in the students’ FYW courses. In parallel, we describe the writing analytics–informed methods used to do so in hopes that others will be compelled to replicate or extend this work in their own contexts. We found that students made corresponding revisions at both macro and microstructural levels when provided with directive or declarative feedback, and they made few revisions when tutors provided open-ended questions.

<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Problem:</b> This tutorial aims to guide readers through key concepts, basic processes, and common decision points that inform computer-assisted corpus-based research in technical, professional, and scientific communication (TPSC). <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Key concepts:</b> Based on our collaborative experiences and an example developed for this tutorial, key concepts of corpus analysis useful to TPSC researchers and practitioners include the following: corpus location, text preparation, and programming language and software selection. <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Key lessons:</b> These key concepts can be used to establish basic processes and decision points that, in turn, yield lessons related to the usefulness of lexicogrammatical language models and the significance of multidisciplinarity. <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Implications:</b> Although corpus research is a growing and important part of the field of TPSC, challenges remain in terms of language model variety and ethical considerations. At least in part, these challenges can be met, respectively, by alignment between corpus and analytic tools and reference to the Common Rule and related international standards.

AbstractThis article examines a required undergraduate empirical methods course in writing, rhetoric, and literacy to assess how well it introduces humanities students to empirical research methods. The common curriculum contains a commitment to affordable learning as well as to making students agents of their own learning. Student work artifacts, pre- and post-course surveys, and course evaluations were collected and analyzed to examine the impact of the course on student understanding of and engagement in undergraduate research. Initial results indicate that students are gaining skills that will enable them to function as researchers going forward.

Background: Research incorporating large data sets and data and text mining methodologies is making initial contributions to writing studies. In writing program administration (WPA) work, one could best characterize the body of publications as small but growing, led by such work as Moxley and Eubanks’ 2015 “On Keeping Score: Instructors' vs. Students' Rubric Ratings of 46,689 Essays” and Arizona State University’s Science of Learning & Educational Technology (SoLET) Lab. Given the information that large-scale textual analysis can provide, it seems incumbent on program administrators to explore ways to make regular and aggressive use of such opportunities to give both students and instructors more resources for learning and development. This project is one attempt to add to this corpus of work; the sample for the study consisted of 17,534 pieces of student writing representing 141,659 discrete comments on that writing, with 58,300 unique words out of over 8.25 million total words written. This data is used to examine trends in the program’s instructor commentary over five years’ time.  By doing so, this study revisits a fundamental task of writing instruction—responding to student writing, and from the data’s results considers how large writing programs with constant turnover of graduate teaching assistants (GTAs) might manage their ongoing instructor professional development and how those GTAs will improve their ability to teach and respond to writing.Literature Review: Researchers have attempted to unpack and understand the task of instructor commentary for several decades; the published literature demonstrates a complex and occasionally ambivalent relationship with this central task of writing instruction. Recent scholarship has moved from the small-scale studies long used by the field to implement large-scale examinations of the instruction occurring in writing programs. Research questions: Three questions guided the inquiry:Does the work of new instructors (MA1s) more closely resemble the lexicon of novice or experienced responders to student writing?How does the new instructors’ work compare to that of more experienced (PHD1 or INS) instructors in the program throughout their time?How does their work evolve over a four-semester longitudinal time frame (as MA1 or MA2 experience levels) in the first-year writing program? [Please note that the abbreviations used above and throughout the article to designate instructor experience levels are as follows: MA1 (first-year master’s students); MA2 (second-year master’s students); PHD1 (first-year doctoral students); INS (instructors—those with 3 or more years’ experience teaching and who are not currently pursuing an additional degree—nearly all of these individuals held a Master’s degree)].Methodology: This study extends the work of Anson and Anson (2017) who first surveyed writing instructors and program administrators to create wordlists that survey respondents associated with “high-quality” and “novice” responses, and then examined a corpus of nearly 50,000 peer responses produced at a single university to learn to what extent instructors and student peers adopted this lexicon. Specifically, the study analyzes a corpus of instructor comments to students using the Anson and Anson wordlists associated with principled and novice commentary to see if new writing instructors align more closely with the concepts represented in either list during their first semester in the program.  It then tracks four cohorts for evolution and change in their vocabulary of feedback over their next three semesters in the program; the study also compares the vocabulary used in their comments to that used by experienced instructors in the program over the same time.Results: The study found that from the outset, the new instructors (MA1) incorporated more of the principled response terms than the novice response terms. Overall, in comparing the MA1 instructors with the most experienced group (INS), the results reveal three important findings about the feedback of both MA1s and INSs in this program.While there are some differences in commentary as seen via examination of the two lexicons, the differences are perhaps less than one might assume.The cohorts do increase their use of the principled terms as they move through the two years’ appointment in the program, but few of the increases demonstrate statistical significance.Few of the terms from either the novice or principled lexicon, with the exception of terms that also appear in the assignment descriptions, what I label as “content terms,” appear frequently in the overall corpus.Discussion: Based on the results, the instructors in this program had acquired a more consistent vocabulary, but not primarily one based on Anson and Anson’s two lexicons—instead, the most frequent and commonly used terms seem to come from a more local “canon,” that is, one based on the assignment descriptions and course outcomes. Regardless of whether the acquisition of a common vocabulary came from more global concepts or an assignment-based local canon, using common terms is something that Nancy Sommers (1982) saw as contributing to “thoughtful commentary” on student writing. As no one has previously studied how quickly new instructors acquire a professional vocabulary for responding to student writing, it is hard to know whether or not the results of this particular group of instructors would be considered “typical.” However, it may well be that the context of this writing program contributed to a more accelerated acquisition.Conclusions: Working with the lexicons developed via Anson and Anson’s survey is a useful starting point for understanding more of what our instructors actually do when responding to student writing, as well as for identifying critical differences in our instructors’ comments. The lexicons, though, only provide us with a subset of expected (thus acceptable) terms included in commentary—terms that afford students the opportunity to act upon receiving them via revision or transfer. Directions for Future Research: Additional research is necessary to expand and refine the lexicons and their impact on student writing. One possibility is to return to the current data set to engage in additional lexical analysis of both the novice and principled lexicons as well as the overall frequency tables to understand how terms are used in the context of response by the various instructor groups. Differences in the application of the terms might help us understand why comments might be labeled as more or less helpful to writers.  Another strategy is to examine the data in terms of markers of stance; finally, topic modeling could be used to locate more subtle differences in the instructor comments that are not as easily identifiable with lexical analysis. Such examinations could serve as a baseline for broadening the study out to other sets of assignments and commentary, perhaps helping us build a set of threshold concepts for talking about writing with our students. Ultimately, it is important to replicate and expand Anson and Anson’s survey to other stakeholder groups. As with much research on the teaching of writing, we default to the group most accessible to us—other writing professionals. Replicating this survey with other stakeholders—graduate teaching assistants, undergraduate students at both lower and upper division levels— could help us understand whether or not a gap exists in understanding what constitutes good feedback from the various stakeholders.

This article provides an overview of the ways in which data and text mining have potentialas research methodologies in composition studies. It introduces data mining in thecontext of the field of composition studies and discusses ways in which this methodologycan complement and extend our existing research practices by blending the best of whattechnology and researchers have to offer. The authors examine a process model for datamining, discuss benefits and liabilities, and link to increased calls for accountability.

Discusses how adding a greater technological element into the composing and distribution of dissertations forces educators to consider multiple issues, some new and some that are only brought to the forefront because this electronically assisted or integrated process will make previously tacit behaviors on the part of both students and faculty explicit.