Background: Current research in composition and writing studies is concerned with issues of writing program evaluation and how writing tasks and their sequences scaffold students toward learning outcomes. These issues are beginning to be addressed by writing analytics research, which can be useful for identifying recurring types of language in writing assignments and how those can inform task design and student outcomes. To address these issues, this study provides a three-step method of sequencing, comparison, and diagnosis to understand how specific writing tasks fit into a classroom sequence as well as compare to larger genres of writing outside of the immediate writing classroom environment. By doing so, we provide writing program administrators with tools for describing what skills students demonstrate in a sequence of writing tasks and diagnosing how these skills match with writing students will do in later contexts. Literature Review: Student writing that responds to classroom assignments can be understood as genres, insofar as they are constructed responses that exist in similar rhetorical situations and perform similar social actions. Previous work in corpus analysis has looked at these genres, which helps us as writing instructors understand what kind of constructed responses are required of students and to make those expectations explicit. Aull (2017) examined a corpus of first-year undergraduate writing assignments in two courses to create “sociocognitive profiles” of these assignments. We analyze student writing that responds to similar writing tasks, but use a different corpus method that allows us to understand the tasks in both local and global contexts. By doing so, we gain confidence and depth in our understanding of these tasks, analyze how they sequence together, and are able to compare argumentative writing across institutions and contexts. Research Questions: Two questions guided our study: What is the trajectory of skills targeted by the sequence of tasks in the two first-year writing courses, as evidenced by the rhetorical strategies employed by the writers in successive assignments? Focusing on the final argument assignments, how similar are they to argumentative writing in other contexts, in terms of rhetorical profiles? Methodology: We first conducted a local analysis, in which we used a dictionary-based corpus method to analyze the rhetorical strategies used by writers in the first-year writing courses to understand how they built on each other to form a sequence. Having understood what skills students are demonstrating in a course, we then conducted a global analysis which calculated a “distance” between the first-year argument writing and a corpus of argument writing drawn from other contexts. Recognizing that there was a non-trivial distance, we then identified and evaluated the sources of the distance so that the writing tasks could be assessed or modified. Results: The local analysis revealed eight key rhetorical strategies that student writing exhibits between the two first-year writing courses. With this understanding, we then placed the argument writing in global contexts to find that the assignments in both courses differ somewhat from argument writing in other contexts. Upon analyzing this difference, we found that the first-year writing primarily differs in its usage of academic language, the personal register, assertive language, and reasoning. We suggest that these differences stem primarily from the rhetorical situation and learning objectives associated with first-year writing, as well as the sequencing of the courses. Discussion: The three-step method presented provides a means for writing program administrators to describe and analyze writing that students produce in their writing programs. We intend these steps to be understood as an iterative process, whereby writing programs can use these results to evaluate what rhetorical skills their students are exhibiting and to benchmark those against the program’s goals and/or other similar writing programs. Conclusions: By presenting these analyses together, we ultimately provide a cohesive method by which to analyze a writing program and benchmark students’ use of rhetorical strategies in relation to other argumentative contexts. We believe this method to be useful not only to individual writing programs, but to assessment literature broadly. In future research, we anticipate learning how this process will practically feed back into pedagogy, as well as understanding what placing writing tasks into a global context can tell us about genre theory.

Aim:  The use of validated measures of writing motivation is imperative to improving our understanding and development of interventions to improve student writing utilizing motivation as a mechanism.  One of the most important malleable factors involved in improving student writing is motivation, particularly for secondary school students.  This research note systematically examines the measures of writing motivation for students in grades 4–12 used by researchers over the last ten years and summarizes their psychometric and measurement properties to the extent provided in the underlying literature. This collection of measures and their properties and features is designed to make researchers more aware of the various options and to point out the need for additional measures. Problem Formation:  Writing is crucial to college and career readiness, but adolescents are inadequately prepared to be proficient writers.  Grades 4–12, once students have generally learned the basics of writing, are when students begin to develop more fluent and sophisticated writing abilities.  They turn from learning to write to writing to learn, and writing is increasingly done across content areas and in multiple genres.  Unfortunately, writing is a difficult skill to master, and students in middle and high school suffer from declining motivation.  The ability to measure changes in writing motivation at this developmental stage will allow researchers to more effectively design and assess writing interventions.  What are the current, validated measures of writing motivation available for researchers working with adolescents?  Motivation research has grown significantly in the last ten years, and a variety of motivation constructs (e.g., self-efficacy, expectancy-value) and related measures are used across the field. In addition to the variety of motivation constructs used in research today, researchers require domain- or context-specific measures of motivation (e.g., science motivation) to enable an accurate understanding of the role of motivation in achievement. Despite increased developments in both motivation and writing research over the past few decades, the intersection of these two fields remains relatively unexplored (Boscolo & Hidi, 2007; Troia, Harbaugh, Shankland, Wolbers, & Lawrence, 2013).Information Collection:  A thorough literature search was done to find measures of writing motivation used for this age group within the last 10 years.  Psychometric properties, to the extent available in the underlying articles, of each measure are described.Conclusions:  Ultimately, seven discrete measures of adolescent writing motivation were found, but only limited psychometric details were available for many of the measures.  No “gold standard” measure was found; indeed, the measures utilized varied motivational constructs and rarely reported more than the Cronbach’s alpha of the underlying instrument. Researchers need to carefully parse through the related motivation literature to understand the most likely constructs to be implicated in their intervention. They need to consider factors specifically related to their study, such as how stable the construct being targeted is developmentally, whether the term and type of intervention will be sufficient to make an impact on the students’ motivation as suggested by the underlying motivational literature, and what the target of the intervention is. Appropriate motivational constructs to be measured will vary depending on the intervention and its anticipated theory of change.Directions for Further Research: Several underlying motivation constructs have been used in the measures described in this review, particularly self-efficacy. However, a number of important motivation constructs, such as interest and self-determination theory, were not captured by the measures found.  This review of currently available measures will give researchers options when wanting to include validated measures of writing motivation in their studies and suggests that additional, validated measures are needed to adequately cover the relevant motivational constructs.

Aim: This research note narrates existing and continuing potential crossover between the digital humanities and writing studies. I identify synergies between the two fields’ methodologies and categorize current research in terms of four permutations, or “valences,” of the phrase “writing analytics.” These valences include analytics of writing , writing of analytics , writing as analytics , and analytics as writing . I bring recent work in the two fields together under these common labels, with the goal of building strategic alliances between them rather than to delimit or be comprehensive. I offer the valences as one heuristic for establishing connections and distinctions between two fields engaged in complementary work without firm or definitive discursive borders. Writing analytics might provide a disciplinary ground that incorporates and coheres work from these different domains. I further hope to locate the areas in which my current research in digital humanities, grounded in archival studies, might most shape writing analytics. Problem Formation: Digital humanities and writing studies are two fields in which scholars are performing massive data analysis research projects, including those in which data are writing or metadata that accompanies writing. There is an emerging environment in the Modern Language Association friendly to crossover between the humanities and writing studies, especially in work that involves digital methods and media. Writing analytics accordingly hopes to find common disciplinary ground with digital humanities, with the goal of benefitting from and contributing to conversations about the ethical application of digital methods to its research questions. Recent work to bridge digital humanities and writing studies more broadly has unfortunately focused more on territorial and usability concerns than on identifying resonances between the fields’ methodological and ethical commitments. Information Collection: I draw from a history of meta-academic literature in digital humanities and writing studies to review their shared methodological commitments, particularly in literature that recognizes and responds to pushback against the fields’ ostensible use of extra-disciplinary methods. I then turn to current research in both fields that uses and critiques computational techniques, which is most relevant to writing analytics’ articulated focus on massive data analysis. I provide a more detailed explanation, drawing from my categorization of this work, of the conversations in digital humanities surrounding the digital archives that enable data analysis. Conclusions: A review of past and current research in digital humanities and writing studies reveals shared attention to techniques for tokenizing texts at different scales for analysis, which is made possible by the curation of large corpora. Both fields are writing new genres to compose this analysis. In these genres, both fields emphasize process in their provisional work, which is sociocognitively repurposed in different rhetorical contexts. Finally, both fields recognize that the analytical methods they employ are themselves modes of composition and argumentation. An ethics of data transformation present in digital humanities, however, is largely absent from writing studies. This ethics comes to digital humanities from the influence of textual studies and archival studies. Further research in writing analytics might benefit from reframing writing corpora as archives—what Paul Fyfe (2017) calls a shift from “data mining” to “data archaeology”—in its analyses. This is especially true for analyses of text, which in particular foreground writing and analysis of writing as acts of transformation. Directions for Further Research: I recommend that future efforts to find crossover between digital humanities and writing studies do so by identifying their common values rather than trying to co-opt language and spaces or engaging in broad definitional work. I further provide a set of guiding principles that writing analytics might follow in order to pursue research that draws upon and contributes to both digital humanities and writing studies. These research projects might consider and account for the silences of writing corpora—unseen versions of documents, and documents’ elements not described in structured data—while attending to the silences that these efforts might in turn (re)produce.

Background: Contemporary research in composition studies emphasizes the constitutive power of genres. It also highlights the prevalence of the most common genre in students’ transition into advanced college writing, the argumentative essay. Consistent with most research in composition, and therefore most studies of general, first-year college writing, such research has primarily emphasized genre context. Other research, in international applied linguistics research and particularly English for Academic Purposes (EAP), has focused less on first-year writers but has likewise shown the frequent use of argumentative essays in undergraduate writing. Together, these studies suggest that the argumentative essay is represented more than other genres in early college writing development, and that any given genre favors particular discourse features in contrast with other genres students might write. A productive next step, but one not yet realized, is to bring these discussions together, in research that uses context-informed corpus analysis that investigates students’ assignment contexts and analyzes the discourse that characterizes the tasks and genres students write. This study offers an exploratory, context-informed analysis of argumentative and explanatory writing by first-year college writers. Based on the corpus findings, the article underscores discourse as an integral part of the sociocognitive practices embedded in genres, and accordingly considers new ways to conceptualize student writing genres and to inform instruction and assignment design. Research questions: Four questions guided the inquiry: What are the key discursive practices associated with annotated bibliographies and argumentative essays written by the same students in the same course? What are the key discursive practices associated with visual analyses and argumentative essays written by the same students in the same course? What are the key discursive practices associated with the two argumentative tasks in comparison with the two explanatory tasks? Finally, how might corpus-based findings inform the design of particular assignment tasks and genres in light of a range of writing goals? Methodology: The article outlines a context-informed corpus analysis of lexical and grammatical keywords in part-of-speech tagged writing by first-year college students across courses at a U.S. institution. Using information from assignment descriptions and rubrics, the study considers four projects that also represent two macro-genres: an annotated bibliography and a visual analysis, both part of the explanatory macro-genre, and two argumentative essays, both part of the argumentative macro-genre. Results: The corpus analysis identifies lexical and grammatical keywords in each of the four tasks as well as in the macro-genres of argumentative versus explanatory writing. These include generalized, interpersonal, and persuasive discourse in argumentative essays versus more specified, informational, and elaborated discourse in explanatory writing, regardless of course or task. Based on these findings, the article discusses the discursive practices prioritized in each task and each macro-genre. Conclusions: The findings, based on key discourse patterns in tasks within the same course and in macro-genres across courses, pose important questions regarding writing task design and students’ adaptation to different genres. The macro-genre keywords specifically inform exploratory sociocognitive “profiles” of argumentative and explanatory tasks, offered in the final section. These argument and explanation profiles strive to account for discourse patterns, genre networks, and purposes and processes—in other words, multiple aspects of habituated thinking and writing practices entailed in each one relative to the other. As discussed in the conclusion, the profiles aim to (1) underscore discourse patterns as integral to the work of genres, (2) highlight adaptive discourse strategies as part of students’ meta-language for writing, and (3) identify multiple, macro-level (e.g., audience), meso-level (paragraph- and section-level), and micro-level (e.g., discourse patterns) aspects of genres to help instructors identify and specify multiple goals for writing assignments.

Background: This study focuses on construct representation and inter-reader agreement and reliability in ePortfolio assessment of 1,315 writing portfolios. These portfolios were submitted by undergraduates enrolled in required writing seminars at the University of Pennsylvania (Penn) in the fall of 2014.  Penn is an Ivy League university with a diverse student population, half of whom identify as students of color. Over half of Penn’s students are women, 12% are international, and 12% are first-generation college students. The students’ portfolios are scored by the instructor and an outside reader drawn from a writing-in-the-disciplines faculty who represent 24 disciplines. The portfolios are the product of a shared curriculum that uses formative assessment and a program-wide multiple-trait rubric. The study contributes to scholarship on the inter-reader reliability and validity of multiple-trait portfolio assessments as well as to recent discussions about reconceptualizing evidence in ePortfolio assessment.  Research Questions: Four questions guided our study: What levels of interrater agreement and reliability can be achieved when assessing complex writing performances that a) contain several different documents to be assessed; b) use a construct-based, multi-trait rubric; c) are designed for formative assessment rather than testing; and d) are rated by a multidisciplinary writing faculty?   What can be learned from assessing agreement and reliability of individual traits? How might these measurements contribute to curriculum design, teacher development, and student learning? How might these findings contribute to research on fairness, reliability, and validity; rubrics; and multidisciplinary writing assessment? Literature Review: There is a long history of empirical work exploring the reliability of scoring highly controlled timed writings, particularly by test measurement specialists. However, until quite recently, there have been few instances of applying empirical assessment techniques to writing portfolios.  Developed by writing theorists, writing portfolios contain multiple documents and genres and are produced and assessed under conditions significantly different from those of timed essay measurement. Interrater reliability can be affected by the different approaches to reading texts depending on the background, training, and goals of the rater. While a few writing theorists question the use of rubrics, most quantitatively based scholarship points to their effectiveness for portfolio assessment and calls into question the meaningfulness of single score holistic grading, whether impressionistic or rubric-based. Increasing attention is being paid to multi-trait rubrics, including, in the field of writing portfolio assessment, the use of robust writing constructs based on psychometrics alongside the more conventional cognitive traits assessed in writing studies, and rubrics that can identify areas of opportunity as well as unfairness in relation to the background of the student or the assessor. Scholars in the emergent field of empirical portfolio assessment in writing advocate the use of reliability as a means to identify fairness and validity and to create great opportunities for portfolios to advance student learning and professional development of faculty.  They also note that while the writing assessment community has paid attention to the work of test measurement practitioners, the reverse has not been the case, and that conversations and collaborations between the two communities are long overdue. Methodology: We used two methods of calculating interrater agreement: absolute and adjacent percentages, and Cohen’s Unweighted Kappa, which calculates the extent to which interrater agreement is an effect of chance or expected outcome. For interrater reliability, we used the Pearson product-moment correlation coefficient. We used SPSS to produce all of the calculations in this study.  Results: Interrater agreement and reliability rates of portfolio scores landed in the medium range of statistical significance.  Combined absolute and adjacent percentages of interrater reliability were above the 90% range recommended; however, absolute agreement was below the 70% ideal.  Furthermore, Cohen’s Unweighted Kappa rates were statistically significant but very low, which may be due to “kappa paradox.” Discussion: The study suggests that a formative, rubric-based approach to ePortfolio assessment that uses disciplinarily diverse raters can achieve medium-level rates of interrater agreement and reliability. It raises the question of the extent to which absolute agreement is a desirable or even relevant goal for authentic feedback processes of a complex set of documents, and in which the aim is to advance student learning. At the same time, our findings point to how agreement and reliability measures can significantly contribute to our assessment process, teacher training, and curriculum. Finally, the study highlights potential concerns about construct validity and rater training.  Conclusion: This study contributes to the emergent field of empirical writing portfolio assessment that calls into question the prevailing standard of reliability built upon timed essay measurement rather than the measurement, conditions, and objectives of complex writing performances.  It also contributes to recent research on multi-trait and discipline-based portfolio assessment.  We point to several directions for further research:  conducting “talk aloud” and recorded sessions with raters to obtain qualitative data on areas of disagreement; expanding the number of constructs assessed; increasing the range and granularity of the numeric scoring scale; and investigating traits that are receiving low interrater reliability scores. We also ask whether absolute agreement might be more useful for writing portfolio assessment than reliability and point to the potential “kappa paradox,” borrowed from the field of medicine, which examines interrater reliability in assessment of rare cases. Kappa paradox might be useful in assessing types of portfolios that are less frequently encountered by faculty readers. These, combined with the identification of jagged profiles and student demographics, hold considerable potential for rethinking how to work with and assess students from a range of backgrounds, preparation, and abilities.  Finally, our findings contribute to a growing effort to understand the role of rater background, particularly disciplinarity, in shaping writing assessment. The goals of our assessment process are to ensure that we are measuring what we intend to measure, specifically those things that students have an equal chance at achieving and that advance student learning.  Our findings suggest that interrater agreement and reliability measures, if thoughtfully approached, will contribute significantly to each of these goals.