Journal of Writing Analytics

3 articles
Year: Topic: Clear
Export:
affect and writing ×

January 2018

  1. Going Beyond �That was fun�: Measuring Writing Motivation
    Abstract

    Aim:  The use of validated measures of writing motivation is imperative to improving our understanding and development of interventions to improve student writing utilizing motivation as a mechanism.  One of the most important malleable factors involved in improving student writing is motivation, particularly for secondary school students.  This research note systematically examines the measures of writing motivation for students in grades 4–12 used by researchers over the last ten years and summarizes their psychometric and measurement properties to the extent provided in the underlying literature. This collection of measures and their properties and features is designed to make researchers more aware of the various options and to point out the need for additional measures. Problem Formation:  Writing is crucial to college and career readiness, but adolescents are inadequately prepared to be proficient writers.  Grades 4–12, once students have generally learned the basics of writing, are when students begin to develop more fluent and sophisticated writing abilities.  They turn from learning to write to writing to learn, and writing is increasingly done across content areas and in multiple genres.  Unfortunately, writing is a difficult skill to master, and students in middle and high school suffer from declining motivation.  The ability to measure changes in writing motivation at this developmental stage will allow researchers to more effectively design and assess writing interventions.  What are the current, validated measures of writing motivation available for researchers working with adolescents?  Motivation research has grown significantly in the last ten years, and a variety of motivation constructs (e.g., self-efficacy, expectancy-value) and related measures are used across the field. In addition to the variety of motivation constructs used in research today, researchers require domain- or context-specific measures of motivation (e.g., science motivation) to enable an accurate understanding of the role of motivation in achievement. Despite increased developments in both motivation and writing research over the past few decades, the intersection of these two fields remains relatively unexplored (Boscolo & Hidi, 2007; Troia, Harbaugh, Shankland, Wolbers, & Lawrence, 2013).Information Collection:  A thorough literature search was done to find measures of writing motivation used for this age group within the last 10 years.  Psychometric properties, to the extent available in the underlying articles, of each measure are described.Conclusions:  Ultimately, seven discrete measures of adolescent writing motivation were found, but only limited psychometric details were available for many of the measures.  No “gold standard” measure was found; indeed, the measures utilized varied motivational constructs and rarely reported more than the Cronbach’s alpha of the underlying instrument. Researchers need to carefully parse through the related motivation literature to understand the most likely constructs to be implicated in their intervention. They need to consider factors specifically related to their study, such as how stable the construct being targeted is developmentally, whether the term and type of intervention will be sufficient to make an impact on the students’ motivation as suggested by the underlying motivational literature, and what the target of the intervention is. Appropriate motivational constructs to be measured will vary depending on the intervention and its anticipated theory of change.Directions for Further Research: Several underlying motivation constructs have been used in the measures described in this review, particularly self-efficacy. However, a number of important motivation constructs, such as interest and self-determination theory, were not captured by the measures found.  This review of currently available measures will give researchers options when wanting to include validated measures of writing motivation in their studies and suggests that additional, validated measures are needed to adequately cover the relevant motivational constructs.

    doi:10.37514/jwa-j.2018.2.1.10

January 2017

  1. I Hear What You�re Saying: The Power of Screencasts in Peer-to-Peer Review
    Abstract

    Aim: The screencast (SC), a 21st century analytics tool, enables the simultaneous recording of audio and video feedback on any digital document, image, or website, and may be used to enhance feedback systems in many educational settings. Although previous findings show that students and teachers have had positive experiences with recorded commentary, this method is still rarely used by teachers in composition classrooms. There are many possible reasons for this, some of which include the accelerated pace at which classroom technology has changed over the past decade, concerns over privacy when new technologies are integrated into the classroom, and the general unease instructors may feel when asked to integrate a new technology system into their established composition pedagogy and response routine. The aim of this study was to replicate previous findings in favor of SC feedback and expand that body of research beyond instructor-to-student SC interactions and into the realm of SC-mediated peer review. Thus, this study seeks to improve on the widespread written peer review practices most common among writing instruction today, practices that tend to produce mediocre learning outcomes and fail to capitalize on 21st century technological innovations to enhance student learning. This research note demonstrates the validity of SC as a valuable writing analytics research tool that has the potential to collect and measure student learning. It also seeks to inspire those who have been reluctant to adopt SC in both digital learning and face-to-face educational environments by providing pragmatic guidance for doing so in ways that simultaneously increase student learning and facilitate a more rigorous and discursive peer-to-peer review process. Problem Formation: While research suggests positive student perceptions related to screencast instructor response, results in peer-to-peer screencast response are mixed. After several successful years of experience in instructor-to-student SC feedback, the author wondered what would happen if she asked students to use screencast technology to mediate peer review. How might students’ attitudes and perceptions impact the use of peer-to-peer screencast technology in the composition classroom? In order to address these questions, the author developed a survey measuring the user reliability of this new SC technology and the student affect and revision initiative it produces. Information Collection: This study extends Anson’s (2016) research and insights by reporting findings from a study of 138 writing students. Survey data was collected during the 2015-2016 academic year at three institutions. At High Point University, the author of this research note asked freshmen composition students in a traditional face-to-face lecture course to conduct a series of peer review sessions (including both traditional written comments and SC comments) over a 16-week semester. Students were surveyed after each peer review experience, and the results form the foundation of this research note’s conclusions. In addition to survey responses, researchers also collected the screencasts exchanged among peer-to-peer interactions within each educational setting. Conclusions: The author provides an in-depth analysis of students’ experiences, perceptions, and attitudes toward giving and receiving screencast feedback, focusing on the impact of this method on student revision initiative in comparison to that of a traditional written feedback system. Some conclusions are also drawn regarding the user reliability and effectiveness of the screencast technology, specifically the free software program known as Jing, a product available through Techsmith.com that enables a streamlined and user-friendly SC interface and cloud storage of all SC recordings through individualized hyperlinks, thereby alleviating concerns regarding student privacy. Directions for Further Research: While this research note provides compelling evidence to support the use of SC in composition classrooms, there are also many opportunities for continued study, particularly within the emerging field of writing analytics. While the actual student-to-student screencasts were collected in this study, they were not analyzed as a qualitative data set, and the researchers relied on self-reported survey data to assess the degree of revision initiative among the students surveyed. The screencasts themselves offer a treasure trove of data, should the researcher have the capability to code that data set or utilize automated natural language processing programs in the future. Perhaps this peer-to-peer SC feedback could be compared to similar corpus analyses of instructor-to-student feedback gathered by other writing analytics scholars. In addition, further research in this area could also collect the student writing itself and track revisions made by students after receiving SC feedback and traditional written feedback from their peers. In this way, researchers would be able to make comparisons between the actual changes made by the student writers, the extent of those changes (surface-level or higher-order revisions), and the student’s perceived degree of revision initiative reported in the survey. To facilitate future research in this area, the author has included teaching resources for those new to screencast technology and analytics.

    doi:10.37514/jwa-j.2017.1.1.13
  2. A Text Analytic Approach to Classifying Document Types
    Abstract

    Background: While it is commonly recognized that almost every work and research discipline utilize their own taxonomy, the language used within a specific discipline may also vary depending on numerous factors, including the desired effect of the information being communicated and the intended audience. Different audiences are reached through publication of information, including research results, in different types of publication outlets such as newspapers, newsletters, magazines, websites, and journals. Prior research has shown that students, both undergraduate and graduate, as well as faculty may have a difficult time locating information in different publication outlet types (e.g., magazines, newspapers, journals). The type of publication may affect the ease of understanding and also the confidence placed in the acquired information. A text analytics tool for classifying the source of research as a newsletter (used as a substitute for newspaper articles), a magazine, or an academic journal article has been developed to assist students, faculty, and researchers in identifying the likely source type of information and classifying their own writings with respect to these possible publication outlet types.  Literature Review: Literature on information literacy is discussed as this forms the motivation for the reported research. Additionally, prior research on using text mining and text analytics is examined to better understand the methodology employed, including a review of the original Scale of Theoretical and Applied Research system, adapted for the current research. Research Questions: The primary research question is: Can a text mining and text analytics approach accurately determine the most probable publication source type with respect to being from a newsletter, magazine, or journal? Methodology: A text mining and text analytics algorithm, STAR’ (System for Text Analytics-based Ranking), was developed from a previously researched text mining tool, STAR (Scale of Theoretical and Applied Research), that was used to classify the research type of articles between theoretical and applied research. The new text mining method, STAR’, analyzes the language used in manuscripts to determine the type of publication. This method first mines all words from corresponding publication source types to determine a keyword corpus. The corpus is then used in a text analytics process to classify full newsletters, magazine articles, and journal articles with respect to their publication source. All newsletters, magazine articles, and journal articles are from the library and information sciences (LIS) domain. Results: The STAR’ text analytics method was evaluated as a proof of concept on a specific LIS organizational newsletter, as well as articles from a single LIS magazine and a single LIS journal. STAR’ was able to classify the newsletters, magazine articles, and journal articles with 100% accuracy. Random samples from another similar LIS newsletter and a different LIS journal were also evaluated to examine the robustness of the STAR’ method in the initial proof of concept. Following the positive results of the proof of concept, additional journal, magazine, and newsletter articles were used to evaluate the generalizability of STAR’. The second-round results were very positive for differentiating journals and newsletters from other publication types, but revealed potential issues for distinguishing magazine articles from other types of publications. Discussion: STAR’ demonstrates that the language used for transferring information within a specific discipline does differ significantly depending on the intended recipients of the research knowledge. Further work is needed to examine language usage specific to magazine articles. Conclusions: The STAR’ method may be used by students and faculty to identify the likely source of research or discipline-specific information. This may improve trust in the reliability of information due to different levels of rigor applied to different types of publications. Additionally, the STAR’ classifications may be used by students, faculty, or researchers to determine the most appropriate type of outlet and correspondingly the most appropriate type of audience for the reported information in their own manuscripts, thereby improving the chance for successful sharing of information to appropriate audiences who will deem the information to be reliable, through publication in the most relevant outlet type.

    doi:10.37514/jwa-j.2017.1.1.06