Journal of Academic Writing Apr 2025 OA PDF

Reliability of Large Language Models for Identifying and Classifying Content in Research Articles

Kristin Terrill Iowa State University ; Elena Cotos Iowa State University

Abstract

GenAI has demonstrated functionality that seems, uncannily, to parallel reading and writing by identifying/reformulating information from source texts and generating novel content and argumentation. These skills are essential yet challenging for many students tasked with producing literature reviews. This study takes the first steps to investigating the feasibility of a GenAI-facilitated literature review. This investigation starts from the ‘human-in-the-loop’ position that complex processes can be deconstructed and compartmentalized, and that component functions needed for these processes can be delegated to machines while humans contribute to, or control, the overall process. We explore the hypothesis that certain functions of the literature review process, such as information extraction and content classification, might be able to be automated. Prompts modeled on recommended practices for research synthesis were designed to identify and classify particular types of content in research articles. Outputs produced by two GenAI models, GPT-3.5 and GPT-4o, were assessed for reliability with a human coder. Overall, the results posit concerns about the models’ performance on this task, cautioning against direct uses of GenAI output as learning scaffolding for students developing literature review skills.

Journal
Journal of Academic Writing
Published
2025-04-16
DOI
10.18552/joaw.v15is2.1129
CompPile
Search in CompPile ↗
Open Access
OA PDF Diamond
Topics
argument artificial intelligence literacy studies
Export
BibTeX RIS

Citation Context

Cited by in this index (0)

No articles in this index cite this work.

Cites in this index (0)

No references match articles in this index.

CrossRef global citation count: 0 View in citation network → Build reading path →

Related Articles

  1. Business and Professional Communication Quarterly Dec 2025
    From Chatbot to Classroom: Developing Critical Thinking and Evaluative Judgment With AI ↗
    Daneshwar Sharma; Shiva Kakkar; Ashima Agrawal
    writing pedagogy service learning argument professional writing artificial intelligence literacy studies
  2. College Composition and Communication Sep 2025
    From an Unsettled Middle: A Critical-Ethical Stance for GenAI-Engaged Writing Assignments ↗
    Christopher Basgier; Lydia Wilkes
    first-year composition writing pedagogy writing across the curriculum argument writing centers artificial intelligence literacy studies
  3. College Composition and Communication Sep 2025
    From Cheating to Cheat Codes: Integrating Generative AI Ethics into Collaborative Learning ↗
    Kristi Girdharry
    modern rhetorical theory rhetorical criticism writing pedagogy argument collaborative writing writing centers artificial intelligence literacy studies
  4. The Peer Review Sep 2025
    The Impact of Working at a Writing Center in Brazil: Perspectives of Student Tutors ↗
    Ana Raquel Fialho Ferreira Campos; João Tiago Gaspar Cozechen; Elaine Pereira Lustosa; Marcos Angel De Carvalho Eing; Leonardo Schimiloski
    first-year composition writing pedagogy graduate education argument writing centers professional writing qualitative research artificial intelligence multilingual writers literacy studies race and writing
  5. Written Communication Jul 2025
    Does ChatGPT Write Like a Student? Engagement Markers in Argumentative Essays ↗
    Feng (Kevin) Jiang; Ken Hyland
    discourse analysis writing pedagogy argument artificial intelligence literacy studies