Communication Design Quarterly Mar 2025

The Second Coder was a Robot: Generative AI Tools in Establishing Intercoder Reliability

Erin Friess

Abstract

Content analysis is a common research method in technical and professional communication (TPC) journals. But TPC content analyses often lack intercoder reliability (ICR) statistics, possibly due to the lack of resources required to train human coders. This proof-of-concept study explores the viability of replacing a second coder with a generative AI tool to calculate an ICR. Using three previously published studies, I calculated a Krippendorff's alpha for various data types and various codebooks using non-modified versions of five popular generative AI tools. While the tools could code TPC data, most tools did not produce alphas strong enough to replace human coders. While it is premature in most cases to replace human coders with generative AI tools to calculate an ICR, generative AI may prove to be useful to researchers as a means of codebook and unit of analysis refinement prior to human coding.

Journal
Communication Design Quarterly
Published
2025-03-01
DOI
10.1145/3718959.3718962
CompPile
Search in CompPile ↗
Topics
artificial intelligence professional writing

Citation Context

Cited by in this index (0)

No articles in this index cite this work.

Cites in this index (3)

  1. Friess (2023)
    Scrum in Classroom Collaborations: A Quasi-Experimental Study
    Journal of Business and Technical Communication
  2. Friess et al. (2021)
    “Dude, that Sucks”: Examining Scrum’s Influence on Empathy in Student Teams
    Technical Communication Quarterly
  3. Thayer et al. (2007)
    Content Analysis as a Best Practice in Technical Communication Research
    Journal of Technical Writing and Communication
Also cites 10 works outside this index ↓
  1. 10.1007/978-3-031-42682-7_3
     ↗
  2. 10.1093/oxfordhb/9780195399813.013.023
     ↗
  3. 10.1109/TPC.2010.2077450
     ↗
  4. 10.36834/cmej.72504
     ↗
  5. 10.1007/s00146-023-01715-z
     ↗
  6. 10.1109/TPC.2023.3314250
     ↗
  7. 10.37514/PRA-B.2019.0230
     ↗
  8. 10.1111/j.1468-2958.2002.tb00826.x
     ↗
  9. 10.4135/9781071802878
     ↗
  10. 10.1177/1609406919899220
     ↗
CrossRef global citation count: 1 View in citation network →

Related Articles

  1. Business and Professional Communication Quarterly Feb 2026
    LLMs in Composition: Theory, Ethics, and Implementation in the Workplace and Classroom ↗
    Kylie Alexis Wilczynski
    multimodality online writing instruction artificial intelligence professional writing
  2. Journal of Technical Writing and Communication Jan 2026
    Expanding Human-in-the-Loop: Critical Sensemaking for Technical and Professional Communication With Generative AI ↗
    Gustav Verhulsdonck; Jialei Jiang
    artificial intelligence professional writing multilingual writers
  3. Business and Professional Communication Quarterly Dec 2025
    Complexity of Purpose Revisited: AI-Assisted Cognition in Professional Communication ↗
    Daniel Plung
    artificial intelligence professional writing multilingual writers
  4. College Composition and Communication Sep 2025
    Using the AI Life Cycle to Unblackbox AI Tools: Teaching Résumé 2.0 with Résumé Analytics and Computational Job-Résumé Matching ↗
    Huiling Ding
    qualitative research artificial intelligence professional writing multilingual writers
  5. Business and Professional Communication Quarterly Aug 2025
    Welcome to the (Email) Machine: A Study of Chronemics and Source Cues in Managerial Communication ↗
    David Westerman; Justin Walden
    artificial intelligence professional writing multilingual writers affect and writing