Abstract

Content analysis is a common research method in technical and professional communication (TPC) journals. But TPC content analyses often lack intercoder reliability (ICR) statistics, possibly due to the lack of resources required to train human coders. This proof-of-concept study explores the viability of replacing a second coder with a generative AI tool to calculate an ICR. Using three previously published studies, I calculated a Krippendorff's alpha for various data types and various codebooks using non-modified versions of five popular generative AI tools. While the tools could code TPC data, most tools did not produce alphas strong enough to replace human coders. While it is premature in most cases to replace human coders with generative AI tools to calculate an ICR, generative AI may prove to be useful to researchers as a means of codebook and unit of analysis refinement prior to human coding.

Journal
Communication Design Quarterly
Published
2025-03-01
DOI
10.1145/3718959.3718962
CompPile
Search in CompPile ↗
Open Access
OA PDF Gold
Topics
Export

Citation Context

Cited by in this index (1)

  1. Technical Communication Quarterly

References (24) · 5 in this index

  1. International Journal of Advanced Computer Science and Applications, 9.
  2. 10.1007/978-3-031-42682-7_3
  3. 10.1093/oxfordhb/9780195399813.013.023
  4. IEEE Transactions on Professional Communication
  5. 10.36834/cmej.72504
Show all 24 →
  1. 10.1007/s00146-023-01715-z
  2. International Journal of Internet Science, 8(1).
  3. Journal of Business and Technical Communication
  4. Technical Communication Quarterly
  5. IEEE Transactions on Professional Communication
  6. Statistical methods for practice and research: A guide to data analysis using SPSS
  7. 10.37514/PRA-B.2019.0230
  8. Empathy in patient care: antecedents, development, measurement, and outcomes
    Vol. 77
  9. C. Bazerman & P
  10. 10.1111/j.1468-2958.2002.tb00826.x
  11. Content analysis: An introduction to its methodology
  12. 10.4135/9781071802878
  13. How ChatGPT and other LLMs work—and where they could go next. Wired. https://www.wired.co…
  14. 10.1177/1609406919899220
  15. https://www.interaction-design.org/literature/topics/prompt-engineering
  16. How does ChatGPT really work? New York Times. https://www.nytimes.com/2023/03/28/technolo…
  17. Mass communication research: Contemporary methods and applications
  18. Journal of Technical Writing and Communication
  19. Toner H. (2023). What are generative AI large language models and foundational models? Center for Security an…