Peer and AI Review + Reflection (PAIRR): A human-centered approach to formative assessment

Lisa Sperber University of California, Davis ; Marit MacArthur University of California, Davis ; Sophia Minnillo Center for Applied Linguistics ; Nicholas Stillman University of California, Davis ; Carl Whithaus University of California, Davis

Abstract

• AI can offer useful writing feedback when used combined with peer review. • AI and peer responses were often similar and mutually reinforcing. • When AI and peer responses differed, the perspectives were often complementary. • Evaluating AI feedback fostered student agency and AI literacy. Cycles of drafting and revising are crucial for student writers' growth, and formative assessment plays an important role. However, many teachers lack the time or resources to provide feedback on drafts. While research suggests that AI feedback is high enough quality to be used for draft feedback, especially when assignment-specific criteria are used (Steiss et al., 2024), it must be used in a human-centered process. AI has the potential to reduce educational equity gaps in writing support (Warschauer et al., 2023), but when narrowly implemented, technologies can deepen divides (Stornaiuolo, et al., 2023). Peer and AI Review + Reflection (PAIRR) combines peer review best practices with AI review in an approach that emphasizes student agency and reflection. Using a mixed methods approach, this study examined student perceptions of AI utility in the context of peer review. Results indicate that AI tools offer useful feedback when combined with peer review. Students found the similarity between AI and peer feedback reassuring, while also valuing their complementary perspectives. Moreover, by evaluating AI outputs, students developed AI literacy, gaining familiarity with AI feedback's affordances and limitations while learning ethical ways to use AI in their writing processes.

Journal
Computers and Composition
Published
2025-06-01
DOI
10.1016/j.compcom.2025.102921
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
Export

Citation Context

Cited by in this index (1)

  1. College Composition and Communication

References (77) · 9 in this index

  1. Exploring student-writers’ views on replacing teacher feedback with peer feedback and com…
    Arab World English Journal  
  2. Research in the Teaching of English
  3. Anson, I.G., Anson, C.M., & Andrews, K.L. (2023). Teachers’ beliefs about the language of peer review: Survey…
  4. Bai, X., Wang, A. Sucholutsky, I. and Griffiths, T.L. (2024). Measuring implicit bias in explicitly unbiased …
  5. The future of intelligent tutoring systems for writing
    Digital writing technologies in higher education
Show all 77 →
  1. The effectiveness of feedback for L1-English and L2-writing development: A meta-analysis
    ETS Research Report Series  
  2. Rethinking models of feedback for learning: The challenge of design
    Assessment & Evaluation in Higher Education  
  3. Teaching with AI: a practical guide to a new era of human learning
  4. More than a glitch: confronting race, gender, and ability bias in tech
  5. Assessing our assessments: A collective questioning of what students need-and get
    Journal of Basic Writing  
  6. Writing on the social network: digital literacy practices in social media's first decade
  7. Using LLMs as peer reviewers for revising essays
    TextGenEd: teaching with text generation technologies
  8. Truth telling: Critical Inquiries on LLMs and the corpus texts that train them
    Composition Studies
  9. College Composition and Communication
  10. Discriminating data: correlation, neighborhoods, and the new politics of recognition
  11. infer: An R package for tidyverse-friendly statistical inference
    Journal of Open Source Software  
  12. When artificial intelligence substitutes humans in higher education: The cost of loneline…
    Studies in Higher Education  
  13. Computers and Composition
  14. Can large language models provide feedback to students? A case study on ChatGPT
    2023 IEEE International Conference on Advanced Learning Technologies (ICALT)
  15. Impact of AI assistance on student agency
    Computers & Education  
  16. Downs, D. & Robertson, L. (2015). Threshold concepts in first-year composition. In L. Adler-Kassner & E. Ward…
  17. Beyond knowledge and skills
    Composition Forum
  18. Uses and limitations of automated writing evaluation software
    WPA-CompPile Research Bibliographies
  19. Response to student writing
  20. The influence of teacher and peer relationships on students’ classroom engagement and eve…
    Teachers College Record: The Voice of Scholarship in Education  
  21. Second language writing online: An update
    Language Learning & Technology  
  22. Beyond Chatbot-K: On large language models, “Generative AI,” and the rise of chatbots: An…
    Critical AI  
  23. Formative assessment and writing: A meta-analysis
    The Elementary School Journal  
  24. Utility in a fallible tool: A multi-site case study of automated writing evaluation
    The Journal of Technology, Learning and Assessment
  25. Writing technology: studies on the materiality of literacy
  26. Jacob, S., Tate, T., & Warschauer, M. (2023). Emergent AI-Assisted Discourse: Case study of a second language…
  27. Comparing generative AI and expert feedback to students’ writing: Insights from student t…
    Psychologie in Erziehung Und Unterricht
  28. Automated feedback generation for student project reports: A data-driven approach
    Journal of Educational Data Mining
  29. Jiang, L., Yu, S., & Wang, C. (2020). Second language writing instructors’ feedback practice in response to a…
  30. High-impact teaching practices foster a greater sense of belonging in the college classroom
    Journal of Further and Higher Education  
  31. Graduate students’ use of ChatGPT for academic text revision: Behavioral, cognitive, and …
    Journal of Second Language Writing  
  32. A guide to doing statistics in second language research using spss and
  33. What is AI literacy? Competencies and design considerations
    Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
  34. To give is better than to receive: The benefits of peer review to the reviewer's own writing
    Journal of Second Language Writing  
  35. MacArthur, M. (Sept. 28, 2023). “AI, expertise and the convergence of writing and coding.” Inside Higher Ed. …
  36. Navigating the new frontier of generative AI in peer review and academic writing
    Teaching and Generative AI: Pedagogical Possibilities and Productive Tensions
  37. Reconstructing response to student writing: a national study from across the curriculum
  38. Using LLMs to bring evidence-based feedback into the classroom: AI-generated feedback inc…
    Computers and Education: Artificial Intelligence
  39. The effects of trained peer review on EFL students’ revision types and writing quality
    Journal of Second Language Writing  
  40. MLA-CCCC Joint Task Force on Writing and AI. (2023a). Working paper 1: Overview of the issues, statement of p…
  41. MLA-CCCC Joint Task Force on Writing and AI. (2023b). Working paper 2: Generative AI and policy development. …
  42. MLA-CCCC Joint Task Force on Writing and AI. (2024). Working paper 3: Building a culture for generative AI li…
  43. Towards hybrid human-AI learning technologies
    European Journal of Education  
  44. Mollick, E.R., & Mollick, L. (2023). Assigning AI: Seven Approaches for Students, with Prompts (SSRN Scholarl…
  45. NCTE Position Statement on Machine Scoring. (2013, April). National Council of Teachers of English. https://n…
  46. Conceptualizing AI literacy: An exploratory review
    Computers and Education: Artificial Intelligence
  47. Effectiveness of automated writing evaluation systems in school settings: A systematic re…
    Journal of Computer Assisted Learning  
  48. O'Neill, P. (2015). All writers have more to learn. In L. Adler-Kassner & E. Wardle (Eds.), Naming what we kn…
  49. Journal of Technical Writing and Communication
  50. Automated essay scoring: A cross-disciplinary perspective
    Lawrence Erlbaum
  51. Journal of Writing Research
  52. An introduction to logistic regression analysis and reporting
    The Journal of Educational Research  
  53. OpenAI used Kenyan workers on less than $2 per hour to make ChatGPT less toxic
    Time
  54. Feedback: Focusing attention on engagement
    Studies in Higher Education  
  55. Is It O.K. to Be a Luddite?
    The New York Times
  56. Computers and Composition
  57. The coding manual for qualitative researchers
  58. Teacher-written feedback in L2 writing education: Insights into student tensions and responses
    Feedback Research in Second Language  
  59. College English
  60. Envisioning information access systems: What makes for good tools and a healthy web?
    ACM Transactions on the Web  
  61. Comparing the quality of human and ChatGPT feedback of students' writing
    Learning and Instruction  
  62. The platformization of writing instruction: Considering educational equity in new learnin…
    Review of Research in Education  
  63. Assessment in the age of artificial intelligence
    Computers and Education: Artificial Intelligence
  64. Can AI provide useful holistic essay scoring?
    Computers and Education: Artificial Intelligence
  65. Automated writing assessment in the classroom
    Pedagogies: An International Journal  
  66. The affordances and contradictions of AI-generated text for writers of English as a secon…
    Journal of Second Language Writing  
  67. Always already: Automated essay scoring and grammar checkers in college writing courses
    Machine scoring of student essays: truth and consequences
  68. Writing Task Force. (2019). Writing Task Force final report. University of California, Davis. https://academi…
  69. A rhetoric of reflection
  70. Intelligent enough? Artificial intelligence for online learners
    Journal of Educators Online
  71. Assessing Writing
  72. Assessing Writing