Auditing genAI reasoning in qualitative analysis: A prompt-level case study

Simple item page
dc.contributor.authorBaidwan, Deepkhushi
dc.date.accessioned2026-04-16T22:31:36Z
dc.date.available2026-04-16T22:31:36Z
dc.date.issued2026
dc.description.abstractGenAI tools are increasingly used to assist qualitative researchers with tasks like transcript coding and theme generation. However, most evaluations focus only on whether GenAI outputs match human-produced codes, not on how the model arrived at those outputs. This study audits the reasoning process of NotebookLM, a retrieval-augmented generation (RAG) tool, across a structured 12-prompt sequence applied to eight interview transcripts on open pedagogy and institutional constraints. The audit logged 20 analytic decisions across two coding pathways (inductive and deductive), tracking triggers, rationale, and consequences at each stage. Four key observations emerged: prompt wording directly shaped coding behavior; supporting evidence was retrieved after coding decisions rather than driving them (a ""provenance gap""); ambiguous passages were automatically resolved without surfacing alternatives; and outputs visually resembled rigorous audit reports without being independently verifiable. The central finding is that GenAI outputs can perform methodological accountability without genuinely providing it, with significant implications for peer review, replication, and research integrity.
dc.description.reviewstatusReviewed
dc.description.scholarlevelUndergraduate
dc.description.sponsorshipJamie Cassels Undergraduate Research Awards (JCURA)
dc.identifier.urihttps://hdl.handle.net/1828/23610
dc.language.isoen
dc.publisherUniversity of Victoria
dc.subjectgenAI
dc.subjectqualitative coding
dc.subjectthematic analysis
dc.subjectprompt engineering
dc.subjectreasoning audit
dc.subjectresearch methodology
dc.subjectJamie Cassels Undergraduate Research Awards (JCURA)
dc.subject.departmentDepartment of Computer Science
dc.titleAuditing genAI reasoning in qualitative analysis: A prompt-level case study
dc.typePoster

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
baidwan_deepkhushi_jcura_poster_2026.pdf
Size:
3.15 MB
Format:
Adobe Portable Document Format
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.62 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

Jamie Cassels Undergraduate Research Awards (JCURA)
 
University of Victoria Libraries

We acknowledge and respect the Lək̓ʷəŋən (Songhees and Esquimalt) Peoples on whose territory the university stands, and the Lək̓ʷəŋən and W̱SÁNEĆ Peoples whose historical relationships with the land continue to this day.