Enhancing ChatGPT-based writing research through effective prompt use

An increasing number of studies have investigated how ChatGPT can aid in
written assessment and feedback provision. However, many studies overlook
its conversational design and underlying architecture, raising concerns about
the reliability and validity of their analytical outputs. Therefore, applying first
principles thinking to prompt use, and through a case study approach, we
explore some of the challenges in using ChatGPT to identify logical fallacies
in student essays. Specifically, we examine the outputs from two kinds of
interactions: (1) multiple iterations of a carefully designed prompt across new
conversation windows, and (2) a single-shot application of the designed prompt
followed by two rounds of follow-up prompts. Results show that combining
outputs from multiple iterations of the same prompt can enhance precision and
accuracy, and that follow-up prompts can generate a more exhaustive analysis,
but there is also a risk of diminishing validity with excessive follow-ups.
Therefore, to increase the validity and reliability of generative-AI outputs, we
argue that in addition to rigorous prompt design, researchers and teachers
need to systematically pilot and tailor the use of prompts to their specific
research contexts or task needs. With this in mind, we provide some preliminary
guidelines for prompt use.