Enhancing ChatGPT-Based Writing Research Through Effective Prompt Use

Journal article


Authors/Editors


Strategic Research Themes


Publication Details

Author listBowen N.E.J.A.; Watson Todd R.

Publication year2025

Volume number25

Issue number1

Start page26

End page40

Number of pages15

URLhttps://www.scopus.com/inward/record.uri?eid=2-s2.0-105009427007&doi=10.56297%2Fvaca6841%2FOZFH1876%2FSFKW4394&partnerID=40&md5=f34b659ec060d4b7509c6ad799193f30

LanguagesEnglish-Great Britain (EN-GB)


View on publisher site


Abstract

An increasing number of studies have investigated how ChatGPT can aid in written assessment and feedback provision. However, many studies overlook its conversational design and underlying architecture, raising concerns about the reliability and validity of their analytical outputs. Therefore, applying first principles thinking to prompt use, and through a case study approach, we explore some of the challenges in using ChatGPT to identify logical fallacies in student essays. Specifically, we examine the outputs from two kinds of interactions: (1) multiple iterations of a carefully designed prompt across new conversation windows, and (2) a single-shot application of the designed prompt followed by two rounds of follow-up prompts. Results show that combining outputs from multiple iterations of the same prompt can enhance precision and accuracy, and that follow-up prompts can generate a more exhaustive analysis, but there is also a risk of diminishing validity with excessive follow-ups. Therefore, to increase the validity and reliability of generative-AI outputs, we argue that in addition to rigorous prompt design, researchers and teachers need to systematically pilot and tailor the use of prompts to their specific research contexts or task needs. With this in mind, we provide some preliminary guidelines for prompt use. © 2025 Elsevier B.V., All rights reserved.


Keywords

No matching items found.


Last updated on 2025-24-09 at 00:00