- Main
An Analysis of Students’ One-Shot Prompting Behaviors with GitHub Copilot in Large Code Bases
- Rexin, Thomas
- Advisor(s): Soosai Raj, Gerald
Abstract
Large language models (LLMs) are extensively used by software developers in industry and knowing how to best utilize LLMs as tools is an important skill for upper-division students about to enter industry. Prior work has shown that some students interact with LLMs using one- shot prompting, where students ask for the entire task implementation at once. One-shot prompting behavior could be detrimental to student learning and cause an overreliance on LLMs but there has not been much work investigating how students specifically construct one-shot prompts as well as how students’ interactions with a LLM continue after a one-shot prompt. This study analyzes the interactions of upper-division students who performed one-shot prompting with GitHub Copilot when working within a large code base. Through an open-coding qualitative analysis, this study found that students construct one-shot prompts in a variety of ways beyond copying and pasting the problem description as found in prior work. Some of the one-shot prompting behaviors identified in this work such as decomposing the task and one-shot prompting after asking comprehension questions align with developer practices from prior work. However, other one-shot prompting behaviors such as copying and pasting the problem description are similar to the behavior of novice programmers. A similar analysis on the students’ prompts after a one-shot prompt revealed two main ways in which students respond to Copilot responses to their one-shot prompt which are not correct: asking for debugging help or re-prompting in an attempt to get an entirely new implementation. An in-depth case study analysis of some students’ complete chat transcripts supported the existence of these behaviors and further revealed how these interaction patterns can extend throughout multiple interactions with Copilot. This study takes some first steps towards a more informed understanding of how upper-division students one-shot prompt with GitHub Copilot in a large code base and how these interaction behaviors relate to those of professional developers and novice programmers.