UC Irvine

Large language models (LLMs) have demonstrated strong capabilities on a wide range of code-related tasks. When performing such a task, an LLM must first understand the given input, then produce an output based on that understanding. Depending on the task, this output can take the form of a label or free-form text such as source code. For example, in clone detection, the LLM takes two code snippets as input and produces a label indicating whether they are semantically equivalent. To give a correct label, the LLM must understand two code snippets and have the correct understanding of what is semantically equivalent. In code generation, the LLM must understand the natural-language specification and generate source code that fulfills those requirements. Despite their strong performance, LLMs can make mistakes at either stage (understanding and generation), which can undermine users' trust. Therefore, understanding and mitigating LLMs' mistakes at both stages is key to improving their performance on code-related tasks.Among various code understanding tasks, we select clone detection because it is widely used to support software maintenance activities, including plagiarism detection, refactoring, and bug identification. The first work aims to understand LLMs' mistakes in clone detection and improve their performance. Existing work has mainly evaluated LLMs' performance on clone detection but has not systematically investigated the underlying reasons for their misclassifications. To investigate this, we manually analyzed the rationale for each LLM's misclassification and found that LLMs' understanding of clone code differs from that of human annotators. For example, we found that LLMs often fail to recognize Type-4 clones, code fragments that are functionally equivalent but syntactically dissimilar, as clones, revealing a narrower notion of clone code than the one intended by human annotators. In total, we identified eight distinct mistakes LLMs made during clone detection. For each mistake, we derived corresponding natural-language guidance that can be inserted into the prompt to mitigate LLMs' errors. We evaluated the effectiveness of the proposed guidance and found that it can improve the LLM's performance on clone detection, achieving up to a 10.81% improvement in F1 score.As the first work focuses on the LLM's mistakes in the code understanding stage, the second work focuses on the mistakes during the generation stage. Among various code-related generation tasks, we selected code generation due to its popularity and importance, as writing source code is one of the most fundamental activities in software engineering. Despite LLMs' strong performance on code generation, they still produce incorrect code. Thus, the second work aims to understand the mistakes in the generated code. Prior work analyzing mistakes in LLM-generated code has largely relied on earlier, less capable models and small-scale, Python-only datasets, thereby restricting the scope of the identified mistakes. Furthermore, prior work does not investigate the reasons for those mistakes. To fill these gaps, we analyze code generated by two state-of-the-art LLMs across two large-scale, multilingual benchmarks comprising 2,268 standalone coding questions in Python and Java. Through extensive manual analysis, we identified 17 types of non-syntactic mistakes, organized into seven high-level categories, ten of which were overlooked by prior literature. We further identified six underlying reasons for these mistakes. Finally, we assessed LLMs' ability to automatically identify mistakes in generated code and the underlying reasons for these mistakes. Results indicate that top-performing models like GPT-4 achieve high precision of 0.95 in detecting mistakes and an F1 score of up to 0.78 in identifying the underlying reasons. These findings enriched the taxonomy of mistakes in LLM-generated code, providing developers with practical guidance for using LLMs more effectively and a deeper understanding of the reasons behind these mistakes.While the second work focuses on mistakes in generated code and their underlying reasons, it does not propose a way to improve LLM performance. The third work aims to do so. Instead of focusing on standalone code generation, the third work focuses on repository-level code generation, as it better reflects real-world software development, where new code is written within the context of an existing codebase. Learning from our second work, context plays a central role in successful code generation, especially for repository-level code generation, as the generated code may call existing functions in the repository or be called by them. Therefore, the third work aims to improve repository-level code generation by identifying better contexts. Existing work often retrieves contexts based on lexical or semantic similarity to the function to be completed, overlooking functions that share similar logic. We refer to these overlooked contexts as procedurally similar contexts. To fill this gap, we propose ProjAgent, a system that retrieves procedurally similar contexts through agentic search over the codebase. These contexts serve as the necessary logic to complete the target function. Furthermore, ProjAgent retrieves semantically similar contexts from the codebase that serve as the ingredients for implementing the logic. Our evaluation shows that ProjAgent improves code generation performance by up to 41.14% over baseline approaches.In summary, the contributions in this dissertation advance the understanding of LLMs' mistakes in code-related tasks and their underlying reasons, and propose methods to improve LLM performance on these tasks.