Chain-of-Thought and Chain-of-Verification Prompting for Grammar-based Test Case Generation

Aditi, Sang-Ki Ko

http://doi.org/10.5626/JOK.2025.52.1.29

Software testing is an essential but cost-intensive work in the software development process. Automatic test case generation tools are utilized to distinguish between the correct and the incorrect solutions more effectively than manually generating them. Many researchers have recently proposed deep learning-based methods to generate test cases automatically for given logical specifications of problems or programs. In this work, we propose teaching the large language models (LLMs) such as ChatGPT and Google Gemini to generate ‘test case grammars’ from problem specifications, particularly using the chain-of-thought (CoT) prompting. Additionally, we implemented it using the CoT to verify and by providing the details of generalized rules to the LLMs, termed “chain-of-verification” (CoVe). We further evaluate our method with the publicly available dataset, DeepMind CodeContests dataset, which consists of numerous programming problems ranging from beginner to advanced level and is submitted by programming students with test cases for verifying the correctness of programs.