TestART: Improving LLM-based Unit Testing via Co-evolution of Automated Generation and Repair Iteration

TestART is an innovative coverage-driven automated unit testing framework that leverages Large Language Models (LLMs) to generate high-quality unit tests for Java code. The framework employs template-based code repair techniques and iterative feedback mechanisms to continuously improve test case quality and coverage.

Key Features

• Template-based error correction for LLM-generated test cases
• Coverage-guided iterative test generation
• Automated test case refinement based on execution feedback
• Support for both branch and line coverage metrics

Project Components

The repository contains:

• Frontend UI interface
• Backend service implementation
• Docker container for easy deployment
• Experimental results and datasets

Architecture Overview

The workflow consists of the following key components:

1. Dataset parsing and method extraction (utils/dataset.py)
2. Task allocation (allocator.py)
3. Initial test suite generation (tasks/unit_test.py)
4. Test suite repair and iteration (utils/fix.py, utils/iter.py)

Getting Started

Prerequisites

Required Resources

Before starting, download the necessary files:

1. Docker Image:
   • Download from Baidu Drive
   • Extraction code: 5ped
2. Dataset:
   • Download from Figshare

You can either run TestART locally or using Docker.

Local Setup

• Python 3.11
• Maven 3.9.6
• Java 8
• PostgreSQL 16
• TinyDB
• Vue
• Required Python dependencies

Configure Prefect:

prefect config set PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://postgres:yourPassWord@localhost:5432/prefect"
prefect config set PREFECT_API_URL="http://127.0.0.1:4200/api"
prefect config set PREFECT_UI_API_URL="http://127.0.0.1:4200/api"

Start the backend Prefect server(TestART-Java):

prefect server start

Start the front-end Vue server(Prefect-ui):

1.Project setup

npm ci

2.Compiles and hot-reloads for development

npm run serve

3.Compiles and minifies for production

npm run build

4.Lints and fixes files

npm run lint

5.Components

Many components in the Prefect UI come from the prefect ui library.

6.Customize configuration

See Configuration Reference.

Docker Setup

1. Load the image:

docker load -i llmtest4j.tar

2. Run the container:

docker run -p 4200:4200 -p 5173:5173 -p 25734:25734 -e URL=${URL} -it llmtest4j:1.0

Usage

Dataset Upload

Access the web interface and upload your dataset through the Dataset menu.The port number of this page is 5173

Running Tests

Execute tests using:

python main.py --api_key xxxxx --api_base http://xxx.xx/v1 --dataset_name dat --dataset_start_index 0 --dataset_end_index 1

Monitoring Progress

Viewing Results

Generated test files are saved in ~/GPT-Java-Tester/result.

Quality Assessment

We use PIT (Pitest) for mutation testing to evaluate test suite quality. Key metrics include:

• Test Strength: Ratio of killed mutations to covered mutations
• Mutation Coverage: Ratio of killed mutations to total mutations
• Line Coverage: Percentage of code lines covered by tests

Configuration includes:

• 4 threads for parallel execution
• DEFAULT mutators
• Custom filter for public methods
• HTML output format

Notes

• Datasets can be single .java files or .zip archives with src/main/java structure
• Pre-generated test cases are available through our provided dataset link
• Use --dataset_end_index 99999 to process all methods in a dataset