Missing files for running the code

[drixs2050]

Hello there, it seems that I cannot reopen the previous post I made, so I had to open a new issue.
I think I am still missing data.extract_all_methods_and_tests file, this file is called in: rq1/assistant_methods.py
Thank you!

[LeonYang95]

Hi @jundaz ,

Sorry we have lost the extract_all_methods_and_tests.py file. Probably because we used the evosuite support within defects4j to generate unit tests instead of using evosuite.jar. Hopefully you can remove the redundant import and the run_evosuite method in assistant_methods.py.

[drixs2050]

Hello @LeonYang95
Thank you very much for your quick response, I did notice that this method is not necessary and removed it.
I have a few other follow up question I would like to confirm:

1. I noticed that the number of focal methods included in source_data.jsonl you provided is a bit more than claimed in paper (1243 vs 778), Are there more focal methods included or just duplicates? Do you mind sharing a more specific list of the focal methods/defects/projects involved as mentioned in your paper?
2. In case of the backup file (source_data.jsonl) missing, I see that you read from the following file to gather the required information:
   with open( os.path.join( code_base, "data/fixed_projects_source/" + project_name + "/" + bug_id + "_fixed.jsonl", ), "r", encoding="utf-8",
   May I ask how these files are obtained?
3. For file import_v2, are these the imports you directly add as mentioned in the paper? I did notice that for each model (deepseek and codellama and all model size) you have separate import statements for each bug. I checked a few bugs' import statements under different models but they seem exactly the same, for example:
   "Lang_38_CodeLlama-13b-hf": { "fixed": [ "import org.apache.commons.lang3.*;", "import org.apache.commons.lang3.builder.*;", "import org.apache.commons.lang3.concurrent.*;", "import org.apache.commons.lang3.exception.*;", "import org.apache.commons.lang3.math.*;", "import org.apache.commons.lang3.mutable.*;", "import org.apache.commons.lang3.reflect.*;", "import org.apache.commons.lang3.text.*;", "import org.apache.commons.lang3.text.translate.*;", "import org.apache.commons.lang3.time.*;" ], "buggy": [ "import org.apache.commons.lang3.*;", "import org.apache.commons.lang3.builder.*;", "import org.apache.commons.lang3.concurrent.*;", "import org.apache.commons.lang3.exception.*;", "import org.apache.commons.lang3.math.*;", "import org.apache.commons.lang3.mutable.*;", "import org.apache.commons.lang3.reflect.*;", "import org.apache.commons.lang3.text.*;", "import org.apache.commons.lang3.text.translate.*;", "import org.apache.commons.lang3.time.*;" ] },
   and
   "Lang_38_WizardCoder-Python-34B-V1.0": { "fixed": [ "import org.apache.commons.lang3.*;", "import org.apache.commons.lang3.builder.*;", "import org.apache.commons.lang3.concurrent.*;", "import org.apache.commons.lang3.exception.*;", "import org.apache.commons.lang3.math.*;", "import org.apache.commons.lang3.mutable.*;", "import org.apache.commons.lang3.reflect.*;", "import org.apache.commons.lang3.text.*;", "import org.apache.commons.lang3.text.translate.*;", "import org.apache.commons.lang3.time.*;" ], "buggy": [ "import org.apache.commons.lang3.*;", "import org.apache.commons.lang3.builder.*;", "import org.apache.commons.lang3.concurrent.*;", "import org.apache.commons.lang3.exception.*;", "import org.apache.commons.lang3.math.*;", "import org.apache.commons.lang3.mutable.*;", "import org.apache.commons.lang3.reflect.*;", "import org.apache.commons.lang3.text.*;", "import org.apache.commons.lang3.text.translate.*;", "import org.apache.commons.lang3.time.*;" ] },
   If they should be exactly the same is there a specific reason for including them separately? Otherwise, how are they differentiated? And is there a specific version for chatgpt?
4. For the imports generated by LLM, I read the code in assemble_first_round_test_class and just want to confirm, are they added only on the condition that they exist in the src directory (so that they do not cause any import errors) and the ones that do not match the existing package in src are directly omitted?
5. is d4j_assistant file in data directory the same file as the source_data.jsonl you provided?
6. For rq3 are the tests generated based on the buggy code or the corrected code? if on buggy code, do you mind sharing the backup file (source_data.jsonl) for them?

Thank you very much for your time and I look forward to you response.
Again, really appreciate your work and your help!

[LeonYang95]

Here are the answers:

1. There are some private methods, which were excluded in the final version.
2. I have uploaded these files in your previous issue.
3. These were not used during the inference. It was used in the post-processing to help reduce the missing import issue.
4. The imports generated by LLMs are added into the constructed test class by default. However, if there exists an import xxx.xx.xx.*; and the LLM-generated imports are in the same package, we will omit the LLM-generated ones (since it was included in the *) to simplify the import statements. It is worth noting that this will not affect the compilation success rate since if the imported class does not exist, the code that uses that class will report a symbol (class) not found error.
5. The focal methods should be the same, but there maybe some minor differences in the json data. Unfortunately I forgot the details, but if some necessary key-value pairs are missing, there should be an error when running the code.
6. We used fixed version to generate unit tests for all experiments since it is the common way of using defects4j. That is, one test case that passed on the fixed version but failed on the buggy version reveals one defect. One more thing, the rq3 experiments were actually done using the rq1 script since there is no difference between the evaluation of rq1 and rq3. The scripts in rq3 folder are useless.

Hope these answers could help, if anything wrong, please let me know.

[drixs2050]

Thank you very much for you quick response! @LeonYang95
Sorry if I caused any misunderstanding here and here are some clarification and follow up questions:
For question 2 I was basically trying to ask how did you get the data in source_data.jsonl.zip, as I was trying to run some evaluation on other projects in defects4j (I.E. not in source_data.jsonl.zip). When I try to run the generation, the script tries to collect from path like this "LLM4UT/data/fixed_projects_source/Chart/Chart_10_fixed.jsonl", which is missing in data folder. I couldn't find such file in the checkout folder or any other place so I was wondering if you can share the script for how these file was obtained or point to where I can find these files.
For question 3 I understand that they are added in the post-processing, the reason I was asking is just because I did not see the import to add for chatgpt/gpt4 so I was wondering if there should be any difference in the import statement that was added for different models since I was making generation using GPT4 API.

Again, really appreciate the help and clarification you provide.

[LeonYang95]

Sorry for my misunderstanding.

As for the scripts, unfortunately, we are not able to share the script since it belongs to the company. But it should be easy to reproduce since we only used AST parsers, such as tree-sitter and JavaParser to extract those information.

To clarify the usage of import_v2.json file, we only used those that have project_name as key. For example,

  "Lang_12": {
      "fixed": [
          "import org.apache.commons.lang3.*;",
          "import org.apache.commons.lang3.builder.*;",
          "import org.apache.commons.lang3.concurrent.*;",
          "import org.apache.commons.lang3.event.*;",
          "import org.apache.commons.lang3.exception.*;",
          "import org.apache.commons.lang3.math.*;",
          "import org.apache.commons.lang3.mutable.*;",
          "import org.apache.commons.lang3.reflect.*;",
          "import org.apache.commons.lang3.text.*;",
          "import org.apache.commons.lang3.text.translate.*;",
          "import org.apache.commons.lang3.time.*;",
          "import org.apache.commons.lang3.tuple.*;"
      ],
      "buggy": [
          "import org.apache.commons.lang3.*;",
          "import org.apache.commons.lang3.builder.*;",
          "import org.apache.commons.lang3.concurrent.*;",
          "import org.apache.commons.lang3.event.*;",
          "import org.apache.commons.lang3.exception.*;",
          "import org.apache.commons.lang3.math.*;",
          "import org.apache.commons.lang3.mutable.*;",
          "import org.apache.commons.lang3.reflect.*;",
          "import org.apache.commons.lang3.text.*;",
          "import org.apache.commons.lang3.text.translate.*;",
          "import org.apache.commons.lang3.time.*;",
          "import org.apache.commons.lang3.tuple.*;"
      ]
  }

Also, in this line, we used project_name as key to collect imports.
So the answer to your question is no. In order to have a fair comparison, we have to keep them as the same for all studied LLMs.

I hope this can help, if anything wrong, please let me know.

[drixs2050]

That resolves all my current problems, I will close this issue for now, thank you very much!

[drixs2050]

Sorry to bother, another quick question, what is the difference between strategy in generationg prompts (I.E. generation vs extend)

[LeonYang95]

It is related to how the LLMs respond to your instructions (not prompts).
As it is under patent protection, I am not allowed to provide more details. You can refer to the prefilling mode of DeepSeek.

[drixs2050]

Hello @LeonYang95
Sorry to bother you again, I see that you have included

"import static org.mockito.Mockito.*;",
"import org.mockito.Mockito;",
"import static org.hamcrest.MatcherAssert.assertThat;",
"import static org.hamcrest.Matchers.*;",

as part of the predefined imports (varaible focal_imports) in file "utils/output_analyzer.py"
However it seems that this 2 package are not included after checking out from defects4j using the command given by defects4j (I.E. they weren't part of the original project), and will cause compilation error.
May I ask if you manually included them (for exaple changed maven) or is there a different way of checking out? If they were manually included is there a specific reason of doing so?
Thank you very much!

[LeonYang95]

Yes, they were intentionally added to the dependencies of the projects.

We want to make sure that the LLM-generated unit tests will not suffer from missing import issues due to the lack of some key packages. You can take this as a post-processing strategy that fixes missing imports to improve the compilability of generated test classes.

There are two steps to make sure that these dependencies can be successfully added. First, you need to download the jar file from maven repository sites and keep it somewhere. Second, please change the path to the jar files in dependency_analyzer.py. The program will automatically change the pom.xml file to add dependencies.

[drixs2050]

Got it, thanks for the quick response

[LeonYang95]

One more thing, we decided to publish a simple version of the code analyze tool and the inference scripts and this project will be refactored as an evaluation benchmark shortly.
Hopefully the new project can help you understand some of your previous questions, especially those that I could not respond before.

The project is published in UTGen_LLM repo.

You are very welcomed to use this project for unit test generation, this supports a project-level generation.
If you have any questions, please do not hesitate to ask!

[drixs2050]

Thanks a lot for sharing, I just starred it, I assume this can directly accept projects from defects4j?