[lit] Make a separate tmpdir for every test in the suite

[ocelaiwo]

I would like every test in the test suite to have its own temporary directory so that is it easier to track execution artifacts of each test. The suggested change is the simplest one that can be made and it should not break anything since tests should not rely on sharing the temporary directory. We can also make this change configurable so that users can specify desired behavior in their local lit config.

[github-actions]

Thank you for submitting a Pull Request (PR) to the LLVM Project!

This PR will be automatically labeled and the relevant teams will be notified.

If you wish to, you can add reviewers by using the "Reviewers" section on this page.

If this is not working for you, it is probably because you do not have write permissions for the repository. In which case you can instead tag reviewers by name in a comment by using @ followed by their GitHub username.

If you have received no comments on your PR for a week, you can request a review by "ping"ing the PR by adding a comment “Ping”. The common courtesy "ping" rate is once a week. Please remember that you are asking for valuable time from other developers.

If you have further questions, they may be answered by the LLVM GitHub User Guide.

You can also ask questions in a comment on this PR, on the LLVM Discord or on the forums.

[llvmbot]

@llvm/pr-subscribers-testing-tools

Author: Aleksei Babushkin (ocelaiwo)

Changes

I would like every test in the test suite to have its own temporary directory so that is it easier to track execution artifacts of each test. The suggested change is the simplest one that can be made and it should not break anything since tests should not rely on sharing the temporary directory. We can also make this change configurable so that users can specify desired behavior in their local lit config.

---

Full diff: https://github.com/llvm/llvm-project/pull/153422.diff

1 Files Affected:

• (modified) llvm/utils/lit/lit/TestRunner.py (+1-1)

diff --git a/llvm/utils/lit/lit/TestRunner.py b/llvm/utils/lit/lit/TestRunner.py
index e7cd70766a3dd..2755bd7154533 100644
--- a/llvm/utils/lit/lit/TestRunner.py
+++ b/llvm/utils/lit/lit/TestRunner.py
@@ -1369,7 +1369,7 @@ def getTempPaths(test):
     root, not test source root."""
     execpath = test.getExecPath()
     execdir, execbase = os.path.split(execpath)
-    tmpDir = os.path.join(execdir, "Output")
+    tmpDir = os.path.join(execdir, "Output", f"{execbase}.dir")
     tmpBase = os.path.join(tmpDir, execbase)
     return tmpDir, tmpBase
 

[jh7370]

This feels like a big departure from how the LLVM tests currently work and probably warrants a Discourse thread to discuss it, as I suspect there may be implications that I've not thought of.

That being said, I'm not sure I necessarily see the point in changing this. Lit already has a %t substitution which expands to a path name that is unique to the test. This gives you two options:

1. Start a test with something like # RUN: rm -r %t && mkdir %t && cd %t, meaning the rest of the test will run inside a unique directory.
2. Create all of your files with names prefixed with %t, e.g. %t1, %t2 etc. These will end up in the same directory as other test artifacts, but it'll be easy to see them, because they have a name derived from the test name.

[ocelaiwo]

Thank you for the fast reply. I understand your concerns about changing this. The options you described require changing the existing testcases (and thinking about it when writing new ones). Also, if we decide to change the behavior, we can also add timestamps to temporary directories (test.dir.1, test.dir.2, ...) so that we don't have to explicitly save test artifacts before running lit again, which can be useful when, for example, debugging flaky tests etc. So, if you think the thing is not worth a discussion, please let me know so that I close the PR. Otherwise, how should I start a Discourse thread?

[jh7370]

Thank you for the fast reply. I understand your concerns about changing this. The options you described require changing the existing testcases (and thinking about it when writing new ones). Also, if we decide to change the behavior, we can also add timestamps to temporary directories (test.dir.1, test.dir.2, ...) so that we don't have to explicitly save test artifacts before running lit again, which can be useful when, for example, debugging flaky tests etc. So, if you think the thing is not worth a discussion, please let me know so that I close the PR. Otherwise, how should I start a Discourse thread?

Most existing tests should already be using one of the two mechanisms I mentioned (and if they aren't, it's a potential test bug), so there should be no need to change them.

The LLVM Discourse forum is available at https://discourse.llvm.org/. I'd recommend posting a new thread under the LLVM Project, with the topic prefixed with "[RFC]".