Description
LLVM's lit utility includes a builtin implementation of the env command in llvm/utils/lit/lit/TestRunner.py; see here. When the env command is used without a subcommand argument, the behavior is to print the (possibly augmented) environment to stdout. Lit's builtin implementation does this, but further processing of the command pipeline is then skipped.
The existing llvm/utils/lit/tests/shtest-env-positive.py test (in conjunction with llvm/utils/lit/tests/Inputs/shtest-env-positive/env-no-subcommand.txt) appears to exercise this situation, but the test actually passes spuriously. This can be demonstrated in several ways. First by observing that "executed command" log output is not generated for commands that follow the env command in a pipeline, and second by augmenting the test in a way that should cause the test to fail, but doesn't.
To demonstrate the issue, here first is an invocation of the llvm/utils/lit/tests/shtest-env-positive.py test (on Linux) that shows that "execute command" log output should appear for each command in the pipeline. Note that the command pipeline to run in this case is of the form env ... python ... | FileCheck ... and the log output reflects both commands from the pipeline.
$ llvm-lit -a llvm/utils/lit/tests/shtest-env-positive.py
...
# RUN: at line 3
env -u FILECHECK_OPTS "/usr/bin/python3.9" /iusers/thonerma/llvm-project/llvm/utils/lit/lit.py -j1 --order=lexical -a -v Inputs/shtest-env-positive | FileCheck -match-full-lines /localdisk2/thonerma/build-llvm-project-main-Release-On/utils/lit/tests/shtest-env-positive.py
# executed command: env -u FILECHECK_OPTS /usr/bin/python3.9 /iusers/thonerma/llvm-project/llvm/utils/lit/lit.py -j1 --order=lexical -a -v Inputs/shtest-env-positive
# executed command: FileCheck -match-full-lines /localdisk2/thonerma/build-llvm-project-main-Release-On/utils/lit/tests/shtest-env-positive.py
...
Total Discovered Tests: 1
Passed: 1 (100.00%)
The following demonstrates an independent run of the llvm/utils/lit/tests/Inputs/shtest-env-positive/env-no-subcommand.txt test. Note that the output does not include "executed command" logs for the FileCheck portions of the pipelines in that test. The test exercises five distinct scenarios but I've only included output from the first. Note that the command to run in the pipeline is of the form env | FileCheck ..., but the log output only reflects the env command from the pipeline; the FileCheck command is skipped.
$ llvm-lit -a llvm/utils/lit/tests/Inputs/shtest-env-positive/env-no-subcommand.txt
...
# RUN: at line 4
env | FileCheck -check-prefix=NO-ARGS /localdisk2/thonerma/llvm-project/llvm/utils/lit/tests/Inputs/shtest-env-positive/env-no-subcommand.txt
# executed command: env
# .---command stdout------------
...
# `-----------------------------
# RUN: at line 11
...
The following demonstrates that a modification to the test that should cause it to fail does not do so. Apply the following diff (or make similar changes) and observe that the test still passes.
$ git diff
diff --git a/llvm/utils/lit/tests/Inputs/shtest-env-positive/env-no-subcommand.txt b/llvm/utils/lit/tests/Inputs/shtest-env-positive/env-no-subcommand.txt
index 761a8061a0b0..f17eeb01c2c2 100644
--- a/llvm/utils/lit/tests/Inputs/shtest-env-positive/env-no-subcommand.txt
+++ b/llvm/utils/lit/tests/Inputs/shtest-env-positive/env-no-subcommand.txt
@@ -3,9 +3,9 @@
## Check default environment.
# RUN: env | FileCheck -check-prefix=NO-ARGS %s
#
-# NO-ARGS: BAR=2
-# NO-ARGS: FOO=1
-# NO-ARGS: QUX=3
+# NO-ARGS: BAR=This can be anything!
+# NO-ARGS: FOO=So can this!
+# NO-ARGS: QUX=And this!
## Set environment variables.
# RUN: env FOO=2 BAR=1 | FileCheck -check-prefix=SET-VAL %s
$ llvm-lit llvm/utils/lit/tests/Inputs/shtest-env-positive/env-no-subcommand.txt
-- Testing: 1 tests, 1 workers --
PASS: shtest-env :: env-no-subcommand.txt (1 of 1)
Testing Time: 0.01s
Total Discovered Tests: 1
Passed: 1 (100.00%)