clang-repl output differences on Windows when piped into something else

[DavidSpickett]

Relates to #127467.

Where we saw the added test case failing on Windows on Arm and Windows x64.

https://lab.llvm.org/buildbot/#/builders/161/builds/6473
https://lab.llvm.org/buildbot/#/builders/63/builds/7023

Exit Code: 1
Command Output (stdout):
--
# RUN: at line 3
cat C:\Users\tcwg\llvm-worker\clang-arm64-windows-msvc\llvm-project\clang\test\Interpreter\lambda.cpp | c:\users\tcwg\llvm-worker\clang-arm64-windows-msvc\build\bin\clang-repl.exe | c:\users\tcwg\llvm-worker\clang-arm64-windows-msvc\build\bin\filecheck.exe C:\Users\tcwg\llvm-worker\clang-arm64-windows-msvc\llvm-project\clang\test\Interpreter\lambda.cpp
# executed command: cat 'C:\Users\tcwg\llvm-worker\clang-arm64-windows-msvc\llvm-project\clang\test\Interpreter\lambda.cpp'
# executed command: 'c:\users\tcwg\llvm-worker\clang-arm64-windows-msvc\build\bin\clang-repl.exe'
# .---command stderr------------
# | In file included from <<< inputs >>>:1:
# | input_line_22:1:17: error: non-local lambda expression cannot have a capture-default
# |     1 | auto capture = [&]() { return x * 2; };
# |       |                 ^
# | error: Parsing failed.
# `-----------------------------
# executed command: 'c:\users\tcwg\llvm-worker\clang-arm64-windows-msvc\build\bin\filecheck.exe' 'C:\Users\tcwg\llvm-worker\clang-arm64-windows-msvc\llvm-project\clang\test\Interpreter\lambda.cpp'
# RUN: at line 4
cat C:\Users\tcwg\llvm-worker\clang-arm64-windows-msvc\llvm-project\clang\test\Interpreter\lambda.cpp | c:\users\tcwg\llvm-worker\clang-arm64-windows-msvc\build\bin\clang-repl.exe -Xcc -Xclang -Xcc -verify -Xcc -O2 | c:\users\tcwg\llvm-worker\clang-arm64-windows-msvc\build\bin\filecheck.exe C:\Users\tcwg\llvm-worker\clang-arm64-windows-msvc\llvm-project\clang\test\Interpreter\lambda.cpp
# executed command: cat 'C:\Users\tcwg\llvm-worker\clang-arm64-windows-msvc\llvm-project\clang\test\Interpreter\lambda.cpp'
# executed command: 'c:\users\tcwg\llvm-worker\clang-arm64-windows-msvc\build\bin\clang-repl.exe' -Xcc -Xclang -Xcc -verify -Xcc -O2
# .---command stderr------------
# | error: Parsing failed.
# `-----------------------------
# executed command: 'c:\users\tcwg\llvm-worker\clang-arm64-windows-msvc\build\bin\filecheck.exe' 'C:\Users\tcwg\llvm-worker\clang-arm64-windows-msvc\llvm-project\clang\test\Interpreter\lambda.cpp'
# .---command stderr------------
# | C:\Users\tcwg\llvm-worker\clang-arm64-windows-msvc\llvm-project\clang\test\Interpreter\lambda.cpp:26:11: error: CHECK: expected string not found in input
# | // CHECK: x = 42
# |           ^
# | <stdin>:4:28: note: scanning from here
# | clang-repl> clang-repl> TWO
# |                            ^
# | <stdin>:5:9: note: possible intended match here
# | clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> 
# |         ^
# | 
# | Input file: <stdin>
# | Check file: C:\Users\tcwg\llvm-worker\clang-arm64-windows-msvc\llvm-project\clang\test\Interpreter\lambda.cpp
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |             1: x = 42 
# |             2: clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> ONE 
# |             3: clang-repl> clang-repl> TWO 
# |             4: clang-repl> clang-repl> TWO 
# | check:26'0                                X error: no match found
# |             5: clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl>  
# | check:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | check:26'1             ?                                                                                                                                                     possible intended match
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1
--
********************

The PR is correct and so is the test case, on Linux and in pre-commit Windows CI.

Somehow on these buildbots, the output changes order when piped into something:

C:\Users\tcwg\llvm-worker\lldb-aarch64-windows\build>cat C:\Users\tcwg\llvm-worker\lldb-aarch64-windows\llvm-project\clang\test\Interpreter\lambda.cpp | .\bin\clang-repl.exe -Xcc -Xclang -Xcc -verify -Xcc -O2
clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> ONE
clang-repl> clang-repl> TWO
clang-repl> clang-repl> TWO
clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> error: Parsing failed.
clang-repl> clang-repl> clang-repl> x = 42
clang-repl> clang-repl> clang-repl>

Note x = 42 is last, this is what we expect.

C:\Users\tcwg\llvm-worker\lldb-aarch64-windows\build>cat C:\Users\tcwg\llvm-worker\lldb-aarch64-windows\llvm-project\clang\test\Interpreter\lambda.cpp | .\bin\clang-repl.exe -Xcc -Xclang -Xcc -verify -Xcc -O2 | cat
error: Parsing failed.
x = 42
clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> ONE
clang-repl> clang-repl> TWO
clang-repl> clang-repl> TWO
clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl> clang-repl>

Somehow x = 42 has now moved?

Linux does not show any difference:

$ cat ../llvm-project/clang/test/Interpreter/lambda.cpp | ./bin/clang-repl -Xcc -Xclang -Xcc -verify -Xcc -O2
ONE
TWO
TWO
error: Parsing failed.
x = 42

$ cat ../llvm-project/clang/test/Interpreter/lambda.cpp | ./bin/clang-repl -Xcc -Xclang -Xcc -verify -Xcc -O2 | cat
error: Parsing failed.
ONE
TWO
TWO
x = 42

Though, note that it does not print a clang-repl> for every line. So perhaps on Windows we are not detecting whether the output is interactive?

Even so, if the ordering is correct, it should not matter.

I can "fix" the test by passing -O0 instead of -O2. -O1 still fails. This might be Arm specific, not tried it on x86.

I can also "fix" the test by making the first two prints have formatting arguments. Even if they are constants:

auto l1 = []() { printf("ONE %s\n", ""); return 42; };

This makes me suspect puts vs. printf but I would have thought we could compile both to puts anyway.

Another suspect is the piping mechanism, or the cat binary we are using. Which comes from an install of git for windows.

The problem reproduces outside of lit, so if we are using lit's internal terminal here, whatever it's doing, the real terminal is doing the same thing.

(I'm not sure which one lit is using tbh)

[DavidSpickett]

Disabled the second RUN line on Windows while we figure out this problem.

[hahnjo]

FWIW we are seeing similar issues between (suspected) puts and printf downstream in ROOT on Windows as well. We could never get to the bottom of it...

[DavidSpickett]

Interesting, with this test or something else?

fflush can also fix this:

diff --git a/clang/test/Interpreter/lambda.cpp b/clang/test/Interpreter/lambda.cpp
index db8c9db9b344..5838a79cb78b 100644
--- a/clang/test/Interpreter/lambda.cpp
+++ b/clang/test/Interpreter/lambda.cpp
@@ -6,6 +6,8 @@
 // RUN: %if !system-windows %{ cat %s | clang-repl -Xcc -Xclang -Xcc -verify -Xcc -O2 | FileCheck %s %}
 
 extern "C" int printf(const char *, ...);
+#define FILE void*
+extern "C" int fflush(FILE* stream);
 
 auto l1 = []() { printf("ONE\n"); return 42; };
 auto l2 = []() { printf("TWO\n"); return 17; };
@@ -16,6 +18,7 @@ auto r2 = l2();
 // CHECK: TWO
 auto r3 = l2();
 // CHECK: TWO
+fflush(0);
 
 // Verify non-local lambda capture error is correctly reported
 int x = 42;

Though I don't see why we'd need it. Even if stdout is fully buffered, the order of the printf calls should be the same.

[hahnjo]

Interesting, with this test or something else?

With a number of downstream tests that do approximately the same, effectively mixing printf and puts.

fflush can also fix this:
[...]
Though I don't see why we'd need it. Even if stdout is fully buffered, the order of the printf calls should be the same.

Yes, we observed the same and couldn't make sense of it either. If somebody can figure this out, we would be very interested to learn the underlying reason...

[bellenot]

IIRC, mixing std::cout << ..., fprintf(...), and printf(...) causes issues, and adding fflush didn't help. And if I'm not mistaken, the behavior between debug and release was also different...

[DavidSpickett]

This is the equivalent non-interpreted C++, and I didn't see it reordering the output when piped. Definitely turns the first printfs into puts though: https://godbolt.org/z/xr6cPYPM9

That likely does not represent exactly the resulting code from clang-repl though, depending on it's compilation model.

I also confirmed that removing the -Verify option did not change the behaviour.