Internal test coverage for deduplicated header units

[StephanTLavavej]

This ports the test coverage for deduplicated (topologically sorted) header units, added by #2516 for the Python-powered GitHub test harness, to the Perl-powered internal test harness. This will allow @cdacamar to avoid compiler regressions while working on modules. It will also inform us when compiler and library changes trigger assertions in the internal "chk" builds of the compiler.

In the Python code, we don't need sorted(set()) anymore. Originally, I combined header-units.json with the Standard's list of importable C++ library headers. This had the effect of adding version, but also added a whole bunch of duplicates (like algorithm, bitset, etc.), so sorted(set()) eliminated the duplicates and then put the list back into sorted order. Now, we're taking header-units.json and appending a small, completely disjoint list of headers. (It's okay that the resulting list isn't sorted; the order isn't important.)

Next, to avoid having so many files to update when we add new Standard headers, I'm extracting that list of importable C++ library headers to a JSON With Comments file. In the Python code, we need to update the comment in loadJsonWithComments() (as it's not specific to header-units.json anymore), then we add getImportableCxxLibraryHeaders(sourcePath). The sourcePath is the full path to test.cpp, so we can extract its directory name to get to the new file in the same folder.

The Perl code is a mostly direct translation of the Python code, including a rewrite of the classic independent header unit scenario. (Previously, the Perl code concatenated strings of compiler options. Now it builds arrays that are joined with spaces later, which is what the Python code does.) The comments are mostly identical (with a bit of variation marked as being for Perl), making it easier to read the source code side-by-side.

The internal port of this PR will contain a non-GitHub commit to re-add STL_INCLUDE_DIR to the environment variables that the internal test harness defines; this is the location of the STL's header files (and header-units.json which is in the same directory). I have verified that this passes internally and inspected the resulting command lines.

A minor bit of variation between the Python and Perl is how remainingDependencies is updated. In Perl, it's easier to do two separate passes - a first pass to eliminate headers that were just built, followed by a second step to filter the dependencies of remaining headers.

Finally, we're able to remove some compiler workarounds in the internal test harness. The Perl is always internal, so it can use "logical-name" in the /scanDependencies output - this is considerably simpler. (The Python will need to wait until VS 2022 17.2 Preview 2 ships.) Also, the test.cpp can disable workarounds for the topo-sort scenario when it's running internally, again until the fixes ship. Thanks again to @cdacamar for fixing these bugs.

⚠️ Notes to self:

• When merging this, include the non-GitHub commit for STL_INCLUDE_DIR.
• After merging this, update the Files To Edit When Adding Or Removing Files checklist.

[barcharcraz]

lg2m

[barcharcraz]

it kinda is a general purpose solution, it's not like a json lexer would do something else ..

[StephanTLavavej]

Well, as the comment explains, a json lexer would know where strings begin and end, so it wouldn't be confused by "cats // dogs" (or more realistically something like "c:/temp/dir1/dir2//file.txt"), whereas my regex here would immediately damage that.

[barcharcraz]

my $filename = shift(@_); is a little more idiomatic, but this is fine too. Personally I prefer just using signatures, but I have no idea if the perl in runall is sufficiently new for that.

Noting for posterity no need to redo the backend integration just for this.

[StephanTLavavej]

For the record, we're running Perl 5.26 (May 2017 🙀) which looks like it would support signatures.

[barcharcraz]

apparently faster to just write decode_json $jsonStr;

[StephanTLavavej]

Agreed. Although that would be shorter and faster, speed isn't important here (these JSON files are very small and the compilation process takes the vast majority of the time), and I felt that having this line look similar to the relaxed line made their near-symmetry more obvious. I suppose I could have just made it a comment.

[StephanTLavavej]

I'm mirroring this to the MSVC-internal repo - please notify me if any further changes are pushed. (I'll handle verifying that this works with the recent LLVM update.)