Reland "[LLVM] Add IRNormalizer Pass"

[justinfargnoli]

IRNormalizer will reorder instructions. Thus, we need to invalidate analyses. Done in cd500d2. This should resolve the BuildBot failure.

---

Original PR: #68176
Original commit: 1295d2e
Reverted with: 8a12e01

---

Add the llvm-canon tool. Description from the original PR:

Added a new llvm-canon tool which aims to transform LLVM Modules into a canonical form by reordering and renaming instructions while preserving the same semantics. This tool makes it easier to spot semantic differences while diffing two modules which have undergone different transformation passes.

The current version of this tool can:

• Reorder instructions within a function.
• Rename instructions based on the operands.
• Sort commutative operands.

This code was originally written by @michalpaszkowski and submitted to mainline LLVM. However, it was quickly reverted to do BuildBot errors.

Michal presented his version of the tool in LLVM-Canon: Shooting for Clear Diffs.

@AidanGoldfarb and I ported the code to the new pass manager, added more tests, and fixed some bugs related to PHI nodes that may have been the root cause of the BuildBot errors that caused the patch to be reverted. Additionally, we rewrote the implementation of instruction reordering to fix cases where the original algorithm would break use-def chains.

Note that this is @AidanGoldfarb and I's first time submitting to LLVM. Please liberally critique the PR!

CC @plotfi for initial review.

[github-actions]

✅ With the latest revision this PR passed the C/C++ code formatter.

[vvereschaka]

@justinfargnoli ,

I wasn't able to build and check your branch (TheManWithTheGoldenCanon:main) locally on the builder because of these linker errors:

 /home/buildbot/worker/temp/llvm-project/llvm/include/llvm/IR/PassManager.h:111: error: undefined reference to 'llvm::PreservedFunctionHashAnalysis::Key'
/home/buildbot/worker/temp/llvm-project/llvm/include/llvm/IR/PassManager.h:111: error: undefined reference to 'llvm::PreservedModuleHashAnalysis::Key'
collect2: error: ld returned 1 exit status

Would you check your changes once again? Probably you need to sync your branch with the llvm-project:main top-of-the-tree.

UPDATE: @justinfargnoli
the problem is still there. This is the full error log:

[3186/3834] Linking CXX executable bin/llvm-extract
FAILED:bin/llvm-extract
: && /usr/bin/c++ -U_GLIBCXX_DEBUG -Wno-misleading-indentation -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -fno-lifetime-dse -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-uninitialized -Wno-nonnull -Wno-class-memaccess -Wno-redundant-move -Wno-pessimizing-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wno-misleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -g -fuse-ld=gold -Wl,--gdb-index tools/llvm-extract/CMakeFiles/llvm-extract.dir/llvm-extract.cpp.o -o bin/llvm-extract  -Wl,-rpath,"\$ORIGIN/../lib:"  lib/libLLVMAnalysis.a  lib/libLLVMBitWriter.a  lib/libLLVMCore.a  lib/libLLVMipo.a  lib/libLLVMIRReader.a  lib/libLLVMIRPrinter.a  lib/libLLVMPasses.a  lib/libLLVMSupport.a  lib/libLLVMIRPrinter.a  lib/libLLVMCFGuard.a  lib/libLLVMCodeGen.a  lib/libLLVMCGData.a  lib/libLLVMCodeGenTypes.a  lib/libLLVMCoroutines.a  lib/libLLVMipo.a  lib/libLLVMBitWriter.a  lib/libLLVMFrontendOpenMP.a  lib/libLLVMFrontendOffloading.a  lib/libLLVMLinker.a  lib/libLLVMVectorize.a  lib/libLLVMSandboxIR.a  lib/libLLVMInstrumentation.a  lib/libLLVMScalarOpts.a  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a  lib/libLLVMHipStdPar.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMTransformUtils.a  lib/libLLVMTarget.a  lib/libLLVMAnalysis.a  lib/libLLVMProfileData.a  lib/libLLVMSymbolize.a  lib/libLLVMDebugInfoPDB.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMDebugInfoBTF.a  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMObject.a  lib/libLLVMIRReader.a  lib/libLLVMAsmParser.a  lib/libLLVMBitReader.a  lib/libLLVMMCParser.a  lib/libLLVMTextAPI.a  lib/libLLVMCore.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMMC.a  lib/libLLVMBinaryFormat.a  lib/libLLVMTargetParser.a  lib/libLLVMSupport.a  lib/libLLVMDemangle.a  -lrt  -ldl  -lm  /usr/lib/x86_64-linux-gnu/libz.so && :
/usr/bin/ld.gold: warning: lib/libLLVMScalarOpts.a(CorrelatedValuePropagation.cpp.o): top level DIE is not DW_TAG_compile_unit or DW_TAG_type_unit
/usr/bin/ld.gold: warning: lib/libLLVMScalarOpts.a(LoopFuse.cpp.o): top level DIE is not DW_TAG_compile_unit or DW_TAG_type_unit
/usr/bin/ld.gold: warning: lib/libLLVMScalarOpts.a(WarnMissedTransforms.cpp.o): top level DIE is not DW_TAG_compile_unit or DW_TAG_type_unit
/home/buildbot/worker/temp/llvm-project/llvm/include/llvm/IR/PassManager.h:111: error: undefined reference to 'llvm::PreservedFunctionHashAnalysis::Key'
/home/buildbot/worker/temp/llvm-project/llvm/include/llvm/IR/PassManager.h:111: error: undefined reference to 'llvm::PreservedModuleHashAnalysis::Key'
collect2: error: ld returned 1 exit status
[3201/3834] Building X86GenAsmMatcher.inc...
[3209/3834] Building AMDGPUGenAsmMatcher.inc...
ninja: build stopped: subcommand failed.

[justinfargnoli]

@vvereschaka I'm unable to reproduce the build failure locally or via the buildkite jobs.

Are the jobs that you're running on the BuildBot clean builds?

[vvereschaka]

@vvereschaka I'm unable to reproduce the build failure locally or via the buildkite jobs.

Are the jobs that you're running on the BuildBot clean builds?

@justinfargnoli in few words the problem is in the broken link order for those two Key references.

You have rearranged the declarations/definitions for PreservedFunctionHashAnalysis[::Key] and PreservedModuleHashAnalysis[::Key]. They became the public and global vars.
The implementations of those Keys are now sitting inside of lib/libLLVMPasses.a, but the references to them are inside of lib/libLLVMTransformUtils.a. The lib/libLLVMTransformUtils.a library is processing after lib/libLLVMPasses.a during the linking stage (see the error log above) and these references remain unresolved.
You need to move the PreservedFunctionHashAnalysis::Key and PreservedModuleHashAnalysis::Key back into lib/libLLVMTransformUtils.a (as example into llvm/lib/Transforms/Utils/IRNormalizer.cpp or any other inside of Transform/Utils).
Or fix the linking order within the CMake configuration files.

UPDATE: no unexpected test failures with these changes:

Total Discovered Tests: 65557
  Skipped          :    17 (0.03%)
  Unsupported      :  1259 (1.92%)
  Passed           : 64114 (97.80%)
  Expectedly Failed:   167 (0.25%)

[aeubanks]

I believe the failure reported in the original bot is only under expensive checks (-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON)

[justinfargnoli]

@justinfargnoli in few words the problem is in the broken link order for those two Key references.

@vvereschaka thank you for your detail description of the issue.

I've attempted to address the:

• Build issues with Port PreservedFunctionHashAnalysis and PreservedModuleHashAnalysis to Analysis.
• Expensive checks test failure with Abandon PreservedFunctionHashAnalysis and PreservedModuleHashAnalysis in IRNormalizer.

I've verified locally that the expensive checks test failure is resolved. I think I've also verified that the build error is resolved. However, could you double-check on my behalf?

[vvereschaka]

Thank you @justinfargnoli ,
sure, I'll check these changes tomorrow.

[boomanaiden154]

Is it possible/easy to split ef6a751 into a new PR? I think that would make things easier to go through and keep the commit history clean/make it easy to discern the motivation from a git blame.

[justinfargnoli]

Is it possible/easy to split ef6a751 into a new PR?

Certainly, @boomanaiden154! Please see #116108 :)

[vvereschaka]

@justinfargnoli ,
I have checked the latest changes locally on the ubuntu expensive builder. They works fine there, thank you. LGTM.

[nikic]

The pass invalidation changes look good to me, I didn't look at the rest.

[llvm-ci]

LLVM Buildbot has detected a new failure on builder clang-ppc64-aix running on aix-ppc64 while building llvm at step 3 "clean-build-dir".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/64/builds/1464

Here is the relevant piece of the build log for the reference

Step 3 (clean-build-dir) failure: Delete failed. (failure) (timed out)
Step 6 (test-build-unified-tree-check-all) failure: test (failure)
******************** TEST 'Clang :: ClangScanDeps/verbose.test' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 1: rm -rf /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/tools/clang/test/ClangScanDeps/Output/verbose.test.tmp
+ rm -rf /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/tools/clang/test/ClangScanDeps/Output/verbose.test.tmp
RUN: at line 2: split-file /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/clang/test/ClangScanDeps/verbose.test /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/tools/clang/test/ClangScanDeps/Output/verbose.test.tmp
+ split-file /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/clang/test/ClangScanDeps/verbose.test /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/tools/clang/test/ClangScanDeps/Output/verbose.test.tmp
RUN: at line 3: sed -e "s|DIR|/home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/tools/clang/test/ClangScanDeps/Output/verbose.test.tmp|g" /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/tools/clang/test/ClangScanDeps/Output/verbose.test.tmp/cdb.json.in > /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/tools/clang/test/ClangScanDeps/Output/verbose.test.tmp/cdb.json
+ sed -e 's|DIR|/home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/tools/clang/test/ClangScanDeps/Output/verbose.test.tmp|g' /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/tools/clang/test/ClangScanDeps/Output/verbose.test.tmp/cdb.json.in
RUN: at line 5: /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/bin/clang-scan-deps -compilation-database /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/tools/clang/test/ClangScanDeps/Output/verbose.test.tmp/cdb.json -v -o /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/tools/clang/test/ClangScanDeps/Output/verbose.test.tmp/result.json 2>&1 | /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/bin/FileCheck /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/clang/test/ClangScanDeps/verbose.test
+ /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/bin/clang-scan-deps -compilation-database /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/tools/clang/test/ClangScanDeps/Output/verbose.test.tmp/cdb.json -v -o /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/tools/clang/test/ClangScanDeps/Output/verbose.test.tmp/result.json
+ /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/build/bin/FileCheck /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/clang/test/ClangScanDeps/verbose.test
/home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/clang/test/ClangScanDeps/verbose.test:6:11: error: CHECK: expected string not found in input
// CHECK: *** Virtual File System Stats:
          ^
<stdin>:1:1: note: scanning from here
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
^
<stdin>:1:8: note: possible intended match here
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
       ^

Input file: <stdin>
Check file: /home/powerllvm/powerllvm_env/aix-ppc64/clang-ppc64-aix/llvm-project/clang/test/ClangScanDeps/verbose.test

-dump-input=help explains the following input dump.

Input was:
<<<<<<
           1: PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace. 
check:6'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
check:6'1            ?                                                                                                     possible intended match
>>>>>>

--

********************