[LTO] Setup LTO pipeline and swift-lto tool #32233

kateinoigakukun · 2020-06-07T16:57:17Z

Start to implement the LTO library and set up a basis of incremental testing tool named swift-lto. The swiftLTO library accepts SIB files for multi-modules, optimizes them at SIL level and emit LLVM bitcode. It will be dynamically loaded into linker to support language specific LTO.

CC: @compnerd

include/swift/Serialization/SerializedModuleLoader.h

lib/FrontendTool/FrontendTool.cpp

lib/LTO/LTO.cpp

compnerd · 2020-06-09T01:59:50Z

lib/Serialization/ModuleFormat.h

+  using RuntimeResourcePathLayout = BCRecordLayout<
+    RUNTIME_RESOURCE_PATH,
+    BCBlob
+  >;


Why serialize these? Why is it not sufficient to pass them along from the driver?

I see. it seems sufficient for this use.

I agree with compnerd here.

kateinoigakukun · 2020-06-09T12:59:51Z

@compnerd I want to confirm which commit style is easier to review.

Always squash commits even while under reviewing
Commit for each addressing change while under reviewing and squash them before merge.

compnerd · 2020-06-09T15:33:42Z

Its definitely easier to review (at least for me) when the change is squashed into a single change. However, any change that is not directly related (e.g. refactoring or renaming) can be pulled into a separate change that can be merged prior to the change itself being in review is helpful in keeping the size of the change low.

gottesmm

Meta comment. Do we really need an extra tool here? This is going to increase link times. Can you try and fit this into another tool? Maybe we can combine this with swift-llvm-gen? That is supposed to be a tool that maps *.sil files -> llvm-ir.

gottesmm · 2020-06-09T16:45:09Z

lib/Serialization/ModuleFormat.h

+  using RuntimeResourcePathLayout = BCRecordLayout<
+    RUNTIME_RESOURCE_PATH,
+    BCBlob
+  >;


I agree with compnerd here.

kateinoigakukun · 2020-06-09T17:09:08Z

@gottesmm Thanks for your comment. The dependent libraries are pretty much the same as sil-llvm-gen 👀 I'll combine them and make a symlink like swift-autolink-extract to separate the exec file.

kateinoigakukun · 2020-06-10T14:41:46Z

@gottesmm I updated to combine the tool with sil-llvm-gen 👍

lib/LTO/LTO.cpp

tools/sil-llvm-gen/swift_lto_main.cpp

compnerd · 2020-06-11T18:22:50Z

test/LTO/pipeline.swift

+
+// Examine loading order
+// RUN: cd %t && %swift-lto main.sib lib.sib
+// RUN: cd %t && %swift-lto lib.sib main.sib


This test doesn't actually check anything ... what is the purpose of the test?

This test checks that modules can be deserialized using the pipeline and ensure that the pipeline can resolve dependent modules regardless of input order.

I think Im not understanding something then. The test doesn't verify that the module are deserialized and the dependency modules are resolved. It just assumes that if the process exited everything is correct.

I see. added more checks

compnerd · 2020-06-11T18:24:03Z

tools/sil-llvm-gen/swift_lto_main.cpp

+    if (EC)
+      return std::unique_ptr<llvm::raw_ostream>(nullptr);
+    return RawOS;
+  });


Can you clarify why exact this tool is needed? This really feels like its executing sil-opt + swift-llvm-gen. Is there a reason that you cannot just pipe the tools in the test?

The tool accepts SIB files for multiple modules and optimizes them at once. This is something swift-llvm-gen does not do. If piping sil-opt and swift-llvm-gen, it's impossible to optimize across modules.

And the main purpose of this tool is simulating linker integration without linker. So it doesn't match sil-opt and swift-llvm-gen's purposes.

But you can load the SIB file in sil-opt, run it through the pipeline and write that out, then use swift-llvm-gen to generate IR from the SIB.

If piping sil-opt and swift-llvm-gen, it's impossible to optimize across modules.

I think Im not understanding the missing piece in the tools. Can you explain that and write that up in the commit message as well? I think its critical to explain the deficiency.

And the main purpose of this tool is simulating linker integration without linker. So it doesn't match sil-opt and swift-llvm-gen's purposes.

I disagree on this. Its not simulating linker integration - its running the SIL pipeline on a set of SIB files. That is not related to linking. Running the SIL pipeline is precisely what sil-opt does. You seem to generate LLVM IR from the binary at the end, and thats precisely what swift-llvm-gen does - generates LLVM IR from SIL/SIB. That is where the disconnect is, and what Im trying to understand.

As you say, this swift-lto and sil-opt have the same part of roles in optimizing SIBs. But the sil-opt also accept SIL as input, whereas the LTO pipeline only accepts SIBs. The LTO pipeline accepts inputs only from linkers, so SILs will never be inputted. If LTO pipeline accepts SILs as input, the LTO pipeline becomes more complex only for testing.

For example, SIL doesn't serialize SDKPath, etc., so we have to assemble those missing pieces of information before passing input them into the pipeline.

So creating new tool that only accepts only SIBs helps keep the tool simple.

And sil-opt really depends on Frontend module that assumes that input module is always one. But LTO pipeline is independent from the module because LTO pipeline needs to be acceptable for multi-modules. This difference makes it difficult to merge these tools.

Right, sil-opt accepts SIL and SIB, but that makes it more flexible and keeps everything in one place :).

The flags are precisely what all the LTO work is about - its how to thread all the appropriate flags through the driver into the frontend and the linker. I think that is expected. That will also be needed for this tool.

So given that the tools can accomplish what is needed, the question is why is the new tool better than building on the existing tools? Is it going to be far more code to just add support for multiple files to sil-opt than writing a new tool?

Perhaps I've made a big misunderstanding. Do you mean that the linker will run sil-opt directly?

I was thinking that LTOPipeline class will be linked to the linker. And swift-lto tool was only going to be used for testing.

Start to implement the LTO library and set up a basis of incremental testing tool named `swift-lto`. The swiftLTO library accepts SIB files for multi-modules, optimizes them at SIL level and emit LLVM bitcode. It will be dynamically loaded into linker to support language specific LTO.

kateinoigakukun · 2020-06-14T05:29:55Z

lib/FrontendTool/FrontendTool.cpp

@@ -1384,6 +1384,7 @@ static bool serializeSIB(SILModule *SM, const PrimarySpecificPaths &PSPs,
  serializationOpts.OutputPath = moduleOutputPath.c_str();
  serializationOpts.SerializeAllSIL = true;
  serializationOpts.IsSIB = true;
+  serializationOpts.SerializeOptionsForDebugging = true;


I'll refine after #32237 merged. I'll serialize options only when -lto=swift is given.

kateinoigakukun · 2023-06-10T10:21:23Z

Close as it's too stale for now.

compnerd reviewed Jun 9, 2020

View reviewed changes

gottesmm reviewed Jun 9, 2020

View reviewed changes

kateinoigakukun force-pushed the katei/swift-lto-tools branch 4 times, most recently from 2a2a112 to 65aa63a Compare June 9, 2020 16:57

kateinoigakukun force-pushed the katei/swift-lto-tools branch from 65aa63a to e4ddb0b Compare June 9, 2020 17:12

kateinoigakukun mentioned this pull request Jun 9, 2020

[NFC] Rename SerializeOptionsForDebugging to SerializeOptions #32270

Closed

kateinoigakukun force-pushed the katei/swift-lto-tools branch 4 times, most recently from 02aae36 to 0268c27 Compare June 10, 2020 14:38

compnerd reviewed Jun 11, 2020

View reviewed changes

lib/LTO/LTO.cpp Outdated Show resolved Hide resolved

compnerd reviewed Jun 11, 2020

View reviewed changes

kateinoigakukun force-pushed the katei/swift-lto-tools branch 7 times, most recently from f285e10 to c71fb6a Compare June 11, 2020 08:15

compnerd reviewed Jun 11, 2020

View reviewed changes

kateinoigakukun force-pushed the katei/swift-lto-tools branch 4 times, most recently from 12ea471 to 459638f Compare June 14, 2020 05:17

kateinoigakukun force-pushed the katei/swift-lto-tools branch from 459638f to 7acfb5a Compare June 14, 2020 05:18

kateinoigakukun commented Jun 14, 2020

View reviewed changes

MaxDesiatov changed the base branch from master to main September 24, 2020 08:45

kateinoigakukun closed this Jun 10, 2023

[LTO] Setup LTO pipeline and swift-lto tool #32233

[LTO] Setup LTO pipeline and swift-lto tool #32233

Uh oh!

Conversation

kateinoigakukun commented Jun 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kateinoigakukun commented Jun 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

compnerd commented Jun 9, 2020

Uh oh!

gottesmm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kateinoigakukun commented Jun 9, 2020

Uh oh!

kateinoigakukun commented Jun 10, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kateinoigakukun Jun 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

compnerd Jun 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kateinoigakukun Jun 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kateinoigakukun Jun 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kateinoigakukun commented Jun 10, 2023

Uh oh!

Uh oh!

kateinoigakukun commented Jun 7, 2020 •

edited

Loading

kateinoigakukun commented Jun 9, 2020 •

edited

Loading

kateinoigakukun Jun 12, 2020 •

edited

Loading

compnerd Jun 14, 2020 •

edited

Loading

kateinoigakukun Jun 14, 2020 •

edited

Loading

kateinoigakukun Jun 15, 2020 •

edited

Loading