Total build graph #2551
Open
Total build graph #2551
+546
−401
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In order to compile a project with dependencies (and in turn their dependencies) we used to invoke ourselves, i.e. start another actonc process to first compile the dependency projects (which in turn would invoke a sub-compiler in case they had dependencies). Once done, by carefully assembling a search-path, we could reach all the .ty files and in the final build step, the build.zig and generated C source code of all dependencies necessary in order to link the final binaries. We now perform all that work the proper way, staying within a single compiler instance, by first constructing a total build graph; a DAG that includes the modules of all projects. Just like before, we compile modules concurrently up to the maximum number of capabilities (Haskell speak for worker threads) and our critical-path cost approximation is still active, but can now track the cost along the whole path, from the cost of modules within dependencies of dependencies, which should yield a better total order potentially. It's also interesting to note that in dependency projects, we won't necessarily compile all modules - only those that have been chased down by an import. Pure test files for example, should not be needed and thus won't be compiled. Some of the output has been updated to reflect in which project a module is to make it easier to distinguish. I've left it at this point though - I think we should do a major revamp of this now that we do concurrent compilation across projects. It deserves some proper love!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In order to compile a project with dependencies (and in turn their dependencies) we used to invoke ourselves, i.e. start another actonc process to first compile the dependency projects (which in turn would invoke a sub-compiler in case they had dependencies). Once done, by carefully assembling a search-path, we could reach all the .ty files and in the final build step, the build.zig and generated C source code of all dependencies necessary in order to link the final binaries.
We now perform all that work the proper way, staying within a single compiler instance, by first constructing a total build graph; a DAG that includes the modules of all projects. Just like before, we compile modules concurrently up to the maximum number of capabilities (Haskell speak for worker threads) and our critical-path cost approximation is still active, but can now track the cost along the whole path, from the cost of modules within dependencies of dependencies, which should yield a better total order potentially. It's also interesting to note that in dependency projects, we won't necessarily compile all modules - only those that have been chased down by an import. Pure test files for example, should not be needed and thus won't be compiled.
Some of the output has been updated to reflect in which project a module is to make it easier to distinguish. I've left it at this point though - I think we should do a major revamp of this now that we do concurrent compilation across projects. It deserves some proper love!