Restrict COFF to a single thread when symbol count is high #50874
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pchintalapudi
added
compiler:codegen
pchintalapudi
added
the
backport 1.10
pchintalapudi
force-pushed
the
pc/coff-single-thread
branch
from
44c7472
to
590a63b
Compare
gbaraldi
approved these changes
Aug 10, 2023
| @@ -1413,6 +1415,13 @@ static unsigned compute_image_thread_count(const ModuleInfo &info) { | |||
| LLVM_DEBUG(dbgs() << "32-bit systems are restricted to a single thread\n"); | |||
| return 1; | |||
| #endif | |||
| // COFF has limits on external symbols (even hidden) up to 65536. We reserve the last few | |||
| // for any of our other symbols that we insert during compilation. | |||
| if (info.triple.isOSBinFormatCOFF() && info.globals > 64000) { | |||
There was a problem hiding this comment.
In general, the usage of magic constants in the code shall be avoided.
This number, 64000, shall be defined as a constant into a specific place in the code, among other constants.
There was a problem hiding this comment.
I don't think there's a good reason to define the constant somewhere else, that's more likely to reduce understanding since the constant would be separated from its sole use site. There's also a fairly descriptive comment immediately above the constant describing why it exists, so I don't think the situation would be improved by extracting it into another variable and naming it.
There was a problem hiding this comment.
I don't know the details of aotcompile.hpp and aotcompile.cpp, but imagine you a situation where you have 20 such magic constants in the code and at some point in time a newer colleague will want to change one such constant. He will have to search the code using the magic constant (he does not know the exact value) and good luck finding that piece of code. In general is best to have these static const variables grouped for instance in a class Params in the header or at the beginning of the cpp file.
There was a problem hiding this comment.
I disagree, if someone is modifying this piece of code, which is the only one that depends on this constant, it makes more sense for the value to be right here. Specially because it has the comment right above it.
This isn't a parameter to be played with.
This was referenced Aug 11, 2023
| LLVM_DEBUG(dbgs() << "COFF is restricted to a single thread for large images\n"); | ||
| return 1; | ||
| } | ||
|
|
||
| // This is not overridable because empty modules do occasionally appear, but they'll be very small and thus exit early to | ||
| // known easy behavior. Plus they really don't warrant multiple threads | ||
| if (info.weight < 1000) { |
There was a problem hiding this comment.
yet another magic constant (not because of this PR)
and line 1434 contains another one:
auto max_threads = info.globals / 100
IanButterworth
pushed a commit
that referenced
this pull request
Aug 19, 2023
(cherry picked from commit eb4416b)
58 tasks
KristofferC
added a commit
that referenced
this pull request
Oct 2, 2023
Backported PRs: - [x] #48625 <!-- add replace(io, str, patterns...) --> - [x] #48387 <!-- Improve documentation of sort-related functions --> - [x] #48363 <!-- Revise sort.md and docstrings in sort.jl --> - [x] #48977 <!-- Update SparseArrays.jl stdlib for SuiteSparse 7 --> - [x] #50719 <!-- fix `CyclePadding(::DataType)` --> - [x] #50694 <!-- inference: permit recursive type traits --> - [x] #50860 <!-- Add `Base.get_extension` to docs/API --> - [x] #50594 <!-- Disallow non-index Integer types in isassigned --> - [x] #50802 <!-- Makes IntrusiveLinkedListSynchronized mutable to avoid allocation on poptask --> - [x] #50858 <!-- Add a `threadpool` parameter to `Channel` constructor --> - [x] #50874 <!-- Restrict COFF to a single thread when symbol count is high --> - [x] #50822 <!-- Add default method for setmodifiers! --> - [x] #50730 <!-- Fix integer overflow in `isapprox` --> - [x] #50850 <!-- Remove weird Rational dispatch and add pi functions to list --> - [x] #50809 <!-- Limit type-printing in MethodError --> - [x] #50915 <!-- Add note the `Task` about sticky bit --> - [x] #50929 <!-- when widening tuple types in tmerge, only widen the complex parts --> - [x] #50928 <!-- Bump JuliaSyntax to 0.4.6 --> - [x] #50959 <!-- Update libssh2 patches --> - [x] #50823 <!-- Make ranges more robust with unsigned indexes. --> - [x] #48542 <!-- Add docs on task-specific buffering using multithreading --> - [x] #50912 <!-- Separate foreign threads into a :foreign threadpool --> - [x] #51010 <!-- Add ORIGIN to SuiteSparse rpath on Linux/FreeBSD --> - [x] #50753 <!-- cat: remove unused promote_eltype methods that confuse inference --> - [x] #51027 <!-- Implement realloc accounting correctly --> - [x] #51019 <!-- fix a case of potentially use of undefined variable when handling error in distributed message processing --> - [x] #51039 <!-- Revert "Optimize findall(f, ::AbstractArray{Bool}) (#42202)" --> - [x] #51036 <!-- add missing invoke edge for nospecialize targets --> - [x] #51042 <!-- inference: fix return_type_tfunc modeling of concrete functions --> - [x] #51114 <!-- Workaround upstream FreeBSD issue #272992 --> - [x] #50892 <!-- Add `JL_DLLIMPORT` to `small_typeof` declaration --> - [x] #51154 <!-- broadcast: use recursion rather than ntuple to map over a tuple --> - [x] #51153 <!-- fix debug typo in "add missing invoke edge for nospecialize targets (#51036)" --> - [x] #51222 <!-- Check again if the tty is open inside the IO lock --> - [x] #51236 <!-- Add lock around uv_unref during init --> - [x] #51243 <!-- GMP: Gracefully handle more overflows. --> - [x] #51254 <!-- Ryu: make sure adding zeros does not overwrite trailing dot --> - [x] #51175 <!-- shorten stale_age for cachefile lock --> - [x] #51300 <!-- fix method definition error for bad vararg --> - [x] #51307 <!-- fix force-throw ctrl-C on Windows --> - [x] #51303 <!-- ensure revising structs is safe --> - [x] #51393 - [x] #51403 Need manual backport: - [x] #51009 <!-- fix #50562, regression in `in` of tuple of Symbols --> - [x] #51053 <!-- Bump Statistics stdlib --> - [x] #51013 <!-- fix #50709, type bound error due to free typevar in sparam env --> - [x] #51305 <!-- reduce test time for rounding and floatfuncs --> Contains multiple commits, manual intervention needed: - [ ] #50663 <!-- Fix Expr(:loopinfo) codegen --> - [ ] #51035 <!-- refactor GC scanning code to reflect jl_binding_t are now first class --> - [ ] #51092 <!-- inference: fix bad effects for recursion --> - [x] #51247 <!-- Check if malloc has succeeded before incrementing gc stats --> - [x] #51294 <!-- use LibGit2_jll for LibGit2 library --> Non-merged PRs with backport label: - [ ] #51132 <!-- Handle `AbstractQ` in concatenation --> - [x] #51029 <!-- testdefs: make sure that if a test set changes the active project, they change it back when they're done --> - [ ] #50919 <!-- Code loading: do the "skipping mtime check for stdlib" check regardless of the value of `ispath(f)` --> - [ ] #50824 <!-- Add some aliasing warnings to docstrings for mutating functions --> - [x] #50385 <!-- Precompile pidlocks: add to NEWS and docs --> - [ ] #49805 <!-- Limit TimeType subtraction to AbstractDateTime -->
KristofferC
removed
the
backport 1.10
nalimilan
pushed a commit
that referenced
this pull request
Nov 5, 2023
(cherry picked from commit eb4416b)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since COFF is unhappy with large numbers of external hidden symbols, judging by #50729, we'll just force COFF to use a single thread when approaching the limit so that users don't have to manually request single-threadedness.