Support COFF/bigobj for large images on Windows.

[ufechner7]

MWE:
Check out from git:

cd repos # any folder of your choice, but without spaces in the folder name
git clone https://github.com/ufechner7/Tethers.jl

Build the system image:

cd repos/Tethers.jl
cd bin
./create_sys_image

Now look into the task manager and check the CPU load. My CPU load was mostly at 7%, sometimes a bit higher,
but 8 threads where not used when creating the system image.

Doing the same on Linux is much faster, and the CPU load reaches 800% after about 2 minutes of the progress
of "building system image".

I used juliaup to run Julia. The machine had 24GB of RAM, never more than 50% where in use.

julia> versioninfo()
Julia Version 1.10.0-rc2
Commit dbb9c46795 (2023-12-03 15:25 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 16 × AMD Ryzen 9 7950X 16-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
  Threads: 1 on 16 virtual cores

Summary: Building a system image is slow on Windows and fast on Linux because on Windows Julia 1.10-rc2 uses only one thread.

Is this a known limitation?

---

How it should work is documented:
JULIA_IMAGE_THREADS

An unsigned 32-bit integer that sets the number of threads used by image compilation in this Julia process. The value of this variable may be ignored if the module is a small module. If left unspecified, the smaller of the value of JULIA_CPU_THREADS or half the number of logical CPU cores is used in its place.

[vtjnash]

We discovered that Windows did not behave itself well, so we disabled it, especially as very little development is actually done directly on that platform

[ufechner7]

That is a pity... I want to use ModelingToolkit in a class, and I have to suggest the students to create a system image because loading ModelingToolkit is still slow otherwise, even with Julia 1.10. It makes a difference if that takes 15 min or 30 min, though...
Any chance to re-enable it at some point in time?

[vchuravy]

Any chance to re-enable it at some point in time?

I don't think we hard disable parallelism per-se, but ironically only when the system images are very large:

julia/src/aotcompile.cpp

Line 1454 in b57f8d1

const char *env_threads = getenv("JULIA_IMAGE_THREADS");

x-ref: #50874

At the core of the issue is that COFF (the default binary format on Windows) uses 16bit for the number of symbols.

[ufechner7]

How do you calculate what is "very large"?

[KristofferC]

I think it is

julia/src/aotcompile.cpp

Line 1420 in 590a63b

if (info.triple.isOSBinFormatCOFF() && info.globals > 64000) {

[ufechner7]

Could we use the https://en.wikipedia.org/wiki/Portable_Executable format on Windows?

[gbaraldi]

PE is pretty much COFF, COFF was the unix name and PE is the microsoft name

[ufechner7]

PE is pretty much COFF, COFF was the unix name and PE is the microsoft name

Maybe a version with 64-bit address space support called PE32+ ?

[gbaraldi]

It's going to be the same, the issue is that they use 16bit integer do identify stuff, and that has a 65k maximum value. People have complained to microsoft lots of times about this, but they haven't fixed this

[ufechner7]

Or can we reduce the number of exported symbols?

And why does the number of exported symbols depend on the number of threads used in the first place?

[vchuravy]

When we use multiple threads we split the output into multiple files itself, but you still want calls across the split files to be fast.
So we export the symbols in the split files and then we merge them into one final output file, which then fails since we have to many symbols.

In single-threaded compilation we never make the symbols visible.

[ufechner7]

This sounds to me as if we could use another file format for the split files...

[vchuravy]

To my knowledge that is not feasible, but I would be happy to be proven wrong.
Windows is not a platform that we have many (or really any developers for).

[ufechner7]

To my knowledge that is not feasible

Why not? Which program produces the intermediate files? And which program combines them?