Base.julia_cmd(): correctly forward the --sysimage-native-code=no flag if it is provided

[DilumAluthge]

Closes #42173
Ref #41238 (comment)

This is a replacement for #42173.

[DilumAluthge]

@vtjnash @staticfloat We don't need to backport this, right? Since the motivation here is to fix the code coverage, and we only run code coverage on master.

[vtjnash]

oops, it was even already documented

[DilumAluthge]

Or do we need to backport this because it is a bug fix? Since it was in the docstring.

[DilumAluthge]

I'm going to mark this for backport as a bugfix. If that is not correct, please remove the backport labels.

[DilumAluthge]

@staticfloat @vtjnash This PR seems to have broken the coverage job. See e.g. https://buildkite.com/julialang/julia-master-scheduled/builds/225

Run Julia tests in parallel with code coverage enabled
┌ Info: 
└   Sys.CPU_THREADS = 16
┌ Info: 
│   tests = "all"
└   ncores = 16
      From worker 15:	Master process (id 1) could not connect within 60.0 seconds.
      From worker 15:	exiting.
Worker 15 terminated.
Unhandled Task ERROR: Version read failed. Connection closed by peer.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] process_hdr(s::TCPSocket, validate_cookie::Bool)
   @ Distributed /cache/build/amdci5-2/julialang/julia-master-scheduled/usr/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:267
 [3] message_handler_loop(r_stream::TCPSocket, w_stream::TCPSocket, incoming::Bool)
   @ Distributed /cache/build/amdci5-2/julialang/julia-master-scheduled/usr/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:151
 [4] process_tcp_streams(r_stream::TCPSocket, w_stream::TCPSocket, incoming::Bool)
   @ Distributed /cache/build/amdci5-2/julialang/julia-master-scheduled/usr/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:126
 [5] (::Distributed.var"#99#100"{TCPSocket, TCPSocket, Bool})()
   @ Distributed ./task.jl:411
      From worker 9:	ErrorException("Process(1) - Invalid connection credentials sent by remote.")CapturedException(ErrorException("Process(1) - Invalid connection credentials sent by remote."), Any[(error(s::String) at error.jl:33, 1), (process_hdr(s::Sockets.TCPSocket, validate_cookie::Bool) at process_messages.jl:257, 1), (message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) at process_messages.jl:151, 1), (process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) at process_messages.jl:126, 1), ((::Distributed.var"#99#100"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})() at task.jl:411, 1)])
      From worker 9:	Process(1) - Unknown remote, closing connection.
Worker 9 terminated.
Unhandled Task ERROR: Version read failed. Connection closed by peer.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] process_hdr(s::TCPSocket, validate_cookie::Bool)
   @ Distributed /cache/build/amdci5-2/julialang/julia-master-scheduled/usr/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:267
 [3] message_handler_loop(r_stream::TCPSocket, w_stream::TCPSocket, incoming::Bool)
   @ Distributed /cache/build/amdci5-2/julialang/julia-master-scheduled/usr/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:151
 [4] process_tcp_streams(r_stream::TCPSocket, w_stream::TCPSocket, incoming::Bool)
   @ Distributed /cache/build/amdci5-2/julialang/julia-master-scheduled/usr/share/julia/stdlib/v1.8/Distributed/src/process_messages.jl:126
 [5] (::Distributed.var"#99#100"{TCPSocket, TCPSocket, Bool})()
   @ Distributed ./task.jl:411
      From worker 9:	Master process (id 1) could not connect within 60.0 seconds.
      From worker 9:	exiting.

[DilumAluthge]

I'm removing the backport labels until we can figure out why this broke the code coverage Buildkite job.

[vtjnash]

looks like buildkite needs to increase the timeout threshold for starting workers, as we launch a lot of them, and these are probably noisy machines

[vtjnash]

Note that it is a 32-core machine, and it looks like buildkite is mis-configured to force the launching of 16*16 (or 256) threads

[DilumAluthge]

looks like buildkite needs to increase the timeout threshold for starting workers, as we launch a lot of them, and these are probably noisy machines

We could increase the timeout. That being said, before this PR was merged, the code coverage Buildkite job had no trouble starting all of its workers within the default timeout. See e.g. the following successful jobs:

• https://buildkite.com/julialang/julia-master-scheduled/builds/224
• https://buildkite.com/julialang/julia-master-scheduled/builds/223
• https://buildkite.com/julialang/julia-master-scheduled/builds/222
• https://buildkite.com/julialang/julia-master-scheduled/builds/221

Is it possible that this PR specifically would increase the time it takes a worker to start up?

[DilumAluthge]

To increase the timeout, I'm assuming I just need to set the JULIA_WORKER_TIMEOUT environment variable to something greater than 60?

[vtjnash]

Yes, but also, we need to fix that configuration bug too

[DilumAluthge]

Yes, but also, we need to fix that configuration bug too

Ah, the 16 workers * 16 threads per worker?

For that one, I think we can just set JULIA_NUM_THREADS=1 in the code coverage Buildkite job, right?

[DilumAluthge]

Increasing the worker timeout seems to have resolved the issue with the code coverage job, so I'm adding the backport labels back.