Cache generation for stdlib on read-only FS fails

[vchuravy]

Failed to precompile OrdinaryDiffEq [1dea7af3-3e70-54e6-95c3-0bf5283fa5ed] to "/home/pkgeval/.julia/compiled/v1.10/OrdinaryDiffEq/jl_uICwpR".
ERROR: LoadError: IOError: open("/opt/julia/share/julia/stdlib/v1.10/DelimitedFiles/src/DelimitedFiles.jl.pidlock", 194, 292): read-only file system (EROFS)
Stacktrace:
  [1] uv_error(prefix::String, c::Int32)
    @ Base ./libuv.jl:100 [inlined]
  [2] open(path::String, flags::UInt16, mode::UInt16)
    @ Base.Filesystem ./filesystem.jl:121
  [3] tryopen_exclusive(path::String, mode::UInt16)
    @ FileWatching.Pidfile /opt/julia/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:210
  [4] open_exclusive(path::String; mode::UInt16, poll_interval::Int64, wait::Bool, stale_age::Int64)
    @ FileWatching.Pidfile /opt/julia/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:236
  [5] mkpidlock(at::String, pid::Int32; stale_age::Int64, refresh::Float64, kwopts::@Kwargs{wait::Bool})
    @ FileWatching.Pidfile /opt/julia/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:66
  [6] mkpidlock(f::Base.var"#962#963"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{wait::Bool})
    @ FileWatching.Pidfile /opt/julia/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:90
  [7] mkpidlock(f::Function, at::String; kwopts::@Kwargs{wait::Bool})
    @ FileWatching.Pidfile /opt/julia/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:87 [inlined]
  [8] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{})
    @ FileWatching.Pidfile /opt/julia/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:110
  [9] trymkpidlock(::Function, ::Vararg{Any})
    @ FileWatching.Pidfile /opt/julia/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:108
 [10] invokelatest(::Any, ::Returns{Nothing}, ::String; kwargs::@Kwargs{})
    @ Base ./essentials.jl:864 [inlined]
 [11] invokelatest(::Any, ::Base.var"#962#963"{Base.PkgId}, ::String)
    @ Base ./essentials.jl:861 [inlined]
 [12] maybe_cachefile_lock(f::Base.var"#962#963"{Base.PkgId}, pkg::Base.PkgId, srcpath::String)
    @ Base ./loading.jl:2826
 [13] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1905
 [14] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1747
 [15] invoke_in_world(::UInt64, 

Seen during https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/d9a5a29_vs_65523e4/MultiStateSystems.primary.log

cc: @IanButterworth

[IanButterworth]

How much of the file system is read only? Where could these be saved?

[giordano]

From the log:

JULIA_DEPOT_PATH = ::/usr/local/share/julia

The first entry of the depots list is supposed to be writable, it looks like this isn't the case in PkgEval?

Edit: nevermind, it fails to write in /opt/julia/share, which is different from the depot.

[IanButterworth]

Yeah. It could save the pidfile in the depot, but the issue is the cachefile directory slug isn't known ahead of spawning off the child to do the precomp, which is why it's specific to the sourcepath currently.

I guess the lock could be more general for a package (and Julia version) and save as $depot/compiled/v1.10/Foo/.pidfile, rather than what it currently is which is specific package to sourcepath (which could be used by different julia versions)

[vchuravy]

Yeah we should put the PID lock into the same location as the output file.

We could also sink the lock closer to where we determine the slug

[IanButterworth]

The latter would mean you couldn't avoid spawning a julia process.

What about $depot/compiled/v1.10/Foo/$(hash_of_source_path).pidfile ?

[vchuravy]

The latter would mean you couldn't avoid spawning a julia process.

Why? The slug is determined by the parent.

What about $depot/compiled/v1.10/Foo/$(hash_of_source_path).pidfile ?

Maybe simply $depot/compiled/v1.10/Foo.pidfile?

[IanButterworth]

The slug is determined by the parent.

AFAICT that's not true. The full path isn't known by the parent because of things like preferences etc

[maleadt]

From the log:

JULIA_DEPOT_PATH = ::/usr/local/share/julia

The first entry of the depots list is supposed to be writable, it looks like this isn't the case in PkgEval?

FWIW, double :: implies that the default depot dirs are kept, so that env var only adds a read-only layer to the end. PkgEval also doesn't actually store anything there, the layer is just for the registry. (We originally did, but because cache files aren't relocatable we stopped doing that.)

[vchuravy]

like preferences etc

Yeah... Only due to preferences.

julia/base/loading.jl

Line 2316 in 9501540

cachefile = compilecache_path(pkg, prefs_hash)

[maleadt]

Still seems to happen on https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2023-06/19/report.html, which includes f555b4b

e.g. https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2023-06/19/SoilWater_ToolBox.against.log

ERROR: LoadError: Error building `TimeZones`: 
ERROR: LoadError: IOError: open("/opt/julia/share/julia/stdlib/v1.10/LazyArtifacts/src/LazyArtifacts.jl.pidlock", 194, 292): read-only file system (EROFS)

or https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2023-06/19/AEMS.against.log

ERROR: LoadError: Error building `AEMS`: 
ERROR: LoadError: IOError: open("/opt/julia/share/julia/stdlib/v1.10/SuiteSparse/src/SuiteSparse.jl.pidlock", 194, 292): read-only file system (EROFS)