Error loading Oscar from different nodes on HPC cluster

[VAddala9]

Oscar does not load when on different compute nodes on an HPC cluster (MIT Supercloud). Oscar does load on the login node which is the one I used to install Oscar.

On other nodes, when I run using Oscar, I get the following error:

ERROR: InitError: ArgumentError: '/home/gridsan/vaddala9/.julia/scratchspaces/d720cf60-89b5-51f5-aff5-213f193123e7/polymake_4089907222526935278_1.7_depstree/deps/FLINT_jll' exists. `force=true` is required to remove '/home/gridsan/vaddala9/.julia/scratchspaces/d720cf60-89b5-51f5-aff5-213f193123e7/polymake_4089907222526935278_1.7_depstree/deps/FLINT_jll' before copying.
Stacktrace:
  [1] checkfor_mv_cp_cptree(src::String, dst::String, txt::String; force::Bool)
    @ Base.Filesystem ./file.jl:325
  [2] cp(src::String, dst::String; force::Bool, follow_symlinks::Bool)
    @ Base.Filesystem ./file.jl:364
  [3] rename(src::String, dst::String; force::Bool)
    @ Base.Filesystem ./file.jl:967
  [4] rename
    @ ./file.jl:964 [inlined]
  [5] force_symlink(source::String, target::String)
    @ Polymake ~/.julia/artifacts/e8b92adf0c63efefd4d6e7f697525c23844133fa/share/polymake/generate_deps_tree.jl:31
  [6] prepare_deps_tree(targetdir::String)
    @ Polymake ~/.julia/artifacts/e8b92adf0c63efefd4d6e7f697525c23844133fa/share/polymake/generate_deps_tree.jl:53
  [7] __init__()
    @ Polymake ~/.julia/packages/Polymake/w7g9z/src/Polymake.jl:104
  [8] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base ./loading.jl:768
  [9] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
    @ Base ./loading.jl:854
 [10] _tryrequire_from_serialized(modkey::Base.PkgId, build_id::UInt64, modpath::String)
    @ Base ./loading.jl:783
 [11] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
    @ Base ./loading.jl:843
 [12] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1097
 [13] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1013
 [14] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:997
during initialization of module Polymake

The output of Oscar.versioninfo(full=true) from the login node is below:

OSCAR version 0.11.2
  combining:
    AbstractAlgebra.jl   v0.27.8
    GAP.jl               v0.9.2
    Hecke.jl             v0.16.7
    Nemo.jl              v0.32.7
    Polymake.jl          v0.8.3
    Singular.jl          v0.14.0
  building on:
    Antic_jll               v0.201.500+0
    Arb_jll                 v200.2300.0+0
    Calcium_jll             v0.401.100+0
    FLINT_jll               v200.900.4+0
    GAP_jll                 v400.1200.200+0
    Singular_jll            v403.101.500+0
    libpolymake_julia_jll   v0.8.3+1
    libsingular_julia_jll   v0.27.0+1
    polymake_jll            v400.700.1+1
See `]st -m` for a full list of dependencies.

Julia Version 1.7.3
Commit 742b9abb4d (2022-05-06 12:58 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake-avx512)
Environment:
  JULIA_LOAD_PATH = @:@v#.#:@stdlib:/state/partition1/llgrid/pkg/julia/julia-1.7.3/local/share/julia/environments/v1.7/:.
  JULIA_PROJECT = /home/gridsan/vaddala9/.julia/environments/v1.7
Official https://julialang.org/ release

[benlorenz]

Are you using GNU parallel to run the code on multiple nodes?
If so please check the last FAQ entry here, i.e. either upgrade to julia 1.8 or set ENV["TMPDIR"] = nothing before using Oscar.

[VAddala9]

Thank you. Loading Oscar is now working to run code on multiple nodes in parallel.
I just wanted to bring up some things I noticed. For me, ENV["TMPDIR"] = nothing did not work. I get the following error on both Julia v1.7.3 and v1.8.5:

ERROR: InitError: ArgumentError: "nothing" is not a directory
Stacktrace:
  [1] tempname(parent::String; cleanup::Bool)
    @ Base.Filesystem ./file.jl:595
  [2] tempname (repeats 2 times)
    @ ./file.jl:595 [inlined]
  [3] select_compiler(lang::String, candidates::Vector{String}, extension::String)
    @ GAP.Setup ~/.julia/packages/GAP/p0xxt/src/setup.jl:60
  [4] regenerate_gaproot()
    @ GAP.Setup ~/.julia/packages/GAP/p0xxt/src/setup.jl:106
  [5] __init__()
    @ GAP ~/.julia/packages/GAP/p0xxt/src/GAP.jl:242
  [6] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base ./loading.jl:768
  [7] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
    @ Base ./loading.jl:854
  [8] _tryrequire_from_serialized(modkey::Base.PkgId, build_id::UInt64, modpath::String)
    @ Base ./loading.jl:783
  [9] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
    @ Base ./loading.jl:843
 [10] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1097
 [11] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1013
 [12] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:997
during initialization of module GAP

Second, when loading Oscar for the first time on a node, an attempt is made to access the internet. This might not a problem in most cases. I just happened to notice this because non-login nodes on the cluster I'm using do not have internet access, so I get an error (below). However, if I catch the error and proceed, things seem to work as normal.

-----    -----    -----      -      -----   
|     |  |     |  |     |    | |    |     |  
|     |  |        |         |   |   |     |  
|     |   -----   |        |     |  |-----   
|     |        |  |        |-----|  |   |    
|     |  |     |  |     |  |     |  |    |   
 -----    -----    -----   -     -  -     -  

...combining (and extending) ANTIC, GAP, Polymake and Singular
Version 0.11.2 ... 
 ... which comes with absolutely no warranty whatsoever
Type: '?Oscar' for more information
(c) 2019-2022 by The OSCAR Development Team
ERROR: InitError: Error thrown by GAP: Error, Couldn't connect to server while requesting https://raw.githubusercontent.com/gap-system/gap-distribution/master/DistributionUpdate/PackageUpdate/currentPackageInfoURLList in
  Julia.Core._apply( julia_obj, args ) at /home/gridsan/vaddala9/.julia/packages/GAP/p0xxt/pkg/JuliaInterface/gap/calls.gi:15 called from 
PKGMAN_DownloadURL( PKGMAN_PackageInfoURLList ) at /home/gridsan/vaddala9/.julia/scratchspaces/c863536a-3901-11e9-33e7-d5cd0df7b904/gap_16903213223646344887_1.8/pkg/packagemanager/gap/PackageManager.gi:19 called from
GetPackageURLs(  ) at /home/gridsan/vaddala9/.julia/scratchspaces/c863536a-3901-11e9-33e7-d5cd0df7b904/gap_16903213223646344887_1.8/pkg/packagemanager/gap/PackageManager.gi:125 called from
InstallPackageFromName( string, version, interactive ) at /home/gridsan/vaddala9/.julia/scratchspaces/c863536a-3901-11e9-33e7-d5cd0df7b904/gap_16903213223646344887_1.8/pkg/packagemanager/gap/PackageManager.gi:82 called from
<function "InstallPackage">( <arguments> )
 called from read-eval loop at *defin*:0

Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] ThrowObserver(depth::Int32)
    @ GAP ~/.julia/packages/GAP/p0xxt/src/GAP.jl:88
  [3] _call_gap_func(func::GAP_jll.GapObj, a1::GAP_jll.GapObj, a2::Bool)
    @ GAP ~/.julia/packages/GAP/p0xxt/src/ccalls.jl:271
  [4] call_gap_func_nokw
    @ ~/.julia/packages/GAP/p0xxt/src/ccalls.jl:224 [inlined]
  [5] (::GAP_jll.GapObj)(a1::GAP_jll.GapObj, a2::Bool)
    @ GAP ~/.julia/packages/GAP/p0xxt/src/ccalls.jl:236
  [6] install(spec::String, version::String; interactive::Bool, quiet::Bool, pkgdir::String)
    @ GAP.Packages ~/.julia/packages/GAP/p0xxt/src/packages.jl:151
  [7] load(spec::String, version::String; install::Bool, quiet::Bool)
    @ GAP.Packages ~/.julia/packages/GAP/p0xxt/src/packages.jl:100
  [8] __init__()
    @ Oscar ~/.julia/packages/Oscar/Iei9b/src/Oscar.jl:167
  [9] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
    @ Base ./loading.jl:831
 [10] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
    @ Base ./loading.jl:1039
 [11] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1315
 [12] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
 [13] macro expansion
    @ ./loading.jl:1180 [inlined]
 [14] macro expansion
    @ ./lock.jl:223 [inlined]
 [15] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
during initialization of module Oscar

[benlorenz]

I thought that setting it to nothing did work for other users a while ago, so I am not really sure what changed there but using delete!(ENV, "TMPDIR") instead should work more reliably (and I will update the FAQ entry).

Improvements for loading Oscar (or GAP) while offline are tracked here:
oscar-system/GAP.jl#852
Edit: This might be fixed in Oscar 0.11.3 (GAP.jl 0.9.3), so maybe you can try updating?

[micjoswig]

I thought that setting it to nothing did work for other users a while ago, so I am not really sure what changed there but using delete!(ENV, "TMPDIR") instead should work more reliably (and I will update the FAQ entry).

Improvements for loading Oscar (or GAP) while offline are tracked here: oscar-system/GAP.jl#852 Edit: This might be fixed in Oscar 0.11.3 (GAP.jl 0.9.3), so maybe you can try updating?

@VAddala9 Can you please re-check if v0.11.3 works for you? Please explain what is still missing or close this ticket.

[VAddala9]

Yes things are working with v0.11.3.