Julia

Managing depots

Julia uses a shared depot to store files like packages, artifacts and precompiled images. Unfortunately, this is susceptible to race conditions when multiple processes (either from multiple Slurm jobs, or multiple MPI processes within a Slurm job) attempt to perform package operations.

To reduce the likelihood of this occurring, by default we set the JULIA_DEPOT_PATH variable so that each build will have its own depot. The recommended way to set up a pipeline is to then have a single step which performs the necessary package operations, followed by a buildkite wait step before any other steps.

In order to reduce the time spent precompiling, you can allocate multiple CPU cores to this step (via --cpus-per-task), and set JULIA_NUM_PRECOMPILE_TASKS.

For example:

steps:
  - label: "initialize"
    key: "init"
    command:
      - echo "--- Instantiate project"
      - julia --project -e 'using Pkg; Pkg.instantiate(;verbose=true); Pkg.precompile(;strict=true)'
    agents:
      slurm_cpus_per_task: 8
    env:
      JULIA_NUM_PRECOMPILE_TASKS: 8

  - wait

  # all other steps

Preferences

Some packages use Preferences.jl to manage compile-time specific options. A useful pattern to apply them to the whole pipeline is to create a custom [Julia]Project.toml file in a specific location (I commonly use .buildkite/JuliaProject.toml to emphasize it applies to Julia):

# .buildkite/JuliaProject.toml
[extras]
CUDA_Runtime_jll = "76a88914-d11a-5bdc-97e0-2f5a05c973a2"
HDF5_jll = "0234f1f7-429e-5d53-9886-15a909be8d59"
MPIPreferences = "3da0fdf6-3ccc-4f1b-acd9-58baa6c99267"

[preferences.CUDA_Runtime_jll]
version = "local"

[preferences.HDF5_jll]
libhdf5_path = "libhdf5"
libhdf5_hl_path = "libhdf5_hl"

[preferences.MPIPreferences]
_format = "1.0"
abi = "OpenMPI"
binary = "system"
libmpi = "libmpi"
mpiexec = "srun"

And then append this to the JULIA_LOAD_PATH in the pipeline:

env:
  JULIA_LOAD_PATH: "${JULIA_LOAD_PATH}:${BUILDKITE_BUILD_CHECKOUT_PATH}/.buildkite"