`Error: failed to collect prefix records` during concurrent execution on HPC (SLURM + Lustre)

[fmerinocasallo]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

1. Set up a Pixi project on a shared network drive (Lustre).
2. Submit multiple SLURM jobs (e.g., 10+) that execute pixi install --frozen simultaneously in the same project directory.
3. Some jobs will succeed, while others fail with the File was modified during parsing error.

Commands I ran and their output:

pixi install
Error:   × failed to collect prefix records from '/***/.pixi/envs/default'
  ╰─▶ File was modified during parsing
  help: try `pixi clean` to reset the environment and run the command again

pixi.toml/pyproject.toml file that reproduces my issue:

[workspace]
channels = ["conda-forge", "bioconda", "biobakery"]
platforms = ["linux-64"]

[system-requirements]
libc = { family = "glibc", version = "2.17" }

[dependencies]
pyega3 = ">=5.2.0,<6"
r-base = ">=4.4,<4.5"
humann = "4.0.0a1.*"
metaphlan = "4.1.1.*"
# ... (other dependencies)

pixi info output:

System
------------
       Pixi version: 0.63.2
        TLS backend: rustls
           Platform: linux-64
   Virtual packages: __unix=0=0
                   : __linux=3.10.0=0
                   : __glibc=2.17=0
                   : __archspec=1=skylake_avx512
          Cache dir: /***/.cache/rattler/cache
       Auth storage: /***/.rattler/credentials.json
   Config locations: No config files found

Global
------------
            Bin dir: /***/.pixi/bin
    Environment dir: /***/.pixi/envs
       Manifest dir: /***/.pixi/manifests/pixi-global.toml

Workspace
------------
               Name: ***
            Version: 0.1.0
      Manifest file: /***/pixi.toml
       Last updated: 19-01-2026 07:27:29

Environments
------------
        Environment: default
           Features: default
           Channels: conda-forge, bioconda, biobakery
   Dependency count: 13
       Dependencies: pyega3, r-base, humann, metaphlan
   Target platforms: linux-64
    Prefix location: /***/.pixi/envs/default
System requirements: libc = { family = "glibc", version = "2.17" }

Issue description

I am encountering an intermittent race condition when running multiple instances of a pipeline via SLURM on an HPC cluster.

Even when using pixi install --frozen to prevent environment modifications, concurrent tasks on a shared filesystem (Lustre) sometimes collide when parsing prefix records.

This behavior is non-deterministic:

• In a batch of 14 concurrent samples, 6 failed with this error while the rest succeeded.
• Re-running those 6 failed samples later (with fewer processes competing) worked without issue.
• Conversely, a separate batch of 88 samples, processed with a maximum concurrency of 18 tasks (the CPU quota limit for my user), finished successfully on the first try.

This suggests that the error is not strictly tied to the number of concurrent tasks, but rather to a transient race condition during the initial environment validation on the shared filesystem.

I’m curious if pixi currently implements a locking mechanism for these read/verify operations, or if this is an area where the high latency of Lustre might be causing unexpected behavior.

---

Regardless of this issue, I would like to thank the maintainers for their incredible work on Pixi; it has significantly improved our bioinformatics workflows! 💙

Expected behavior

Pixi should ideally use a file-locking mechanism (like flock) to ensure that only one process modifies or reads the environment prefix records at a time, or handle concurrent read/verify operations gracefully without crashing when a file is accessed by multiple processes.

[RaphaelRibes]

I'm a heavy user of pixi on SLURM in HPCs, you should do a pixi install -a before starting your job as that when your script run in a SLURM instance, it will act completely independently from another and mess with pixi's cache.
It's like trying to build 18 singularity images at the same time pointing to the same file, some may pass because they started first but other will stop in the middle seeing their cache being modified not by them.

[wolfv]

Thanks for chiming in @RaphaelRibes. On local machines I'd still expect it to work, because we use file locks to try to guarantee exclusivity (in the cache).
But we've seen more issues around file locking / slowness in distributed / HPC systems.

If network is no concern, you coudl also use different "cache" directories for each Pixi task?

@fmerinocasallo lmk if @RaphaelRibes suggestions helps, please

[RaphaelRibes]

I don't think that @fmerinocasallo has to install pixi x amount of time.

He will confirm this but how you start a job on SLURM based task manager is that you run a .sh file some configuration. He must have pixi install --frozen at the beginning of it to make sure that everything is well installed. But he is running the same script x amount of time with different parameter later in the script.

So he basically has a base that is install the pixi environment then do the computing. He doesn't need to reinstall it.
If he need it tho, you're right, he has to precise a cache folder for each job.

[wolfv]

However, we might actually not have a "lock" file on the prefix records. Let me ask @baszalmstra if he thinks that would be a good addition.

[fmerinocasallo]

Thank you both again for the helpful comments, @RaphaelRibes and @wolfv! 🙌

Your explanations were very useful to solidify my understanding about what’s going on under the hood.

After digging deeper into this issue, I think I have a clearer picture of why this race condition appears on Lustre, even when every SLURM task is supposed to be read‑only.

Please let me know if any part of the following is incorrect or overlooks something.

---

1. pixi install --frozen triggers substantial prefix and cache reads

As far as I understand, even with --frozen, Pixi still needs to:

• Read all prefix record files under conda-meta/*.json
• Validate the environment state (hashes, paths, etc.)
• Check the rattler cache metadata (which may involve lightweight temp files or metadata writes)

On a local filesystem this is normally trivial, but on Lustre it can become a source of metadata contention.

That may explain why running pixi install --frozen concurrently across many SLURM array tasks can still hit the exact race condition I observed.

2. Multiple readers can still cause "file modified during parsing" on Lustre

Even when every job is “read-only”, Lustre performs:

• aggressive client-side metadata caching
• frequent stat() / getattr() RPCs to the metadata server
• metadata invalidations when multiple nodes access the same directory
• possible partial/stale attribute reads under load
• delayed propagation of file attributes (e.g., size or timestamp)

So two nodes simultaneously reading many small JSON files in the same directory (e.g., conda-meta/) may observe transient inconsistencies, even without an actual writer modifying anything.

This aligns with the intermittent “file was modified during parsing” error that I’m encountering, and it seems consistent with how other package managers behave on HPC systems as well (Conda, Mamba, Spack, Singularity, etc.).

3. Prefix-level locking might mitigate this on HPC

Even though the underlying cause appears to be the distributed filesystem behavior rather than Pixi itself, a lightweight lock around prefix‑record parsing (similar to the cache locking Pixi already implements) might help avoid these race conditions entirely on systems like Lustre.

If this aligns with Pixi’s design goals, I’d be very supportive of such an addition.

---

Based on your suggestions, here’s the plan I’ll adopt to avoid these races:

1. Run a single serialized pixi install -a before submitting any SLURM jobs through sbatch

This ensures the prefix is fully materialized before concurrency begins.

I may run it directly on the login node (simplest), unless the cluster policy requires offloading it to a dedicated SLURM job.

2. Avoid running pixi install --frozen inside every SLURM task

Since --frozen can still trigger significant prefix and cache access, I plan to switch to:

pixi run <task>

Or some light-weight validation step if needed.

❓ Question for maintainers

Could pixi run <task> itself still trigger prefix-record parsing that could race on Lustre, or is pixi run safe under concurrency?

3. Use a per-job cache directory

I am considering @wolfv’s suggestion to avoid shared cache contention entirely by isolating the cache per job:

export RATTLER_CACHE_DIR=/tmp/$USER/rattler_cache/$SLURM_JOB_ID

so that multiple tasks do not simultaneously access the same rattler cache on Lustre.

❓ Clarification

Would using a per-job cache directory be sufficient to prevent all cache‑related conflicts, or does Pixi still read shared cache state elsewhere that I should isolate as well?

---

Thanks again for the guidance! Pixi is already a big step up from our previous Conda-based setup, and I really appreciate your responsiveness and the ongoing work. Happy to test any locking improvements or experimental builds if needed. Keep up the great work! 💙

[RaphaelRibes]

pixi run is safe inside a slurm job as long as if the environment is installed before it run. So just do pixi instal -a --frozen && sbatch yourscript.sh and everything should run fine.