Skip to content

Make precompile files relocatable/servable #47943

Closed

Description

Cross-ref: #45215


Precompile files (.ji and soon, package images) should be as relocatable as possible. Use cases for this include:

  • Serving precompiled .so files over the network to obviate precompilation delays when installing new packages.
  • Moving depots around on the local file system

The biggest known problem in this is that precompilation caches currently embed the absolute path to their dependencies (e.g. .jl file sources, Artifacts.toml files, etc...). I propose that we should introduce a notion of "depot-local" files here. Here's how I see it working:

If we have a package Foo.jl which has been precompiled, that cache should reference the source files as something like @depot/packages/Foo_jll/a2b2c3/src/Foo.jl. During cache header verification, we string-replace ^@depot to the same depot that the cache file is currently being loaded from. I do not think it's worthwhile to allow this path to resolve to other depots on the depot path; it's too easy to find the "wrong" files, and I think it's reasonable to expect an atomic move of all associated resources for a package image.

When serializing out dependency paths, we simply check if the depot we're serializing into is a superpath of the dependency path. If it is, we emit @depot/..., otherwise we emit the full path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions