Images not portable between znver3 and "AMD EPYC 7B13" (missing xsaves feature)

[nrontsis]

Description

Pre-compiled code in Julia cannot be used in machines of "similar" architecture. This includes both pre-compiled environments (via Pkg.instantiate() and system images/apps, both of which are demonstrated in minimal examples.

Related discourse threads [1], [2].

Relevance

This is important for production setups where docker images are built in a CI/CD pipeline (where often times there is limited control on the exact hardware), and then deployed in VMs of specified hardware to serve production.

Minimal, reproducible example

In this examples we demonstrate issues with building a docker image with precompiled Julia code from Intel Cascade Lake to an AMD EPYC 7B13. The precompiled code is generated both via Pkg.instantiate() and via PackageCompiler.jl, while taking care to specify the correct target cpu_type.

To reproduce, save the following as Dockerfile in a c2-standard-4 VM (Intel Cascade Lake) in google cloud (say, with google's container-optimised OS):

FROM julia:1.9.0
RUN julia -e 'download("https://github.com/JuliaLang/PackageCompiler.jl/archive/refs/tags/v2.1.7.tar.gz", "v2.1.7.tar.gz")'
RUN tar -xf v2.1.7.tar.gz && rm v2.1.7.tar.gz
RUN julia --project=/PackageCompiler.jl-2.1.7/examples/MyApp -e "using Pkg; Pkg.instantiate()"
CMD JULIA_DEBUG=loading julia --project=/PackageCompiler.jl-2.1.7/examples/MyApp -e "using MyApp"

and then build it and pack it by running:

docker build . --tag julia_precompile
docker save julia_precompile > julia_precompile.tar

Then, transfering the resulting docker image into a t2d-standard-4 (AMD EPYC 7B13) VM in google cloud run:

docker load < julia_precompile.tar
docker run julia_precompile

This shows cache invalidations happening, for example:

┌ Debug: Rejecting cache file /root/.julia/compiled/v1.9/MyApp/6XOLX_U0SK4.ji for MyApp [f943f3d7-887a-4ed5-b0c0-a1d6899aa8f5] since pkgimage can't be loaded on this target
└ @ Base loading.jl:2706

The same issue (incompatibility of targets), appears also when building a system image, even when explicitly specifying the target's CPU (i.e. znver3 for t2d-standard-4). In this case we get ERROR: Unable to find compatible target in system image. To reproduce this simply follow the previous steps, but use the following Dockerfile instead:

FROM julia:1.9.0
RUN julia -e 'download("https://github.com/JuliaLang/PackageCompiler.jl/archive/refs/tags/v2.1.7.tar.gz", "v2.1.7.tar.gz")'
RUN tar -xf v2.1.7.tar.gz && rm v2.1.7.tar.gz
RUN apt-get update && apt-get -f install && apt-get -y install gcc g++  # Required by PackageCompiler
RUN julia -e 'using Pkg; Pkg.develop(path="/PackageCompiler.jl-2.1.7")'
RUN julia --project=/PackageCompiler.jl-2.1.7/examples/MyApp -e 'using PackageCompiler; create_sysimage(["MyApp"]; sysimage_path="/sysimage.so", cpu_target="znver3")'
CMD JULIA_DEBUG=loading julia -J/sysimage.so --project=/PackageCompiler.jl-2.1.7/examples/MyApp -e "using MyApp"

Known, sub-optimal workarounds:

Passing cpu_target="generic" in PackageCompiler. create_sysimage avoids the errors/precompilations. However this can result in particularly suboptimal performance.

[vchuravy]

Thank you for the well written issue! I think the problem here is actually one of documentation.

Julia supports something called multi-versioning allowing users to create image files that can be loaded on machines with varying architecture.

See https://docs.julialang.org/en/v1/devdocs/sysimg/#sysimg-multi-versioning

You do need to generally know what target system you are aiming for, but as an example the default sysimg that we ship to users is built with generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1) (We may update that soon to include more modern systems JuliaCI/julia-buildkite#298)

Pkgimages support the same mechanism, with the exception that you need to set JULIA_CPU_TARGETS as an environment flag. See https://docs.julialang.org/en/v1/devdocs/pkgimg/

So all the tools should be there to make this work, but I agree that it is not super obvious and could benefit from a great blogspot (julialang.com/blog) and section in the manual describing deployment.

(There are also some outstanding issues that folks might encounter like #49912)

[simsurace]

What happens to the caches if you do

FROM julia:1.9.0
RUN julia -e 'download("https://github.com/JuliaLang/PackageCompiler.jl/archive/refs/tags/v2.1.7.tar.gz", "v2.1.7.tar.gz")'
RUN tar -xf v2.1.7.tar.gz && rm v2.1.7.tar.gz
ENV JULIA_CPU_TARGET=generic
RUN julia --project=/PackageCompiler.jl-2.1.7/examples/MyApp -e "using Pkg; Pkg.instantiate()"
CMD JULIA_DEBUG=loading julia --project=/PackageCompiler.jl-2.1.7/examples/MyApp -e "using MyApp"

[nlw0]

@vchuravy I have ran into a similar issue when trying to upgrade a project from 1.8.3 to 1.9.0. I was asking about it on discourse. My impression is there could be a subtle bug somewhere resulting in cache invalidations. Or it might not even be a bug, but a reversion on a non-official feature. Might also be something weird happening in the cloud. I didn't start an issue because I'm not too familiar with the mechanisms of pre-compilation and architecture identifications, but I'm glad @nrontsis started this one.

In my case, using 1.8.3 I can install packages and build a Docker container in my desktop just fine. Once I run it in the cloud I get reasonable startup times, just relying on regular precompilation, I'm not doing anything special with PackageCompiler yet. As far as I know, there are no cache invalidations. Using 1.9.0 I get a longer startup time, what prevents me from using it for this application, and I also get the explicit cache invalidation messages like that one above.

I have already tried different JULIA_CPU_TARGET settings, including generic, the one from the default sysimg you quoted, and another custom one. I got the same behavior every time.

I can try to run the minimal example later, or create something based on my experience, but it would be great to have an idea of what else to look for, because simply setting JULIA_CPU_TARGET was first of all not necessary for me before using 1.8.3, and with 1.9.0 it has not solved my issue.

[simsurace]

@nlw0, I'm curious to see your dockerfile to confirm that JULIA_CPU_TARGET is set when the precompilation happens in the docker build phase. Maybe calling Pkg.precompile() explicitly helps.

The first example of @nrontsis does not set JULIA_CPU_TARGET anywhere in the dockerfile. I'm curious whether the suggestion I posted works in the setting of @nrontsis.

I think the first step is to successfully set the target to generic and get rid of cache invalidations in the docker run phase. The second step (if that works and there is indeed no bug) is to add more targets to allow for optimizations where applicable.

[nrontsis]

Thanks for your replies.

@vchuravy

Pkgimages support the same mechanism, with the exception that you need to set JULIA_CPU_TARGETS as an environment flag.

You do need to generally know what target system you are aiming for

Thanks, I did try to specify the exact cpu target by passing cpu_target="znver3" (which I identified to be the cpu target for AMD EPYC 7B13) in PackageCompiler.create_sysimage. Nevertheless, setting also ENV JULIA_CPU_TARGET=znver3 as the second line in my Dockerfiles does not resolve the issues: I still get cache invalidations for code precompiled with Pkg.instantiate():

┌ Debug: Rejecting cache file /root/.julia/compiled/v1.9/MyApp/6XOLX_xZ2YY.ji for MyApp [f943f3d7-887a-4ed5-b0c0-a1d6899aa8f5] since pkgimage can't be loaded on this target

and the system image not being loadable

ERROR: Unable to find compatible target in system image.

@simsurace

Setting JULIA_CPU_TARGET=generic (and passing cpu_target="generic" in PackageCompiler.create_sysimage) results in no cache invalidations in Pkg.instantiate() and, also makes the system image accepted in the target host. That's also the case when using the multi-microarchitecture generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1) suggested by @vchuravy.

The problem I see with this, is that it is not performant for my actual use case (as per the last section of my original post).

[nlw0]

I'm trying to create a small Dockerfile to reproduce my issue. I hope it's ok I occult "Platform Info".

Original

FROM julia:1.9.0-bullseye

RUN useradd -ms /bin/bash app
USER app

# Careful: HOME is set as `/home` during runtime and `/root` during build.
ENV HOME=/home/app

WORKDIR $HOME
COPY . ./

RUN julia -e 'using Pkg; Pkg.add(["Rotations"])'

ENV JULIA_DEBUG=loading
CMD exec julia -e '@show Sys.CPU_NAME; @show ENV["HOME"]; @show pwd(); using InteractiveUtils; versioninfo(); using Rotations; @show MRP(0.1,0.2,0.3)'

Running locally it seems fine

$ docker run xxx
Sys.CPU_NAME = "tigerlake"
ENV["HOME"] = "/home/app"
pwd() = "/home/app"
Julia Version 1.9.0
Commit 8e630552924 (2023-05-07 11:25 UTC)
Platform Info:
   xxx
Environment:
  JULIA_GPG = 3673DF529D9049477F76B37566E3C7DC03D6E495
  JULIA_PATH = /usr/local/julia
  JULIA_VERSION = 1.9.0
  JULIA_DEBUG = loading
┌ Debug: Skipping mtime check for file /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl used by /usr/local/julia/share/julia/compiled/v1.9/Statistics/ERcPL_kQ23B.ji, since it is a stdlib
└ @ Base loading.jl:2795
┌ Debug: Loading object cache file /usr/local/julia/share/julia/compiled/v1.9/Statistics/ERcPL_kQ23B.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/app/.julia/compiled/v1.9/StaticArraysCore/Tzw28_LVvhN.so for StaticArraysCore [1e83bf80-4336-4d27-bf5d-d5a4f845583c]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/app/.julia/compiled/v1.9/StaticArrays/yY9vm_LVvhN.so for StaticArrays [90137ffa-7385-5640-81b9-e52037218182]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/app/.julia/compiled/v1.9/RealDot/dCVlS_LVvhN.so for RealDot [c1ae055f-0cd5-4b69-90a6-9a35b1a98df9]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/app/.julia/compiled/v1.9/Quaternions/hbPLI_LVvhN.so for Quaternions [94ee1d12-ae83-5a48-8b1c-48b8ff168ae0]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/app/.julia/compiled/v1.9/Rotations/SJ1Dh_LVvhN.so for Rotations [6038ab10-8711-5258-84ad-4b1120ba62dc]
└ @ Base loading.jl:1010
MRP(0.1, 0.2, 0.3) = [0.19975377039088948 -0.6709756848261001 0.714065866420437; 0.9172052939365958 0.3844259772237611 0.1046475838719606; -0.3447214527546937 0.634041243459526 0.6922129886118804]

But remotely I get this

Sys.CPU_NAME = "skylake-avx512"
ENV["HOME"] = "/home/app"
pwd() = "/home/app"
Julia Version 1.9.0
Commit 8e630552924 (2023-05-07 11:25 UTC)
Platform Info:
xxx
Environment:
JULIA_GPG = 3673DF529D9049477F76B37566E3C7DC03D6E495
JULIA_PATH = /usr/local/julia
JULIA_VERSION = 1.9.0
JULIA_DEBUG = loading
┌ Debug: Rejecting cache file /home/app/.julia/compiled/v1.9/Rotations/SJ1Dh_LVvhN.ji for Rotations [6038ab10-8711-5258-84ad-4b1120ba62dc] since pkgimage can't be loaded on this target
└ @ Base loading.jl:2706

Now adding arch

FROM julia:1.9.0-bullseye

RUN useradd -ms /bin/bash app
USER app

# Careful: HOME is set as `/home` during runtime and `/root` during build.
ENV HOME=/home/app

WORKDIR $HOME
COPY . ./

RUN julia -e 'using Pkg; Pkg.add(["Rotations"])'

ENV JULIA_CPU_TARGET=generic
ENV JULIA_DEBUG=loading
CMD exec julia -e '@show Sys.CPU_NAME; @show ENV["HOME"]; @show pwd(); using InteractiveUtils; versioninfo(); using Rotations; @show MRP(0.1,0.2,0.3)'

local

Sys.CPU_NAME = "tigerlake"
ENV["HOME"] = "/home/app"
pwd() = "/home/app"
Julia Version 1.9.0
Commit 8e630552924 (2023-05-07 11:25 UTC)
Platform Info:
xxx
Environment:
  JULIA_GPG = 3673DF529D9049477F76B37566E3C7DC03D6E495
  JULIA_PATH = /usr/local/julia
  JULIA_CPU_TARGET = generic
  JULIA_VERSION = 1.9.0
  JULIA_DEBUG = loading
┌ Debug: Skipping mtime check for file /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl used by /usr/local/julia/share/julia/compiled/v1.9/Statistics/ERcPL_kQ23B.ji, since it is a stdlib
└ @ Base loading.jl:2795
┌ Debug: Loading object cache file /usr/local/julia/share/julia/compiled/v1.9/Statistics/ERcPL_kQ23B.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/app/.julia/compiled/v1.9/StaticArraysCore/Tzw28_LVvhN.so for StaticArraysCore [1e83bf80-4336-4d27-bf5d-d5a4f845583c]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/app/.julia/compiled/v1.9/StaticArrays/yY9vm_LVvhN.so for StaticArrays [90137ffa-7385-5640-81b9-e52037218182]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/app/.julia/compiled/v1.9/RealDot/dCVlS_LVvhN.so for RealDot [c1ae055f-0cd5-4b69-90a6-9a35b1a98df9]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/app/.julia/compiled/v1.9/Quaternions/hbPLI_LVvhN.so for Quaternions [94ee1d12-ae83-5a48-8b1c-48b8ff168ae0]
└ @ Base loading.jl:1010
┌ Debug: Loading object cache file /home/app/.julia/compiled/v1.9/Rotations/SJ1Dh_LVvhN.so for Rotations [6038ab10-8711-5258-84ad-4b1120ba62dc]
└ @ Base loading.jl:1010
MRP(0.1, 0.2, 0.3) = [0.19975377039088948 -0.6709756848261001 0.714065866420437; 0.9172052939365958 0.3844259772237611 0.1046475838719606; -0.3447214527546937 0.634041243459526 0.6922129886118804]

Remote

Sys.CPU_NAME = "broadwell"
ENV["HOME"] = "/home/app"
pwd() = "/home/app"
Julia Version 1.9.0
Commit 8e630552924 (2023-05-07 11:25 UTC)
Platform Info:
xxx
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-14.0.6 (ORCJIT, broadwell)
Threads: 1 on 2 virtual cores
Environment:
JULIA_GPG = 3673DF529D9049477F76B37566E3C7DC03D6E495
JULIA_PATH = /usr/local/julia
JULIA_CPU_TARGET = generic
JULIA_VERSION = 1.9.0
JULIA_DEBUG = loading
┌ Debug: Rejecting cache file /home/app/.julia/compiled/v1.9/Rotations/SJ1Dh_LVvhN.ji for Rotations [6038ab10-8711-5258-84ad-4b1120ba62dc] since pkgimage can't be loaded on this target
└ @ Base loading.jl:2706

You can see it specifically reports JULIA_CPU_TARGET = generic in the log. Now finally, if I set the base image to FROM julia:1.8.3-bullseye, it's a bliss.

Sys.CPU_NAME = "skylake-avx512"
ENV["HOME"] = "/home/app"
pwd() = "/home/app"
Julia Version 1.8.3
Commit 0434deb161e (2022-11-14 20:14 UTC)
Platform Info:
xxx
Environment:
JULIA_GPG = 3673DF529D9049477F76B37566E3C7DC03D6E495
JULIA_PATH = /usr/local/julia
JULIA_CPU_TARGET = generic
JULIA_VERSION = 1.8.3
JULIA_DEBUG = loading
┌ Debug: Loading cache file /home/app/.julia/compiled/v1.8/StaticArraysCore/Tzw28_J9oAM.ji for StaticArraysCore [1e83bf80-4336-4d27-bf5d-d5a4f845583c]
└ @ Base loading.jl:806
┌ Debug: Loading cache file /home/app/.julia/compiled/v1.8/StaticArrays/yY9vm_J9oAM.ji for StaticArrays [90137ffa-7385-5640-81b9-e52037218182]
└ @ Base loading.jl:806
┌ Debug: Loading cache file /home/app/.julia/compiled/v1.8/RealDot/dCVlS_J9oAM.ji for RealDot [c1ae055f-0cd5-4b69-90a6-9a35b1a98df9]
└ @ Base loading.jl:806
┌ Debug: Loading cache file /home/app/.julia/compiled/v1.8/Quaternions/hbPLI_J9oAM.ji for Quaternions [94ee1d12-ae83-5a48-8b1c-48b8ff168ae0]
└ @ Base loading.jl:806
┌ Debug: Loading cache file /home/app/.julia/compiled/v1.8/Rotations/SJ1Dh_J9oAM.ji for Rotations [6038ab10-8711-5258-84ad-4b1120ba62dc]
└ @ Base loading.jl:806
MRP(0.1, 0.2, 0.3) = [0.19975377039088948 -0.6709756848261001 0.714065866420437; 0.9172052939365958 0.3844259772237611 0.1046475838719606; -0.3447214527546937 0.634041243459526 0.6922129886118804]
Container called exit(0).

I hope you can see in my point of view this is a form of regression. Things are just fine on 1.8.3 and fail on 1.9.0 . And whatever might be the arcane reasons for it, perhaps not even Julia's fault, I'm not sure how I could work around this.

[nrontsis]

@nlw0 could you try moving ENV JULIA_CPU_TARGET=generic at the top (e.g. second line) in your last Dockerfile?

[nlw0]

@nlw0 could you try moving ENV JULIA_CPU_TARGET=generic at the top (e.g. second line) in your last Dockerfile?

This did it, you're an angel! :) I'll try with multiple targets now, but I guess that should do the trick, thanks a lot!

[mkitti]

If you want to return to pre-1.9 behavior, you could just disable pkgimages. julia --pkgimages=no.

[simsurace]

The problem I see with this, is that it is not performant for my actual use case (as per the last section of my original post).

I was actually talking about the package images.

If you do

ENV JULIA_CPU_TARGET=generic
RUN julia --project=. -e "using Pkg; Pkg.precompile()"

you should get rid of cache invalidation as the experiment by @nlw0 shows.

To get more performant, you can then add additional targets that cover newer architectures/features.

[simsurace]

If you want to return to pre-1.9 behavior, you could just disable pkgimages. julia ----pkgimages=no.

This doesn't work for the use case of reducing TTFX in production, where 1.9 brings substantial improvements (if it works i.e. caches don't get invalidated).

[nrontsis]

If you do
...
you should get rid of cache invalidation as the experiment by @nlw0 shows.
To get more performant, you can then add additional targets that cover newer architectures/features.

@simsurace please read my comments again:

• Using JULIA_CPU_TARGET=generic works but is not performant
• When specifying the exact target architecture via JULIA_CPU_TARGET=znver3 I get cache invalidations and the system image rejected

[KristofferC]

Using JULIA_CPU_TARGET=generic works but is not performant

I think the idea is to have the generic and then add features on top of that. For example, as written in #50102 (comment), the default one for Julia (x64) is generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1).

[nrontsis]

@KristofferC generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1) is also not performant for my use case. Are you suggesting that I must use generic;znver3 instead of znver3?

[KristofferC]

As an example, yes. Add whatever features you want to multiversion for.