`Err: EvalSymlinks: too many links` when using skopeo to pull images

[Johan-Liebert1]

Skopeo uses /var/tmp as a tmpdir while pulling images and in bootc we symlink /var/tmp to /proc/1/root/var/tmp.

EvalSymlinks function in the Go compiler doesn't understand how to handle this and throws the error Err: EvalSymlinks: too many links

[cgwalters]

Previously in #919 we switched to dynamic bind mounts for other cases, I think it would make sense to do the same for /var/tmp. That will make it an actual mount and avoid this symlink issue.

[cgwalters]

But one thing I don't understand here is; why are we not hitting this code path currently? What changes have you made to trigger it?

[Johan-Liebert1]

It's not on the bootc end. This error popped up when integrating with https://github.com/containers/composefs-rs, which uses https://github.com/containers/containers-image-proxy-rs.

I've only skimmed through the bootc code, but I see that we're always creating a directory inside of /var/tmp before passing it onto skopeo; maybe that's why we haven't faced this issue before? ComposefsRs does no such thing and lets skopeo, or rather https://github.com/containers/containers-image-proxy-rs handle everything.

Again, I might be wrong with the above assumption

[cgwalters]

I see that we're always creating a directory inside of /var/tmp before passing it onto skopeo

Where is that? I don't see it.

One thing I do see is the unprivileged_subprocess() code which could be having an effect here. I think it's a pretty good idea to do this by default; we could refactor to share code there, or change composefs-rs to support passing a proxy config. In fact we will definitely need to support passing a proxy config in order to honor /etc/ostree/auth.json etc.

[Johan-Liebert1]

Where is that? I don't see it.

I assumed wrong, I guess.

I'm guessing, not a 100% sure, skopeo::copy doesn't actually use the tmpdir to do anything, else it would've also run into this issue.

I traced my issue down to https://github.com/containers/skopeo/blob/main/vendor/github.com/containers/image/v5/oci/layout/oci_transport.go#L89

which calls EvalSymlinks internally on /var/tmp and then fails.

[Johan-Liebert1]

Out of curiosity, is there a reason why we're doing this from inside bootc? We could bind mount /var/tmp in the container itself right? Is there some edge case that we tried to avoid with this one?

[cgwalters]

Out of curiosity, is there a reason why we're doing this from inside bootc? We could bind mount /var/tmp in the container itself right? Is there some edge case that we tried to avoid with this one?

We can definitely ask callers to use podman run -v /var/tmp:/var/tmp explicitly, but we used to have a lot of bind mounts. See again #826 which was linked from #919 - basically our "podman run" invocation was incredibly unwieldy.

Of course nowadays we have system-reinstall-bootc but that doesn't cover every case.

Anyways, testing a PR to do a dynamic bind for /var/tmp now.

[cgwalters]

PR in #1294