-
Notifications
You must be signed in to change notification settings - Fork 424
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support singularity pull from quay.io #749
Comments
Hey @nathanweeks ! The --size assumes a properly formed version 1 manifest, so we would need to look at what quay.io is delivering. I suspect based on your errors that it's not the same manifest, but I'll need to check this out. |
ah, and the tag "latest" is used by default if the user doesn't specify one. If there is no tag "latest" then you would see that bug. The reason it is trying to retrieve a manifest / tags is likely because there was a 401 response for the first call. |
Here's an example manifest returned from quay.io for the m4 biocontainer (chosen since it does have a latest tag) https://quay.io/v2/biocontainers/m4/manifests/latest Looks like quay.io is using manifest-v2-1 (which has fsLayers with no size), where singularity is expecting manifest-v2-2 (layers with size). https://docs.docker.com/registry/spec/manifest-v2-1/ Using the --size option to singularity pull it does work ok
|
Hey @dctrud ! We do support both manifest versions, and what we are able to do with the images (eg estimate size) has a dependency on that. It looks like the camu image has the following tags:
so given that you want a tag, and you don't know latest, you should specify it, here is internally what happens when you ask for latest (that doesn't exist):
but if I specify the tag:
I get the manifest:
and the images too:
The reason we default to latest and then fail is because the user should be specific about the tag. There is no good programmatic way to determine which tag is "best." On command line you would need to do:
For the error with --size, there is just no (currently) good way to get it from a version 1 manifest, so our options are to estimate (and then most likely fail) or have the user specify. I think it's less frustrating to fail immediately and then specify then to start download and extraction, and then fail after already having waited a bit for that. |
Hi @vsoch, Thanks for checking. I have no problems with specifying a tag, and I guess I'll have to wait for Quay.io to update to schema 2 to avoid having to overprovision the image size with the --size option. However, while specifying a tag and --size seems to cause the canu Singularity image to be successfully created:
I get warnings when I shell into it:
It looks like /var/singularity doesn't exist, the lack of resolv.conf affects DNS resolution, and "df" emits many errors about /var/singularity/mnt/session not existing:
This also happens when I use @dctrud 's I haven't seen this issue occur for Singularity images created from Docker Hub, just Quay.io. |
I think we've seen some issues with time - @GodloveD could this be related? |
Hi @vsoch. I'm sorry but I can't remember what this is in reference to. Could you job my memory? Did I file an issue? |
you are right @GodloveD ! Apologies for my misremember! I am thinking about this issue #682 with @pescobar and @gmkurtzer and @moskalenko. It seems like there are two issues, possibly related but I'm not sure. The first is missing the timezone, and the second is with regard to the session directory. The bind mount warnings are ok, but those particular locations are unexpected. Just for a sanity check - @nathanweeks did you completely remove the old singularity (as in,all the old libraries) and install afresh? |
@vsoch: I don't have administrative access to that CentOS 6.7 cluster, but I just used Vagrant to spin up a fresh ubuntu/xenial64, and get a similar warning regarding the lack of resolve.conf (with a different pathname), though the warning regarding /etc/localtime is absent:
|
could you give me exact steps to reproduce? |
hey we've reproduced the error - looking into it now! |
hey @nathanweeks ! A few things we learned: 1. Quay tar.gz download url refreshWe sometimes for a layer would actually get an html page:
Notice the message that Quay is loading "please try again" - this will trigger an ERROR when the layer is untarred, but what it means is that Quay is returning status code 200 when it shouldn't be, maybe something to ask them about. 2. Weirdo FileThe root of your issue is the file that is distributed with the image is weird, and @gmkurtzer was able to fix by just removing it and creating an empty one. Here is the quick fix:
and then wget works
So I think perhaps looks into the container generation, that might give hint to why the file is weird. |
Hi @vsoch , Regarding 1: that's good to know. All of the .gz files in my ~/.singularity/docker directory are valid compressed files, so that doesn't appear to have affected me thus far. Regarding 2: I see that in the singularity m4.img, the initial /etc/resolv.conf is a symbolic link:
I verified that by following the directions you provided to remove this /etc/resolv.conf symbolic link and replace it with an empty file, the next time I invoke However, if I use docker to pull the image from quay.io, the resulting container has the same valid /etc/resolv.conf from the host OS without having to go through any workarounds (note that /etc/localtime doesn't exist, but I don't think that's as important):
|
hey @nathanweeks do you know where the recipe for this container lives? I looked in the following places but couldn't find anything: Is quay/biocontainers serving containers without sharing the metadata and recipes? It would be helpful to try and see what's going on. If it isn't available, can you find a container with this issue that does have a recipe available? |
This is more complicated than I originally thought... The BioContainers paper describes its method for auto-generating container images from BioConda packages---without Dockerfiles--- using a tool called involucro, and automatically pushing the resulting images to quay.io. It appears that regular (non-Bio) conda packages are manually sprinkled into the mix, including m4, from a repository of "mulled" applications: https://github.com/BioContainers/mulled/blob/master/packages.tsv However, I think the issue is that one of the image layers (~/.singularity/docker/sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500.tar.gz) appears to be an (old) busybox image:
Checking BusyBox @ Docker Hub (https://hub.docker.com/r/library/busybox/tags/) for an image of similar vintage, I found:
...which sure enough, has /etc/resolv.conf symlinked to ../tmp/resolv.conf. I'm not sure if the BioContainers project could be easily persuaded to update all its container images to use a newer BusyBox image that doesn't have this symlink (and it looks like the ones subsequent to buildroot-2014.02 don't)... However, Docker does not have a problem with /etc/resolv.conf being a symlink. Is there a reason why Singularity will overwrite /etc/resolv.conf when it's a regular file, but not when it's a symlink? |
oh my gosh this makes perfect sense! I was assessing a bunch of operating systems, busy box being one of them, and I did a simple algorithm that compared each OS against a version of itself, at each timepoint with one file removed, and I sorted by file date (meaning I removed newer ones first, so the base operating systems would be last). For most images I saw a nice downward trend, with a small constant line in the case of finding a symbolic link (because "removing" a symbolic link doesn't actually remove the file.) But for busybox I saw that (for it's tiny set of 441 file / folder things) most of them are symbolic links. I'm completely not surprised about the
which shouldn't erase tmp still, because it is basically saying "if I don't find tmp, create it with this permission" You can try it like:
But then we do this:
so perhaps by this step, we have a link that is linked to a file that just got mounted over by the host's '/tmp'? Arguably that wouldn't matter if
and to answer your question:
The problem I think is summarized here - this copy command may have the content updated for the linked file (eg, copying to the symlink would mean that the
and then you see as you would expect, shelling into the container shows the resolv.conf is still linked to the one in TLDR: Singularity will have the same behavior to try and mount the hosts resolv.conf regardless, unless there is an issue that it cannot. My best guesses for why Docker doesn't have a trouble with a symlink are 1) possibly because it is an isolated environment, and not trying to bind to the hosts, or 2) it doesn't do any kind of mounting of '/tmp', for work (at least I don't think). So I think the extent to what Singularity could do is to add a @gmkurtzer what are your thoughts? Would the tweak of the |
@vsoch busybox cp does not have the --remove-destination flag! but remove the destination before copying...
portable solution?
|
@truatpasteurdotfr the cp --remove-destination happens on the host I believe. |
right :P, but if your host on alpine linux, /bin/sh is just busybox! |
@gmkurtzer What do you think about @vsoch comment on target removal before the cp? |
Is this still an issue in the current release? I successfully pulled from quay.io just today! |
@rdwrt : |
Hey all, I think this is solved now that we're directly using the OCI libraries to pull OCI Images. I'm closing, please open a new issue if this is still broken in 3.x |
biocontainers isn't around anymore, but here is a container to test pulling... $ singularity pull docker://quay.io/fhcrc-microbiome/kraken2
INFO: Starting build...
Getting image source signatures
Copying blob sha256:7b722c1070cdf5188f1f9e43b8413157f8dfb2b4fe84db3c03cb492379a42fcc
41.51 MiB / 41.51 MiB [====================================================] 8s
Copying blob sha256:5fbf74db61f1459176d8647ba8f53f8e6cf933a2e56f73f0e8da81213117b7e9
847 B / 847 B [============================================================] 0s
Copying blob sha256:ed41cb72e5c918bdbd78e68f02930a3f1cf1d6079402b0a5b19de8508e67b766
526 B / 526 B [============================================================] 0s
Copying blob sha256:7ea47a67709ebea8efed59fbda703dbd00a0d2cae7e2808959744bfa30bfc0e9
168 B / 168 B [============================================================] 0s
Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
32 B / 32 B [==============================================================] 0s
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Copying blob sha256:5e4d1e49f3c9a7cdefb3aa0ea25b84e21501770a5eae2599145b7f3c554c0e51
189.17 MiB / 189.17 MiB [=================================================] 39s
Copying blob sha256:042425e6b51c4559795baf1b3e7e2a781b30a108cdd2d1f92e35d2fb04e27ca5
553.41 KiB / 553.41 KiB [==================================================] 0s
Copying config sha256:53ebf70cf70f7fa76fd3a50f43479249b7eb30e4a18449a35a3a5b62e5580988
3.46 KiB / 3.46 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO: Creating SIF file...
INFO: Build complete: kraken2_latest.sif Works great! And yes I'm definitely too scared to try running a container named "kraken2." |
Biocontainers does exist :-)
https://twitter.com/ypriverol/status/1105510148511072256?s=21
Can confirm works nicely.
|
But on quay.io? We were testing out that registry, specifically. |
Yes indeedy...
https://quay.io/organization/biocontainers
|
The BioContainers project (https://biocontainers.pro/) uses quay.io as its Docker registry. Per #491, it looks like quay.io should be a supported Docker registry in Singularity 2.3. However, two issues:
Using the singularity size.py script to debug:
The tags/list URL returns:
No "latest" tag, though the tags are listed from least recent to most recent (not sure if that's guaranteed or just a side effect of lexicographic ordering).
The text was updated successfully, but these errors were encountered: