Support singularity pull from quay.io #749

nathanweeks · 2017-06-13T15:14:52Z

The BioContainers project (https://biocontainers.pro/) uses quay.io as its Docker registry. Per #491, it looks like quay.io should be a supported Docker registry in Singularity 2.3. However, two issues:

Singularity 2.3 is unable to determine the image size:

$ singularity pull docker://quay.io/biocontainers/canu
ERROR: Could not obtain the container size, try using --size
ABORT: Aborting with RETVAL=255

(specifying the --size) Singularity doesn't automatically select a tag:

$ singularity pull --size 4096 docker://quay.io/biocontainers/canu
Initializing Singularity image subsystem
Opening image file: canu.img
Creating 4096MiB image 
Binding image to loop
Creating file system within image 
Image is done: canu.img
Docker image path: quay.io/biocontainers/canu:latest
ERROR Improperly formed manifest, layers or fsLayers must be present

Using the singularity size.py script to debug:

$ SINGULARITY_CONTENTS=canu.txt SINGULARITY_CONTAINER=docker://quay.io/biocontainers/canu /usr/libexec/singularity/python/size.py
...
DEBUG GET https://quay.io/v2/biocontainers/canu/tags/list

The tags/list URL returns:

{"name":"biocontainers/canu","tags":["1.3--0","1.4--1","1.4--pl5.22.0_2","1.5--pl5.22.0_0"]}

No "latest" tag, though the tags are listed from least recent to most recent (not sure if that's guaranteed or just a side effect of lexicographic ordering).

The text was updated successfully, but these errors were encountered:

vsoch · 2017-06-13T16:48:57Z

Hey @nathanweeks ! The --size assumes a properly formed version 1 manifest, so we would need to look at what quay.io is delivering. I suspect based on your errors that it's not the same manifest, but I'll need to check this out.

vsoch · 2017-06-13T17:06:51Z

ah, and the tag "latest" is used by default if the user doesn't specify one. If there is no tag "latest" then you would see that bug. The reason it is trying to retrieve a manifest / tags is likely because there was a 401 response for the first call.

dtrudg · 2017-06-19T19:15:08Z

Here's an example manifest returned from quay.io for the m4 biocontainer (chosen since it does have a latest tag)

https://quay.io/v2/biocontainers/m4/manifests/latest
https://gist.github.com/dctrud/24c57e176374ffbf508b9399f32f7349

Looks like quay.io is using manifest-v2-1 (which has fsLayers with no size), where singularity is expecting manifest-v2-2 (layers with size).

https://docs.docker.com/registry/spec/manifest-v2-1/
https://docs.docker.com/registry/spec/manifest-v2-2/

Using the --size option to singularity pull it does work ok

singularity -vvvv pull --size 4096 docker://quay.io/biocontainers/m4

vsoch · 2017-06-20T14:52:00Z

Hey @dctrud ! We do support both manifest versions, and what we are able to do with the images (eg estimate size) has a dependency on that. It looks like the camu image has the following tags:

In [15]: image.get_tags()
VERBOSE Obtaining tags: https://quay.io/v2/biocontainers/canu/tags/list
DEBUG GET https://quay.io/v2/biocontainers/canu/tags/list
Out[15]: 
[u'1.3--0',
 u'1.4--1',
 u'1.4--pl5.22.0_2',
 u'1.5--pl5.22.0_0',
 u'1.5--pl5.22.0_1']

so given that you want a tag, and you don't know latest, you should specify it, here is internally what happens when you ask for latest (that doesn't exist):

image.get_manifest()
VERBOSE Obtaining manifest: https://quay.io/v2/biocontainers/canu/manifests/latest
DEBUG GET https://quay.io/v2/biocontainers/canu/manifests/latest
DEBUG Http Error with code 404
Out[17]: 
{u'errors': [{u'code': u'MANIFEST_UNKNOWN',
   u'detail': {},
   u'message': u'manifest unknown'}]}

but if I specify the tag:

In [19]: image=DockerApiConnection(image='quay.io/biocontainers/canu:1.4--1')
DEBUG Headers found: Content-Type,Accept
VERBOSE Registry: quay.io
VERBOSE Namespace: biocontainers
VERBOSE Repo Name: canu
VERBOSE Repo Tag: 1.4--1
VERBOSE Version: None
VERBOSE Obtaining tags: https://quay.io/v2/biocontainers/canu/tags/list
DEBUG GET https://quay.io/v2/biocontainers/canu/tags/list
VERBOSE3 Response on obtaining token is None.

In [20]: m=image.get_manifest()
VERBOSE Obtaining manifest: https://quay.io/v2/biocontainers/canu/manifests/1.4--1
DEBUG GET https://quay.io/v2/biocontainers/canu/manifests/1.4--1

I get the manifest:

m["schemaVersion"]
1

and the images too:

image.get_images()
VERBOSE Obtaining manifest: https://quay.io/v2/biocontainers/canu/manifests/1.4--1
DEBUG GET https://quay.io/v2/biocontainers/canu/manifests/1.4--1
DEBUG Image manifest version 2.1 found.
DEBUG Adding digest sha256:a20d56eb049ee07edb390a4d4be1c50e555ea77ebc02c02d45cf219c21b22354
DEBUG Adding digest sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
DEBUG Adding digest sha256:10c3bb32200bdb5006b484c59b5f0c71b4dbab611d33fca816cd44f9f5ce9e3c
DEBUG Adding digest sha256:dfda3e01f2b637b7b89adb401f2f763d592fcedd2937240e2eb3286fabce55f0
DEBUG Adding digest sha256:d2ba336f2e4458a9223203bf17cc88d77e3006d9cbf4f0b24a1618d0a5b82053
DEBUG Adding digest sha256:7ff999a2256f84141f17d07d26539acea8a4d9c149fefbbcc9a8b4d15ea32de7
DEBUG Adding digest sha256:00cf8b9f3d2a08745635830064530c931d16f549d031013a9b7c6535e7107b88
DEBUG Adding digest sha256:3aaade50789a6510c60e536f5e75fe8b8fc84801620e575cb0435e2654ffd7f6
DEBUG Adding digest sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500
Out[23]: 
[u'sha256:a20d56eb049ee07edb390a4d4be1c50e555ea77ebc02c02d45cf219c21b22354',
 u'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4',
 u'sha256:10c3bb32200bdb5006b484c59b5f0c71b4dbab611d33fca816cd44f9f5ce9e3c',
 u'sha256:dfda3e01f2b637b7b89adb401f2f763d592fcedd2937240e2eb3286fabce55f0',
 u'sha256:d2ba336f2e4458a9223203bf17cc88d77e3006d9cbf4f0b24a1618d0a5b82053',
 u'sha256:7ff999a2256f84141f17d07d26539acea8a4d9c149fefbbcc9a8b4d15ea32de7',
 u'sha256:00cf8b9f3d2a08745635830064530c931d16f549d031013a9b7c6535e7107b88',
 u'sha256:3aaade50789a6510c60e536f5e75fe8b8fc84801620e575cb0435e2654ffd7f6',
 u'sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500']

The reason we default to latest and then fail is because the user should be specific about the tag. There is no good programmatic way to determine which tag is "best." On command line you would need to do:

singularity pull --size 4096 docker://quay.io/biocontainers/canu:1.4--1
Initializing Singularity image subsystem
Opening image file: canu-1.4--1.img
Creating 4096MiB image
Binding image to loop
Creating file system within image
Image is done: canu-1.4--1.img
Docker image path: quay.io/biocontainers/canu:1.4--1
Cache folder set to /home/vanessa/.singularity/docker
[1/8] ||----------------------------------|   0.0%

For the error with --size, there is just no (currently) good way to get it from a version 1 manifest, so our options are to estimate (and then most likely fail) or have the user specify. I think it's less frustrating to fail immediately and then specify then to start download and extraction, and then fail after already having waited a bit for that.

nathanweeks · 2017-06-20T16:32:47Z

Hi @vsoch,

Thanks for checking. I have no problems with specifying a tag, and I guess I'll have to wait for Quay.io to update to schema 2 to avoid having to overprovision the image size with the --size option. However, while specifying a tag and --size seems to cause the canu Singularity image to be successfully created:

$ singularity -vvvv pull --size 9182  docker://quay.io/biocontainers/canu:1.5--pl5.22.0_0
...
Done. Container is at: canu-1.5--pl5.22.0_0.img

I get warnings when I shell into it:

$ singularity shell canu-1.5--pl5.22.0_0.img
WARNING: Non existant bind point (file) in container: '/etc/localtime'
WARNING: Bind file destination does not exist in container: /var/singularity/mnt/container/etc/resolv.conf
Singularity: Invoking an interactive shell within container...

It looks like /var/singularity doesn't exist, the lack of resolv.conf affects DNS resolution, and "df" emits many errors about /var/singularity/mnt/session not existing:

Singularity canu-1.5--pl5.22.0_0.img:~> ls /var/   
cache   lib     lock    log     pcmcia  run     spool   tmp     www   
Singularity canu-1.5--pl5.22.0_0.img:~> wget http://www.google.com/
wget: bad address 'www.google.com'
Singularity canu-1.5--pl5.22.0_0.img:~> df
df: /var/singularity/mnt/session: No such file or directory
df: /var/singularity/mnt/session: No such file or directory
df: /var/singularity/mnt/session: No such file or directory
...

This also happens when I use @dctrud 's singularity -vvvv pull --size 4096 docker://quay.io/biocontainers/m4 example.

I haven't seen this issue occur for Singularity images created from Docker Hub, just Quay.io.

vsoch · 2017-06-20T17:53:19Z

I think we've seen some issues with time - @GodloveD could this be related?

GodloveD · 2017-06-20T18:10:39Z

Hi @vsoch. I'm sorry but I can't remember what this is in reference to. Could you job my memory? Did I file an issue?

vsoch · 2017-06-20T19:14:55Z

you are right @GodloveD ! Apologies for my misremember! I am thinking about this issue #682 with @pescobar and @gmkurtzer and @moskalenko. It seems like there are two issues, possibly related but I'm not sure. The first is missing the timezone, and the second is with regard to the session directory. The bind mount warnings are ok, but those particular locations are unexpected. Just for a sanity check - @nathanweeks did you completely remove the old singularity (as in,all the old libraries) and install afresh?

nathanweeks · 2017-06-20T22:55:00Z

@vsoch: I don't have administrative access to that CentOS 6.7 cluster, but I just used Vagrant to spin up a fresh ubuntu/xenial64, and get a similar warning regarding the lack of resolve.conf (with a different pathname), though the warning regarding /etc/localtime is absent:

ubuntu@ubuntu-xenial:~$ singularity pull --size 1024 docker://quay.io/biocontainers/m4
Done. Container is at: m4.img
ubuntu@ubuntu-xenial:~$ singularity shell m4.img
WARNING: Bind file destination does not exist in container: /var/lib/singularity/mnt/overlay/final/etc/resolv.conf
Singularity: Invoking an interactive shell within container...

Singularity> ls /var/lib/singularity
ls: /var/lib/singularity: No such file or directory
Singularity> wget http://www.google.com/
wget: bad address 'www.google.com'

vsoch · 2017-06-20T23:08:21Z

could you give me exact steps to reproduce?

vsoch · 2017-06-21T00:06:15Z

hey we've reproduced the error - looking into it now!

vsoch · 2017-06-21T01:12:51Z

hey @nathanweeks ! A few things we learned:

1. Quay tar.gz download url refresh

We sometimes for a layer would actually get an html page:

$ less /home/gmk/.singularity/docker/sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500.tar.gz
<html>
  <head>
    <title>Quay Loading</title>
    <link rel="stylesheet" href="//netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css">
    <link rel="stylesheet" href="//netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap.no-icons.min.css">
    <link href='//fonts.googleapis.com/css?family=Droid+Sans:400,700' rel='stylesheet' type='text/css'>
    <style type="text/css">
      .information {
         padding: 30px;
         text-align: center;
      }
      .information h3 {
         margin-bottom: 20px;
      }
    </style>
  </head>
  <body>
    <div class="information">
      <img src="/static/img/Quay_horizontal_color.svg" style="max-height: 50px;">
      <h3>Quay is currently loading</h3>
      <h4>
        <p>Please wait and refresh the page shortly to try again.</p>
      </h4>
    </div>
  </body>
</html>

Notice the message that Quay is loading "please try again" - this will trigger an ERROR when the layer is untarred, but what it means is that Quay is returning status code 200 when it shouldn't be, maybe something to ask them about.

2. Weirdo File

The root of your issue is the file that is distributed with the image is weird, and @gmkurtzer was able to fix by just removing it and creating an empty one. Here is the quick fix:

singularity pull -s 1024 docker://quay.io/biocontainers/m4
sudo singularity exec -w m4.img rm -f /etc/resolv.conf
sudo singularity exec -w m4.img touch /etc/resolv.conf
sudo singularity exec -w m4.img touch /etc/localtime
singularity shell m4.img 
Singularity: Invoking an interactive shell within container...
Singularity>

and then wget works

Singularity> wget http://www.google.com
Connecting to www.google.com (216.58.219.36:80)
index.html           100% |*******************************| 11251   0:00:00 ETA
Singularity>

So I think perhaps looks into the container generation, that might give hint to why the file is weird.

nathanweeks · 2017-06-26T15:53:22Z

Hi @vsoch ,

Regarding 1: that's good to know. All of the .gz files in my ~/.singularity/docker directory are valid compressed files, so that doesn't appear to have affected me thus far.

Regarding 2: I see that in the singularity m4.img, the initial /etc/resolv.conf is a symbolic link:

Singularity> ls -l /etc/resolv.conf
lrwxrwxrwx    1 root     root            18 Feb 27  2014 /etc/resolv.conf -> ../tmp/resolv.conf

I verified that by following the directions you provided to remove this /etc/resolv.conf symbolic link and replace it with an empty file, the next time I invoke singularity shell it is replaced by a working /etc/resolv.conf (looks like the one from the host OS).

However, if I use docker to pull the image from quay.io, the resulting container has the same valid /etc/resolv.conf from the host OS without having to go through any workarounds (note that /etc/localtime doesn't exist, but I don't think that's as important):

ubuntu@ubuntu-xenial:~$ docker pull quay.io/biocontainers/m4
...
ubuntu@ubuntu-xenial:~$ docker image ls
REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
quay.io/biocontainers/m4      latest              4941f1b3d523        17 months ago       5.74 MB
ubuntu@ubuntu-xenial:~$ docker run -it --entrypoint /bin/sh 4941f1b3d523
/ # ls -l /etc/localtime
ls: /etc/localtime: No such file or directory
/ # ls -l /etc/resolv.conf
-rw-r--r--    1 root     root           195 Jun 26 14:47 /etc/resolv.conf
/ # cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.0.2.3
search my.domain.com
/ # wget http://www.google.com
Connecting to www.google.com (172.217.11.228:80)
index.html           100% |************************************************************************************************| 10354   0:00:00 ETA

vsoch · 2017-06-26T17:13:33Z

hey @nathanweeks do you know where the recipe for this container lives? I looked in the following places but couldn't find anything:

Is quay/biocontainers serving containers without sharing the metadata and recipes? It would be helpful to try and see what's going on. If it isn't available, can you find a container with this issue that does have a recipe available?

nathanweeks · 2017-06-27T20:35:20Z

This is more complicated than I originally thought... The BioContainers paper describes its method for auto-generating container images from BioConda packages---without Dockerfiles--- using a tool called involucro, and automatically pushing the resulting images to quay.io.

It appears that regular (non-Bio) conda packages are manually sprinkled into the mix, including m4, from a repository of "mulled" applications: https://github.com/BioContainers/mulled/blob/master/packages.tsv
So the recipe for m4 is in the conda repository: https://github.com/conda/conda-recipes/blob/master/m4/meta.yaml
(The BioConda containers are "auto-mulled": https://github.com/BioContainers/auto-mulled)

However, I think the issue is that one of the image layers (~/.singularity/docker/sha256:77c6c00e8b61bb628567c060b85690b0b0561bb37d8ad3f3792877bddcfe2500.tar.gz) appears to be an (old) busybox image:

$ singularity shell m4.img
WARNING: Bind file destination does not exist in container: /var/lib/singularity/mnt/overlay/final/etc/resolv.conf
Singularity: Invoking an interactive shell within container...

Singularity> busybox | head -n 1
BusyBox v1.22.1 (2014-05-23 01:24:27 UTC) multi-call binary.

Checking BusyBox @ Docker Hub (https://hub.docker.com/r/library/busybox/tags/) for an image of similar vintage, I found:

$ singularity pull --size 64 docker://busybox:buildroot-2014.02
...
$ singularity shell busybox-buildroot-2014.02.img
WARNING: Bind file destination does not exist in container: /var/lib/singularity/mnt/overlay/final/etc/resolv.conf
Singularity: Invoking an interactive shell within container...

Singularity> busybox | head -n 1
BusyBox v1.22.1 (2014-05-22 23:22:11 UTC) multi-call binary.
Singularity> ls -l /etc/resolv.conf
lrwxrwxrwx    1 root     root            18 Feb 27  2014 /etc/resolv.conf -> ../tmp/resolv.conf

...which sure enough, has /etc/resolv.conf symlinked to ../tmp/resolv.conf.

I'm not sure if the BioContainers project could be easily persuaded to update all its container images to use a newer BusyBox image that doesn't have this symlink (and it looks like the ones subsequent to buildroot-2014.02 don't)... However, Docker does not have a problem with /etc/resolv.conf being a symlink. Is there a reason why Singularity will overwrite /etc/resolv.conf when it's a regular file, but not when it's a symlink?

vsoch · 2017-06-27T21:27:23Z

oh my gosh this makes perfect sense! I was assessing a bunch of operating systems, busy box being one of them, and I did a simple algorithm that compared each OS against a version of itself, at each timepoint with one file removed, and I sorted by file date (meaning I removed newer ones first, so the base operating systems would be last). For most images I saw a nice downward trend, with a small constant line in the case of finding a symbolic link (because "removing" a symbolic link doesn't actually remove the file.) But for busybox I saw that (for it's tiny set of 441 file / folder things) most of them are symbolic links. I'm completely not surprised about the resolv.conf being one of them.

pre-bootstrap: we primarily load functions, make sure the root we are working at is defined, and we create the various metadata folders. We then dump environment starter templates into the base, and pass on to the bootstrap specific script, which in our case is docker...
docker api: the first relevant step in the docker driver is getting the list of .tar.gz layers, one of them being for the metadata folder. This is where the layers explode, and this is also where we would have the busybox resolv.conf dumped into the working container.
bootstrap-sections: we next generate a bunch of base folders that we expect to mount, and notably, there is this line:

test -d "$SINGULARITY_ROOTFS/tmp" || install -d -m 755 "$SINGULARITY_ROOTFS/tmp"

which shouldn't erase tmp still, because it is basically saying "if I don't find tmp, create it with this permission" You can try it like:

test -d "/tmp" || echo "HELLO"           # no output, /tmp exists
test -d "/tmpmeatball" || echo "HELLO"   # does not exist
HELLO

But then we do this:

mount --no-mtab --rbind "/tmp" "$SINGULARITY_ROOTFS/tmp"

so perhaps by this step, we have a link that is linked to a file that just got mounted over by the host's '/tmp'? Arguably that wouldn't matter if /tmp weren't to be bound again. Then what we do is copy our /etc/resolv.conf to the container:

cp /etc/resolv.conf     "$SINGULARITY_ROOTFS/etc/resolv.conf"

and to answer your question:

Is there a reason why Singularity will overwrite /etc/resolv.conf when it's a regular file, but not when it's a symlink?

The problem I think is summarized here - this copy command may have the content updated for the linked file (eg, copying to the symlink would mean that the /tmp file might have the correct content) but what appears to happen (according to the post) is that the file you (thought) you copied is still the link. The problem, then, is that when we try to do the bind, we skip it, because it's not a file:

DEBUG   [U=1000,P=22524]   container_file_bind()                     Called file_bind(/etc/resolv.conf, /etc/resolv.conf()
VERBOSE [U=1000,P=22524]   container_file_bind()                     Skipping bind file, destination does not exist in container: /etc/resolv.conf

and then you see as you would expect, shelling into the container shows the resolv.conf is still linked to the one in /tmp, which actually now might be completely empty/weird because we did mounting of tmp during the bootstrap (I'm not sure).

TLDR: Singularity will have the same behavior to try and mount the hosts resolv.conf regardless, unless there is an issue that it cannot. My best guesses for why Docker doesn't have a trouble with a symlink are 1) possibly because it is an isolated environment, and not trying to bind to the hosts, or 2) it doesn't do any kind of mounting of '/tmp', for work (at least I don't think).

So I think the extent to what Singularity could do is to add a --remove-destination to the copy of the file. This would make it an actual file, and it would bind to the user's, and possibly work. I think generally it was a good move to remove essential files from /tmp - nothing that is needed to persist should be kept there, and perhaps in some older working strategy they would carry that file around in /tmp if another wasn't available? ¯_(ツ)_/¯ This is another strong point of evidence for having some sanity check / best practices for containers!

@gmkurtzer what are your thoughts? Would the tweak of the cp possibly address this?

truatpasteurdotfr · 2017-06-27T22:09:19Z

@vsoch busybox cp does not have the --remove-destination flag! but remove the destination before copying...

alpine:~$ cp --help
BusyBox v1.26.2 (2017-06-11 06:38:32 GMT) multi-call binary.

Usage: cp [OPTIONS] SOURCE... DEST

Copy SOURCE(s) to DEST

	-a	Same as -dpR
	-R,-r	Recurse
	-d,-P	Preserve symlinks (default if -R)
	-L	Follow all symlinks
	-H	Follow symlinks on command line
	-p	Preserve file attributes if possible
	-f	Overwrite
	-i	Prompt before overwrite
	-l,-s	Create (sym)links
	-u	Copy only newer files
alpine:~/cp.d$ echo a > a
alpine:~/cp.d$ echo b > b
alpine:~/cp.d$ ln -s b c
alpine:~/cp.d$ ls -l
total 8
-rw-r--r--    1 tru      wheel            2 Jun 27 22:05 a
-rw-r--r--    1 tru      wheel            2 Jun 27 22:05 b
lrwxrwxrwx    1 tru      wheel            1 Jun 27 22:05 c -> b
alpine:~/cp.d$ head ?
==> a <==
a

==> b <==
b

==> c <==
b
alpine:~/cp.d$ type cp
cp is hashed (/bin/cp)
alpine:~/cp.d$ cp a c
alpine:~/cp.d$ ls -l
total 12
-rw-r--r--    1 tru      wheel            2 Jun 27 22:05 a
-rw-r--r--    1 tru      wheel            2 Jun 27 22:05 b
-rw-r--r--    1 tru      wheel            2 Jun 27 22:06 c
alpine:~/cp.d$ head ?
==> a <==
a

==> b <==
b

==> c <==
a

portable solution?

/bin/rm target && /bin/cp -a source target

vsoch · 2017-06-27T22:58:35Z

@truatpasteurdotfr the cp --remove-destination happens on the host I believe.

truatpasteurdotfr · 2017-06-29T07:21:23Z

right :P, but if your host on alpine linux, /bin/sh is just busybox!

moskalenko · 2017-11-02T22:15:51Z

@gmkurtzer What do you think about @vsoch comment on target removal before the cp?

ps-account · 2018-02-14T12:14:38Z

Is this still an issue in the current release? I successfully pulled from quay.io just today!

nathanweeks · 2018-02-14T13:36:17Z

@rdwrt : singularity pull works from quay.io, though there is still the issue of the missing /etc/resolv.conf (noted in #749 (comment)) in biocontainers images pulled from quay.io due to their use of a version of busybox that is incompatible with how singularity exposes the host resolv.conf (unless a privileged user subsequently modifies the image with the workaround described in #749 (comment)).

bauerm97 · 2019-03-12T22:22:13Z

Hey all, I think this is solved now that we're directly using the OCI libraries to pull OCI Images. I'm closing, please open a new issue if this is still broken in 3.x

vsoch · 2019-03-12T22:35:28Z

biocontainers isn't around anymore, but here is a container to test pulling...

$ singularity pull docker://quay.io/fhcrc-microbiome/kraken2
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:7b722c1070cdf5188f1f9e43b8413157f8dfb2b4fe84db3c03cb492379a42fcc
 41.51 MiB / 41.51 MiB [====================================================] 8s
Copying blob sha256:5fbf74db61f1459176d8647ba8f53f8e6cf933a2e56f73f0e8da81213117b7e9
 847 B / 847 B [============================================================] 0s
Copying blob sha256:ed41cb72e5c918bdbd78e68f02930a3f1cf1d6079402b0a5b19de8508e67b766
 526 B / 526 B [============================================================] 0s
Copying blob sha256:7ea47a67709ebea8efed59fbda703dbd00a0d2cae7e2808959744bfa30bfc0e9
 168 B / 168 B [============================================================] 0s
Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
 32 B / 32 B [==============================================================] 0s
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Copying blob sha256:5e4d1e49f3c9a7cdefb3aa0ea25b84e21501770a5eae2599145b7f3c554c0e51
 189.17 MiB / 189.17 MiB [=================================================] 39s
Copying blob sha256:042425e6b51c4559795baf1b3e7e2a781b30a108cdd2d1f92e35d2fb04e27ca5
 553.41 KiB / 553.41 KiB [==================================================] 0s
Copying config sha256:53ebf70cf70f7fa76fd3a50f43479249b7eb30e4a18449a35a3a5b62e5580988
 3.46 KiB / 3.46 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO:    Creating SIF file...
INFO:    Build complete: kraken2_latest.sif

Works great! And yes I'm definitely too scared to try running a container named "kraken2."

dtrudg · 2019-03-12T23:53:29Z

Biocontainers does exist :-) https://twitter.com/ypriverol/status/1105510148511072256?s=21 Can confirm works nicely.

vsoch · 2019-03-13T00:00:42Z

But on quay.io? We were testing out that registry, specifically.

dtrudg · 2019-03-13T00:13:33Z

Yes indeedy... https://quay.io/organization/biocontainers

vsoch mentioned this issue Jun 21, 2017

Problem with "Registry: quay.io" in Singularity-file singularityhub/singularityhub.github.io#10

Closed

kyleabeauchamp mentioned this issue Jul 21, 2017

Adding Singularity builds to Bioconda? bioconda/bioconda-recipes#5144

Open

SichongP mentioned this issue Apr 27, 2018

step2 failed on HPC where docker cannot be installed dahak-metagenomics/taco-read-filtering#1

Closed

suujia mentioned this issue Jul 10, 2018

Build dockerfile with Quay bcgsc/orca#56

Closed

pvanheus mentioned this issue Oct 25, 2018

Singularity contains built from bioconda have broken /etc/resolv.conf bioconda/bioconda-recipes#11583

Closed

bauerm97 closed this as completed Mar 12, 2019

a-frantz mentioned this issue Jan 18, 2023

ci updates stjudecloud/workflows#70

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support singularity pull from quay.io #749

Support singularity pull from quay.io #749

nathanweeks commented Jun 13, 2017

vsoch commented Jun 13, 2017

vsoch commented Jun 13, 2017

dtrudg commented Jun 19, 2017 •

edited

Loading

vsoch commented Jun 20, 2017

nathanweeks commented Jun 20, 2017

vsoch commented Jun 20, 2017

GodloveD commented Jun 20, 2017

vsoch commented Jun 20, 2017

nathanweeks commented Jun 20, 2017

vsoch commented Jun 20, 2017

vsoch commented Jun 21, 2017

vsoch commented Jun 21, 2017 •

edited

Loading

nathanweeks commented Jun 26, 2017

vsoch commented Jun 26, 2017

nathanweeks commented Jun 27, 2017

vsoch commented Jun 27, 2017

truatpasteurdotfr commented Jun 27, 2017

vsoch commented Jun 27, 2017

truatpasteurdotfr commented Jun 29, 2017

moskalenko commented Nov 2, 2017

ps-account commented Feb 14, 2018

nathanweeks commented Feb 14, 2018

bauerm97 commented Mar 12, 2019

vsoch commented Mar 12, 2019

dtrudg commented Mar 12, 2019 via email

vsoch commented Mar 13, 2019

dtrudg commented Mar 13, 2019 via email

Support singularity pull from quay.io #749

Support singularity pull from quay.io #749

Comments

nathanweeks commented Jun 13, 2017

vsoch commented Jun 13, 2017

vsoch commented Jun 13, 2017

dtrudg commented Jun 19, 2017 • edited Loading

vsoch commented Jun 20, 2017

nathanweeks commented Jun 20, 2017

vsoch commented Jun 20, 2017

GodloveD commented Jun 20, 2017

vsoch commented Jun 20, 2017

nathanweeks commented Jun 20, 2017

vsoch commented Jun 20, 2017

vsoch commented Jun 21, 2017

vsoch commented Jun 21, 2017 • edited Loading

1. Quay tar.gz download url refresh

2. Weirdo File

nathanweeks commented Jun 26, 2017

vsoch commented Jun 26, 2017

nathanweeks commented Jun 27, 2017

vsoch commented Jun 27, 2017

truatpasteurdotfr commented Jun 27, 2017

vsoch commented Jun 27, 2017

truatpasteurdotfr commented Jun 29, 2017

moskalenko commented Nov 2, 2017

ps-account commented Feb 14, 2018

nathanweeks commented Feb 14, 2018

bauerm97 commented Mar 12, 2019

vsoch commented Mar 12, 2019

dtrudg commented Mar 12, 2019 via email

vsoch commented Mar 13, 2019

dtrudg commented Mar 13, 2019 via email

dtrudg commented Jun 19, 2017 •

edited

Loading

vsoch commented Jun 21, 2017 •

edited

Loading