Linear scan of a repository with lots (200+) of tags causes pull IO timeout

[bshi]

This was originally reported at GoogleCloudPlatform#22

It seems when performing "docker pull foo/bar:sometag" the repository performs a linear scan of ALL tags in "foo/bar". When backed by object storage systems like GCS, this can take a long time. It has broken image distribution for us.

[bshi]

As part of this investigation, I discovered that "docker pull foo/bar:sometag" will incur a hit to the API endpoint that lists all tags for 'foo/bar'. This seems a bit wasteful. @dmp42 - as I'm not too familiar with the details of 'docker pull', perhaps you know offhand whether this is indeed unnecessary work and whether it's worth filing a bug in docker?

[dmp42]

@bshi I don't think reporting this is worth the effort - energy IMO is better focused on registry v2 development.

[wking]

On Tue, Oct 21, 2014 at 07:06:37PM -0700, Bo Shi wrote:

As part of this investigation, I discovered that "docker pull
foo/bar:sometag" will incur a hit to the API endpoint that lists all
tags for 'foo/bar'.

I haven't looked at the client-side code (at least recently enough to
remember), but I'd expect you'd need to do this for the new alias
detection (moby/moby#8141). Unless you had a separate endpoint
for "give me all aliases for $NAMESPACE/$REPO:$TAG". We could
actually support something like that efficiently if we had something
like refcount tracking (#606, #409), since I was recording the
referring ([namespace, repository, tag], descendant_id) entry for each
ancestor image id [1,2]. We'd just have to iterate through referrers
and return tags matching $NAMESPACE/$REPO where descendant_id ==
image_id. With my new (proposed) atomic/streaming storage separation
3, accessing the referrers is probably a single hit to your atomic
storage (to the image-references entry for the requested image).

[dmp42]

One of the terribly inefficient thing right now is that we do not only ls but also read the (tag) file contents.
That second part is going away.

Driver will still need to provide an efficient ls (and we can alleviate part of the pain by caching the result).

[dmp42]

@bshi (and other gcs people?) - the new go drivers API #643 is going final and will be merged soon. The time is right to voice concerns :-)

[bshi]

Skimmed the discussion in #643 - it seems like you guys are aware and thinking about the issue of unbounded looping over driver interface methods. One other concern is the underlying storage consistency model and what the registry expects of the drivers. S3 doesn't even have consistent (:P) consistency models across S3 regions.

[dmp42]

Consistency... we think about it, a lot :-) cc @stevvooe