http: Handle missing but unambiguous ETags in response

[rwe]

Fixes #905 and #784.

If a single ETag is requested in If-None-Match, some servers do not include that (unambiguous) ETag header in the response.

This has not been tested. I'm developing on a Mac and not able to devote much more time to this bug, but figured it was worth at least a beginner shot at fixing it.

Detailed description from #905 (comment) partially reproduced below:

Reproduction Steps

Dockerfile

FROM scratch
ADD https://storage.googleapis.com/gpt-2/models/774M/checkpoint

The first run (or a run after the cache is cleared) succeeds:

DOCKER_BUILDKIT=1 docker build .

[+] Building 0.3s (5/5) FINISHED
 => [internal] load .dockerignore                                                              0.0s
 => => transferring context: 244B                                                              0.0s
 => [internal] load build definition from Dockerfile                                           0.0s
 => => transferring dockerfile: 121B                                                           0.0s
 => https://storage.googleapis.com/gpt-2/models/774M/checkpoint                                0.0s
 => [1/1] ADD https://storage.googleapis.com/gpt-2/models/774M/checkpoint .                    0.0s
 => exporting to image                                                                         0.0s
 => => exporting layers                                                                        0.0s
 => => writing image sha256:9a630453cc77705a4f57c121393a38a4f45f65eba6155ba6c3f27ecab18e2b05   0.0s

Running the same command again:

[+] Building 0.2s (3/4)
 => [internal] load build definition from Dockerfile                                           0.0s
 => => transferring dockerfile: 36B                                                            0.0s
 => [internal] load .dockerignore                                                              0.0s
 => => transferring context: 35B                                                               0.0s
 => ERROR https://storage.googleapis.com/gpt-2/models/774M/checkpoint                          0.1s
------
 > https://storage.googleapis.com/gpt-2/models/774M/checkpoint:
------
invalid not-modified ETag:

Clearing the cache with docker builder prune resets the state.

Cause

If a single ETag is requested in If-None-Match, the server may not include that (unambiguous) ETag header in the response.

Detailed demonstration

Requesting the file directly:

curl --http1.1 -s -L -D /dev/stderr -o /dev/null https://storage.googleapis.com/gpt-2/models/774M/checkpoint

Response headers. Notice the ETag.

HTTP/1.1 200 OK
X-GUploader-UploadID: AEnB2Up6PhdYRb_UZ18VAl30f6XLzFkOoBPnSYSjSKqzk90Go8Zqk-zZoenkbL3inKQz1ozoLjcObKKuIbvOV7XFlSSFO6aW0Q
Expires: Wed, 04 Sep 2019 19:23:16 GMT
Date: Wed, 04 Sep 2019 19:23:16 GMT
Cache-Control: private, max-age=0
Last-Modified: Tue, 20 Aug 2019 15:50:08 GMT
ETag: "ca0368fcd3c4c1a99aca42511d0c1f12"
x-goog-generation: 1566316208157027
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 77
Content-Type: application/octet-stream
x-goog-hash: crc32c=BI0EFw==
x-goog-hash: md5=ygNo/NPEwamaykJRHQwfEg==
x-goog-storage-class: MULTI_REGIONAL
Accept-Ranges: bytes
Content-Length: 77
Server: UploadServer
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"

Requesting the file If-None-Match with that ETag:

curl --http1.1 -s -L -D /dev/stderr -o /dev/null -H 'If-None-Match: "ca0368fcd3c4c1a99aca42511d0c1f12"' https://storage.googleapis.com/gpt-2/models/774M/checkpoint

Notice that the ETag is not included in the response headers.

HTTP/1.1 304 Not Modified
X-GUploader-UploadID: AEnB2UpEqNrI5fDJjglD2f4--3CCzskMyUg-Fo1RZxoqqHq17HG8W_gURMO6uUVy9B6Mg4450GyA4yRTjPqEJY8v6dtxhuHcLQ
Expires: Wed, 04 Sep 2019 19:23:36 GMT
Date: Wed, 04 Sep 2019 19:23:36 GMT
Cache-Control: private, max-age=0
Last-Modified: Tue, 20 Aug 2019 15:50:08 GMT
Content-Length: 0
Server: UploadServer
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"

Requesting the file with multiple ETags in If-None-Match:

curl --http1.1 -s -L -D /dev/stderr -o /dev/null -H 'If-None-Match: "ca0368fcd3c4c1a99aca42511d0c1f12", "foobar"' https://storage.googleapis.com/gpt-2/models/774M/checkpoint

Now the ETag is included to disambiguate.

HTTP/1.1 200 OK
X-GUploader-UploadID: AEnB2UpD_nKA4ZMNpJvC97lMJfyXjcr9myMAxojFypxW8lUNiEGiwdaOtezf74-OBCHDXn7T4Ru57oelrDHb01wY9IMU1Qdl6A
Expires: Wed, 04 Sep 2019 19:26:02 GMT
Date: Wed, 04 Sep 2019 19:26:02 GMT
Cache-Control: private, max-age=0
Last-Modified: Tue, 20 Aug 2019 15:50:08 GMT
ETag: "ca0368fcd3c4c1a99aca42511d0c1f12"
x-goog-generation: 1566316208157027
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 77
Content-Type: application/octet-stream
x-goog-hash: crc32c=BI0EFw==
x-goog-hash: md5=ygNo/NPEwamaykJRHQwfEg==
x-goog-storage-class: MULTI_REGIONAL
Accept-Ranges: bytes
Content-Length: 77
Server: UploadServer
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"

[GordonTheTurtle]

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "fix-etag" git@github.com:erydo/buildkit.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

[BrendonW]

Stupid question? Why is this bug still in "Docker for Mac" which is using a much later version of Buildkit?

Docker Desktop (Mac)

• Version 3.2.2
• Engine 20.10.5

Build fails when ADD tries to check Github repository commit with "failed to load cache key: invalid not-modified ETag"

What can I do to help resolve the issue?

[thaJeztah]

@BrendonW could you open a new ticket with an example to reproduce the issue?