Allow arbitrary headers to sent with Bazel Downloader #17829

thesayyn · 2023-03-20T17:41:12Z

Description of the feature request:

Currently, bazel sets some headers when sending an HTTP request to fetch artifacts over the internet. While this works well for most cases, it doesn't work at all when the remote server requires some HTTP headers to be set to special values.

For instance, currently bazel sets all the Accept header to text/html, image/gif, image/jpeg, */* in outgoing HTTP requests. In my case, its Accept header that needs to be set to application/vnd.oci.image.manifest.v1+json so that docker registry sees no problem responding when the image has oci manifest types.

See: #16659 (comment)

Related to #16682

What underlying problem are you trying to solve with this feature?

No response

Which operating system are you running Bazel on?

darwin/arm64

What is the output of `bazel info release`?

No response

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

No response

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

The text was updated successfully, but these errors were encountered:

thesayyn · 2023-03-20T17:41:53Z

cc @Wyverald

alexeagle · 2023-03-24T02:56:47Z

Note, we'll need to add a workaround in rules_oci for now, however it uses a requires using another tool during the repository rule, making it require a fake toolchain resolution and also the download it makes is not cached anywhere.

fmeum · 2023-09-11T08:58:05Z

@meteorcloudy Do you see any potential issues with allowing rctx.download[_and_extract] to set arbitrary headers?

@thesayyn Would you be interested in contributing this functionality if you get the okay from maintainers? I could help with pre-reviews and references if needed.

meteorcloudy · 2023-09-11T09:16:47Z

Do you see any potential issues with allowing rctx.download[_and_extract] to set arbitrary headers?

Not really. @tjgq integrated credential helper for repository download, do you have any concern here?

tjgq · 2023-09-11T09:20:30Z

We already merge authentication headers from the credential helper and the argument to rctx.download* so adding additional headers from the Starlark side should work.

thesayyn · 2023-09-11T18:54:35Z

@thesayyn Would you be interested in contributing this functionality if you get the okay from maintainers? I could help with pre-reviews and references if needed.

Yes, I'd be willing to do that.

thesayyn · 2023-09-12T20:49:39Z

@fmeum I'd appreciate your points on #19501

xuzhenglun · 2023-11-01T17:57:14Z

I know this may not be directly related to bazel, but this is a message for others who are directed here by the rules_oci error log.

An checksum failed was throwed, and the error log looks like below:

WARNING: Could not fetch the manifest. Either there was an authentication issue or trying to pull an image with OCI image media types.
Falling back to using `curl`. See https://github.com/bazelbuild/bazel/issues/17829 for the context.
Analyzing: target ... (89 packages loaded, 6572 targets configured)
INFO: Repository rules_oci~1.4.0~oci~distroless_static_linux_amd64 instantiated at:
callstack not available
Repository rule oci_pull defined at:
/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl:427:27: in <toplevel>
ERROR: An error occurred during the fetch of repository 'rules_oci~1.4.0~oci~distroless_static_linux_amd64':
Traceback (most recent call last):
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 378, column 57, in _oci_pull_impl
mf, mf_len, mf_digest = downloader.download_manifest(rctx.attr.identifier, "manifest.json")
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 319, column 74, in lambda
download_manifest = lambda identifier, output: _download_manifest(rctx, state, identifier, output),
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 281, column 18, in _download_manifest
_download(
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 247, column 23, in _download
return download_fn(
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/download.bzl", line 129, column 29, in _download
cache_it = rctx.download(
Error in download: com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException: Checksum was e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 but wanted 6706c73aae2afaa8201d63cc3dda48753c09bcd6c300762251065c0f7e602b25
ERROR: <builtin>: fetching oci_pull rule //:rules_oci~1.4.0~oci~distroless_static_linux_amd64: Traceback (most recent call last):
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 378, column 57, in _oci_pull_impl
mf, mf_len, mf_digest = downloader.download_manifest(rctx.attr.identifier, "manifest.json")
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 319, column 74, in lambda
download_manifest = lambda identifier, output: _download_manifest(rctx, state, identifier, output),
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 281, column 18, in _download_manifest
_download(
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/pull.bzl", line 247, column 23, in _download
return download_fn(
File "/home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0/oci/private/download.bzl", line 129, column 29, in _download
cache_it = rctx.download(
Error in download: com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException: Checksum was e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 but wanted 6706c73aae2afaa8201d63cc3dda48753c09bcd6c300762251065c0f7e602b25
ERROR: /home/admin/.cache/bazel/_bazel_admin/23e125c776c0411833d0c469fe9d82fd/external/rules_oci~1.4.0~oci~distroless_static/BUILD.bazel:1:6: @rules_oci~1.4.0~oci~distroless_static//:distroless_static depends on @rules_oci~1.4.0~oci~distroless_static_linux_amd64//:distroless_static_linux_amd64 in repository @rules_oci~1.4.0~oci~distroless_static_linux_amd64 which failed to fetch. no such package '@rules_oci~1.4.0~oci~distroless_static_linux_amd64//': com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException: Checksum was e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 but wanted 6706c73aae2afaa8201d63cc3dda48753c09bcd6c300762251065c0f7e602b25

I believe the reason that causes this problem is here. To cache the file downloaded from curl, here using ctx.download downloads file from local which is impossible to fetch from remote downloader machine.

To workaround this, flag --experimental_remote_downloader_local_fallback=true could be helpful.

releate:

@fmeum

Coming in 7.1.0, rctx.download will accept a headers parameter allowing arbitrary headers to be passed to the downloaders. Link to issue: bazelbuild/bazel#17829 cc: @fmeum

sushain97 · 2023-12-20T08:30:59Z

Is this something that will require changes to FetchBlob RPC? That is, https://github.com/bazelbuild/remote-apis/blob/7e33c12ee96192f12b7fbb089e6fe1af6a29ef14/build/bazel/remote/asset/v1/remote_asset.proto#L154 doesn't include a mechanism for the client to specify headers.

thesayyn · 2024-01-03T23:43:35Z

I don't there's a mechanism for it for remote downloaders.

sushain97 · 2024-01-03T23:57:07Z

I agree - it sounds like it'll require an addition to the protocol.

fixes #17829 Closes #19501. PiperOrigin-RevId: 588750878 Change-Id: I11c982864135d8e1c4420099ce27f67accd3fe20

…ct] (#20979) fixes #17829 Closes #19501. PiperOrigin-RevId: 588750878 Change-Id: I11c982864135d8e1c4420099ce27f67accd3fe20 Co-authored-by: thesayyn <thesayyn@gmail.com>

iancha1992 · 2024-02-22T23:53:37Z

A fix for this issue has been included in Bazel 7.1.0 RC1. Please test out the release candidate and report any issues as soon as possible. Thanks!

sushain97 · 2024-02-24T23:53:27Z

I've opened #21490 which is necessary so that rulesets which adopt this feature don't break users of the remote downloader service. If it looks good, I'd appreciate it if we could pull into either 7.1.0 or 7.1.1.

iancha1992 · 2024-02-26T07:16:47Z

I've opened #21490 which is necessary so that rulesets which adopt this feature don't break users of the remote downloader service. If it looks good, I'd appreciate it if we could pull into either 7.1.0 or 7.1.1.

cc: @Wyverald @meteorcloudy

@Wyverald

Related to #17829 and 2697e0c I don't love this design but according to the Remote Asset API spec, this is an intended use of qualifiers: https://docs.google.com/document/d/10ari9WtTTSv9bqB_UU-oe2gBtaAA7HyQgkpP-RFP80c/edit#heading=h.sixrlhdnkfoa. cc @Wyverald @jmillikin Closes #21490. PiperOrigin-RevId: 610688317 Change-Id: I272f63a6bc4ea432503003ee907ca012f6d641cf

@Wyverald

Related to bazelbuild#17829 and bazelbuild@2697e0c I don't love this design but according to the Remote Asset API spec, this is an intended use of qualifiers: https://docs.google.com/document/d/10ari9WtTTSv9bqB_UU-oe2gBtaAA7HyQgkpP-RFP80c/edit#heading=h.sixrlhdnkfoa. cc @Wyverald @jmillikin Closes bazelbuild#21490. PiperOrigin-RevId: 610688317 Change-Id: I272f63a6bc4ea432503003ee907ca012f6d641cf

@Wyverald

Related to #17829 and 2697e0c I don't love this design but according to the Remote Asset API spec, this is an intended use of qualifiers: https://docs.google.com/document/d/10ari9WtTTSv9bqB_UU-oe2gBtaAA7HyQgkpP-RFP80c/edit#heading=h.sixrlhdnkfoa. cc @Wyverald @jmillikin Closes #21490. Commit 2195baa PiperOrigin-RevId: 610688317 Change-Id: I272f63a6bc4ea432503003ee907ca012f6d641cf Co-authored-by: Sushain Cherivirala <sushain@skc.name>

Damangir · 2024-03-12T03:25:11Z

We already merge authentication headers from the credential helper and the argument to rctx.download* so adding additional headers from the Starlark side should work.

@tjgq Just want to flag that credential_helper do not override the headers, it just adds another header line. The headers that Bazel sends when credential_helper returns Accept are something like this:

    Accept: text/html, image/gif, image/jpeg, */*\r\n
    User-Agent: Bazel/release 7.1.0\r\n
    Authorization: Bearer XXXXXX\r\n
    Accept-Encoding: gzip\r\n
    Accept: application/octet-stream\r\n

The tests in starlark_repository_test.sh pass only since testing_server.py does not correctly merge headers despite RFC9110.

If you

python src/test/shell/bazel/testing_server.py --dump_headers headers.txt always /dev/null

and then

curl -L http://localhost:43726/ -H "Accept: this" -H "Accept: that" 
cat header.txt
{"Accept": "that"}

Only the last header is captured.

@fmeum

Coming in 7.1.0, rctx.download will accept a headers parameter allowing arbitrary headers to be passed to the downloaders. Link to issue: bazelbuild/bazel#17829 cc: @fmeum

thesayyn mentioned this issue Mar 20, 2023

Use http_archive with private Gitlab repo #16659

Closed

ShreeM01 added type: feature request untriaged team-Remote-Exec Issues and PRs for the Execution (Remote) team labels Mar 20, 2023

Wyverald added team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. and removed team-Remote-Exec Issues and PRs for the Execution (Remote) team labels Mar 20, 2023

Wyverald added P3 We're not considering working on this, but happy to review a PR. (No assignee) and removed untriaged labels Mar 28, 2023

alexeagle mentioned this issue Apr 25, 2023

Requires newer curl bazel-contrib/rules_oci#183

Closed

thesayyn mentioned this issue May 17, 2023

[remote/downloader] Wire credential helper to repository downloads #16682

Closed

thesayyn mentioned this issue Sep 10, 2023

allow value of --credential_helper to be an inline bash script #19475

Closed

thesayyn mentioned this issue Sep 12, 2023

feat: add support for arbitrary headers rctx.download[_and_extract] #19501

Closed

thesayyn mentioned this issue Sep 14, 2023

feat: implement apko rule chainguard-dev/rules_apko#1

Merged

ihavespoons mentioned this issue Oct 19, 2023

oci_pull gets 401 Unauthorized on curl. bazel-contrib/rules_oci#396

Closed

copybara-service bot closed this as completed in 2b8885e Dec 7, 2023

mortenmj mentioned this issue Dec 18, 2023

Add download_has_headers_param feature bazel-contrib/bazel_features#30

Merged

Wyverald pushed a commit that referenced this issue Jan 22, 2024

Add support for arbitrary headers to rctx.download[_and_extract]

f4e3302

fixes #17829 Closes #19501. PiperOrigin-RevId: 588750878 Change-Id: I11c982864135d8e1c4420099ce27f67accd3fe20

Wyverald mentioned this issue Jan 22, 2024

[7.1.0] Add support for arbitrary headers to rctx.download[_and_extract] #20979

Merged

sushain97 mentioned this issue Feb 24, 2024

Passthrough HTTP headers to remote downloader service #21490

Closed

bazel-io mentioned this issue Feb 26, 2024

[7.1.0] Allow arbitrary headers to sent with Bazel Downloader #21498

Closed

bazel-io mentioned this issue Feb 27, 2024

[7.1.0] Passthrough HTTP headers to remote downloader service #21503

Merged

andrelegault mentioned this issue May 9, 2024

Set Accept header if not set #22308

Closed

luispadron mentioned this issue May 29, 2024

feat: Swift registry support cgrindel/rules_swift_package_manager#1043

Closed

marekkrk mentioned this issue Jun 15, 2024

It appears that remote downloader does not pass HTTP headers buchgr/bazel-remote#758

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow arbitrary headers to sent with Bazel Downloader #17829

Allow arbitrary headers to sent with Bazel Downloader #17829

thesayyn commented Mar 20, 2023

thesayyn commented Mar 20, 2023

alexeagle commented Mar 24, 2023

fmeum commented Sep 11, 2023

meteorcloudy commented Sep 11, 2023

tjgq commented Sep 11, 2023

thesayyn commented Sep 11, 2023

thesayyn commented Sep 12, 2023

xuzhenglun commented Nov 1, 2023 •

edited

Loading

sushain97 commented Dec 20, 2023 •

edited

Loading

thesayyn commented Jan 3, 2024

sushain97 commented Jan 3, 2024

iancha1992 commented Feb 22, 2024

sushain97 commented Feb 24, 2024

iancha1992 commented Feb 26, 2024

Damangir commented Mar 12, 2024

Allow arbitrary headers to sent with Bazel Downloader #17829

Allow arbitrary headers to sent with Bazel Downloader #17829

Comments

thesayyn commented Mar 20, 2023

Description of the feature request:

What underlying problem are you trying to solve with this feature?

Which operating system are you running Bazel on?

What is the output of bazel info release?

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

thesayyn commented Mar 20, 2023

alexeagle commented Mar 24, 2023

fmeum commented Sep 11, 2023

meteorcloudy commented Sep 11, 2023

tjgq commented Sep 11, 2023

thesayyn commented Sep 11, 2023

thesayyn commented Sep 12, 2023

xuzhenglun commented Nov 1, 2023 • edited Loading

sushain97 commented Dec 20, 2023 • edited Loading

thesayyn commented Jan 3, 2024

sushain97 commented Jan 3, 2024

iancha1992 commented Feb 22, 2024

sushain97 commented Feb 24, 2024

iancha1992 commented Feb 26, 2024

Damangir commented Mar 12, 2024

What is the output of `bazel info release`?

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

xuzhenglun commented Nov 1, 2023 •

edited

Loading

sushain97 commented Dec 20, 2023 •

edited

Loading