http_archive should support authentication

[ittaiz]

Description of the problem / feature request:

http_archive should support retrieving resources which require authentication

Feature requests: what underlying problem are you trying to solve with this feature?

I want to use http_archive to download tar.gz from a private github repository since from our benchmarks this is ~25% faster than using git_repository with shallow-since (we use a specific commit)

Our suggestion is to use the popular netrc format and have http_archive read it (if exists and indicated that its usage is desired) and use it to authenticate (basically build the relevant authentication header).

Any other information, logs, or outputs that you want to share?

this was discussed with @aehlig and @dslomov which agreed http_archive should support authentication

[buchgr]

+1. I think this should integrate well with Bazel's existing authentication mechanisms.

[dslomov]

@dkelmer wdyt? related to #7469

[dkelmer]

I'll look into netrc, thanks for the tip @ittaiz

@buchgr, which existing authentication mechanisms are you referring to?

[buchgr]

@dkelmer BES and Remote Execution support --google_* authentication flags and soon --aws_* authentication flags. We also have a common abstraction that ideally remote repos would utilize: https://source.bazel.build/bazel/+/master:src/main/java/com/google/devtools/build/lib/runtime/AuthHeadersProvider.java?q=AuthHeadersProvider.java

[ittaiz]

@dkelmer please note #7978 which is under @aehlig’s mentoring

[keith]

Related #9327

[meisterT]

What's left to do here?

[petemounce]

We keep third party deps in a GCS bucket. What options for authenticating do we have? It's not clear to me that .netrc can suffice here, and I'm unfamiliar with how I'd automate getting credentials from gcloud to a form usable by bazel here.

[tjgq]

See #15013. We are planning to add credential helper support to http_archive.

[tjgq]

Deduplicating against #15013.