Support authentication to private maven repositories (Artifactory)

[wstrange]

I tried to use a private Artifactory repository that requires authentication.

I have the appropriate credentials in my ~/.m2/settings.xml, but it looks like the resolver does not reference these credentials. The http request for the artifact gets a 401 unauthorized response.

[akashdayal]

Hope its not too late but you can use
https://:@
as resolver for private repositories.

[wstrange]

Does that imply the credentials would be in in the clear in the generated workspace.bzl? Is there any trick to pick up the credentials from an external source (.m2/settings.xml would be ideal.)

Thanks!

[johnynek]

Maybe we could look in an environmental variable for Auth or ask the user to enter auth at s prompt? We use certificates at Stripe so I have not been in this situation.

On Thu, Apr 19, 2018 at 06:41 Warren Strange ***@***.***> wrote: Does that imply the credentials would be in in the clear in the generated workspace.bzl? Is there any trick to pick up the credentials from an external source (.m2/settings.xml would be ideal.) Thanks! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#95 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEJduJBdjuuzXZOLwh0aKBRcyWMpzeOks5tqL49gaJpZM4QbHRk> .

-- P. Oscar Boykin, Ph.D. | http://twitter.com/posco | http://pobox.com/~boykin

[diacone]

Hope its not too late but you can use
https://:@
as resolver for private repositories.

This doesn't work either. Even though I can clearly see that url contains username/password:

java.io.IOException: Error downloading [https://<user>:<passwd>@my.private.repo/content/repositories/privaterepo/com/google/guava/guava/19.0/guava-19.0-sources.jar] to /home/myuser/.cache/bazel/_bazel_myuser/4d81720667242c59d08111d8c78d97a8/external/com_google_guava_guava/jar/com_google_guava_guava-sources.jar: GET returned 401 Unauthorized

[deadmoose]

I've seen just that with our setup; the repo doesn't cope with auth in the url, it really has to go into headers.

I've been using a hack described in bazelbuild/bazel#5452 (comment)

TLDR: Flag private repos and fall back to shelling out to mvn & letting it handle authentication for me.

It's not a great approach & not in a form I'd consider floating it as a patch, but it's been good enough to unblock us for now.

[diacone]

I patched the workspace to shell out to curl, but it still requires plaintext password in deps.yml and workspace itself, but at least i'm unblocked.

[guw]

@diacone Are you open to share your patch? I'm in a similar spot trying to fetch from a password protected Nexus.

[diacone]

@diacone Are you open to share your patch? I'm in a similar spot trying to fetch from a password protected Nexus.

diff --git a/src/scala/com/github/johnynek/bazel_deps/templates/jar_artifact_backend.bzl b/src/scala/com/github/johnynek/bazel_deps/templates/jar_artifact_backend.bzl
index c4bc52f..b8f58fe 100644
--- a/src/scala/com/github/johnynek/bazel_deps/templates/jar_artifact_backend.bzl
+++ b/src/scala/com/github/johnynek/bazel_deps/templates/jar_artifact_backend.bzl
@@ -1,22 +1,14 @@
 def _jar_artifact_impl(ctx):
     jar_name = "%s.jar" % ctx.name
-    ctx.download(
-        output=ctx.path("jar/%s" % jar_name),
-        url=ctx.attr.urls,
-        sha256=ctx.attr.sha256,
-        executable=False
-    )
-    src_name="%s-sources.jar" % ctx.name
-    srcjar_attr=""
+    result = ctx.execute(["curl", ctx.attr.urls[0], "-o", ctx.path("jar/%s" % jar_name), "--create-dirs"])
+    if result.return_code:
+        fail(result.stderr)
+    src_name = "%s-sources.jar" % ctx.name
+    srcjar_attr = ""
     has_sources = len(ctx.attr.src_urls) != 0
     if has_sources:
-        ctx.download(
-            output=ctx.path("jar/%s" % src_name),
-            url=ctx.attr.src_urls,
-            sha256=ctx.attr.src_sha256,
-            executable=False
-        )
-        srcjar_attr ='\n    srcjar = ":%s",' % src_name
+        ctx.execute(["curl", ctx.attr.urls[0], "-o", ctx.path("jar/%s" % src_name), "--create-dirs"])
+        srcjar_attr = '\n    srcjar = ":%s",' % src_name
 
     build_file_contents = """
 package(default_visibility = ['//visibility:public'])

[guw]

Thanks a lot @diacone. Just to confirm, this diff is for the "runtime" build part, i.e. when Bazel is downloading dependencies? It does not deal with the "resolution" part, i.e. when bazel-deps is supposed to resolve artifacts from a privat Nexus server which requires authentication?

[diacone]

Thanks a lot @diacone. Just to confirm, this diff is for the "runtime" build part, i.e. when Bazel is downloading dependencies? It does not deal with the "resolution" part, i.e. when bazel-deps is supposed to resolve artifacts from a privat Nexus server which requires authentication?

Yes @guw, this is for the runtime part.

[johnynek]

For resolution, I’d rather prompt for a password if a flag is given, since I’d rather not be checking in auth credentials.

I can help you get started on a patch if you like. The first thing to look at would be making an optional flag to ask the user for user/password. Then we can see about passing authentication data to the resolver.

[guw]

@johnynek Thanks! I'm certainly interested in providing a patch. I'm converting a fairly large legacy project. So far I'm running into resolution issue with Coursier and trying Aether now.

Prompting is not something that's going to work as we need to run this on CI. Ideally, I'd like to re-use authentication information stored in ~/.m2/settings.xml. Bazel's native maven_jar seems to support this ootb. Thus, I was wondering if I can supply a different template which would use native.maven_jar for download.

[johnynek]

Interesting that you run resolution on CI. We don’t do that since it is pretty slow.

For downloading on build, you can pretty easily change how resolution works by adjusting the callback .bzl function that processes the resolved artifacts.

[johnynek]

See:
https://github.com/johnynek/bazel-deps/blob/6585033409e09028852403ec15ec0c77851234be/3rdparty/workspace.bzl#L188

You can pass any function you want in which can use any method you need to set up the dependencies after resolution.

[guw]

FYI: Bazel respository_ctx.download currently doesn't support auth.
bazelbuild/bazel#7469

[AlterEgo7]

Any progress now that bazelbuild/bazel#7469 is resolved?

[guw]

FWIW, the issue became less pressing for us since we started using https://github.com/salesforce/bazel-maven-proxy/

[johnynek]

Happy to accept a PR, but I am personally not working on this issue.

I'm also happy to answer questions and guide someone through making the PR

[grepwood]

I've just hit this issue and I don't know what to do.
While I could inject credentials in a RFC 1738 3.1 compliant manner, I'm terrified they would leak into the output file which is unacceptable. We use different credentials between humans and machine users and using someone else's credentials is a big no-no.
Furthermore, my CI needs to download all the jars specified by the output file - which makes me wonder if this needs a separate method of credentials injection.

So far from what I've read there are a number of avenues:

1. Apply the patch from Support authentication to private maven repositories (Artifactory) #95 (comment) and build our own fork of the jar... :/
2. Use the hack from Too many maven_jar()s handled in parallel bazelbuild/bazel#5452 (comment) which I'm yet to find out if it's viable
3. See if I can hack anything much in line with the idea presented in Support authentication to private maven repositories (Artifactory) #95 (comment)
4. Move on to https://github.com/salesforce/bazel-maven-proxy/

That's a tough cookie to say the least.

Update:
RFC 1738 3.1 i.e. putting the credentials into the repository address yields no satisfactory result. All URLs are reported as unauthorized by bazel_deps.CoursierResolver. On the other hand cURL has no problem accessing those addresses.

[grepwood]

Wait a minute! Wasn't this solved by #293 ?

[johnynek]

Maybe? I merged the PR I think but actually we don't use that feature so I don't really know.

Sorry it isn't more clear in the docs or that I didn't remember.

[grepwood]

I found out that regardless of the version, bazel-deps will successfully download if:

• your sbt is configured with credentials and mirrors
• your Maven is configured with credentials and mirrors

This means these 3 files require adjustment:

1. ~/.m2/settings.xml - particularly mirrors and servers sections
2. ~/.sbt/repositories - needs to reflect the mirrors section from Maven settings
3. ~/.sbt/credentials.sbt - declare as many as you need, but it works more like a netrc so it's more on a per-hostname basis, not on a per-mirror basis like Maven

[grepwood]

I have made this PR with the hopes of fixing this issue: #387

bazel-deps.jar on its own supports credentials for sbt and maven in the usual suspected places. What this PR changes, is that now we can generate code designed for consumption by Bazel, that supports authentication in a broader scope than that described in #293