Feature Request - build to docker without docker binary

[dimovelev]

Description of the issue:
The docker build requires docker CLI to load the image built by jib to the docker daemon. It would be great if you could remove this dependency by implementing a simple HTTP post of the tarball to a configurable target. Example with curl:

curl --unix-socket /var/run/docker.sock --header "Content-Type: application/x-tar" --data-binary "@target/jib-image.tar" http://localhost/images/load

So far the communication flavours that I am aware of are:

• HTTP via unix socket (example above)
• HTTP via TCP
• HTTPS via TCP
• HTTP via windows named pipe

Expected behavior:
Being able to build to docker without a docker binary installed.

[mpeddada1]

Hi @dimovelev, thank you for the feature request! Here are some related discussions: #2130, #1997.
Please let us know if you would be interested in exploring further or contributing.

[dimovelev]

@mpeddada1 i am not sure what the result of the discussion is - do you want to use docker-java library or not?

Assuming that you do not want to use it, we will have a bit more work to do:

• unix socket - we could use https://github.com/kohlschutter/junixsocket combined with the current apache httpclient (4.5) by overriding the socket factory. Starting from java 16 we could use the native support for this (https://openjdk.java.net/jeps/380).
• http / https - natively with the apache httpclient
• windows named pipe - in docker-java there is an implementation (a single class called NamedPipeSocket) which can be used as inspiration (no idea what whether we are allowed to copy and adapt the java class from there - both projects use the Apache 2.0 license). As with unix socket, we override the socket factory for the httpclient.

parsing the request / response with jackson should be fine (we have jackson already as dependency).

configuration-wise we could try to auto-configure depending on OS and socket/named pipe availability or require explicit configuration.

[elefeint]

@dimovelev We try to avoid third-party dependencies as much as possible in Jib, so the native solution you described is preferable. NamedPipeSocket does not seem to be doing anything extremely complicated, so it should not require copying.

To clarify, we would accept a pull request for enabling non-docker scenarios, but this isn't on the team's roadmap to implement.

[dimovelev]

I ended up writing a small maven / gradle plugin and publishing it to maven central - https://github.com/dimovelev/load-to-docker . It does what I needed - perform HTTP POST via named pipe on windows and unix domain socket on linux. I wanted to integrate it into jib but it seemed like a lot more work and I do not mind doing this in two steps - storing tarball with jib and then using the other plugin to load to docker. If at some point you decide that you want to integrate such a feature you could use it as a kind of proof-of-concept.

I will leave it up to you to decide whether you want to close this issue or not.

[elefeint]

Thanks! We'll leave the issue open, so others can comment with their requirements.

[bsideup]

FTR docker-java provides a very lightweight transport abstraction that already supports tcp, unix sockets and npipes. There is also a zerodep transport that only depends on JNA & Slf4j:

so, in case you are not ready to use DockerClient from docker-java, you can still reuse some major parts of the library (and all this prior knowledge of how to talk to Docker properly, as there are a few advanced things like hijacking :))

[elefeint]

@chanseokoh What do you think about taking on docker-java-transport-zerodep as a dependency?

[nakamorichi]

I'm interested in support for nerdctl in containerd mode when using Rancher Desktop. I wonder if there are any plans for support? Unfortunately I don't know enough about Rancher Desktop/nerdctl/containerd details to propose a concrete solution.

[elefeint]

@Kitanotori We have no current plans, but keeping this issue open to evaluate demand.
#3577 was asking about the same setup that you have. All these issues are ultimately blocked on Jib requiring Docker CLI.

#2130 with ability to plug in different docker communication mechanisms is likely the best architectural approach. We've said in the past that DockerClient is an implementation detail that's not meant for reuse, but there is no reason it could not become part of the public API in the future.

[eddumelendez]

Hi,

I will be taking care of this implementation. I would like to discuss the following approach

1. Use com.github.docker-java:docker-java-transport-zerodep to deal with the communication and do not reinvent the wheel
2. turn DockerClient into an interface with 4 method signatures mode, save, load and inspect and create a DefaultDockerClient which is mostly the existing DockerClient but implementing it
3. add a new implementation for DockerClient call SocketDockerClient???
4. Register DefaultDockerClient and SocketDockerClient via SPI
5. Add a new argument mode for gradle, maven where options are default or socket. support for jib-cli can be added later.

This option allows to ship jib with both supports. It also enables to add new dependency to the plugin in maven and gradle and add the custom implementation via SPI. I was also looking at the plugin framework but doesn't seem to resolve this part and adding support for it can be a big one.

Looking forward to hear from you.

[nakamorichi]

@eddumelendez Does that library work without any Docker component in host or target?

[dimovelev]

@eddumelendez great news!

Isn't the current implementation cli? What is the difference between default and cli?

In a default mode it would be great if the plugin could autodetect what can be used - maybe check if the socket client can be used (e.g. named pipe exists, unix socket exists) and as a fallback use the docker cli.

It might be necessary to add one more optional argument e.g. called url to be able to help the plugin find the right pipe, socket, host-port to talk to.

An alternative suggestion for the new client's name could be HttpDockerClient (and mode http).

[elefeint]

@eddumelendez Thank you very much! Your plan sounds good to me. Thank you for considering the dependencies as optional -- adding them as plugin dependencies is ideal.

1. Agreed. While we try not to take on new dependencies, in this case it's merited. And especially if it's an optional plugin dependency, it should not cause issues (and if it does, we can always reinvent the wheel later).
2. Sounds good. Add DockerClient interface to com.google.cloud.ttools.jib.api package. Instead of mode(), consider naming it supported() or some such to allow for inferring whether an implementation is supported from environment, and not just direct user directive. This will cover @dimovelev 's suggestions. Not sure what parameters make sense for supported() -- perhaps a simple map? Either way, you'll have to figure out how to best pass the mode from CommonCliOptions available in Containerizers.create() down to Containerizer.to().
3. I don't have an opinion about the new implementation name; depends on what you put in it :)
4. sounds good
5. Default and CLI will start out the same, right? Might as well only allow "socket" and "cli", and document that "cli" is default.

[bsideup]

@eddumelendez I would also propose to do multiple steps, where the first step is to make Jib's DockerClient part of the public API, to enable "bring your own client" 👍

[eddumelendez]

Does that library work without any Docker component in host or target?

right, no docker installation is required. It talks directly to the daemon that it is configured.

Isn't the current implementation cli? What is the difference between default and cli?

yes, by cli I meant jib cli :D

Sounds good. Add DockerClient interface to com.google.cloud.ttools.jib.api package. Instead of mode(), consider naming it supported() or some such to allow for inferring whether an implementation is supported from environment, and not just direct user directive. This will cover @dimovelev 's suggestions. Not sure what parameters make sense for supported() -- perhaps a simple map? Either way, you'll have to figure out how to best pass the mode from CommonCliOptions available in Containerizers.create() down to Containerizer.to().

It makes sense to me. Thanks for the feedback! I just wanted to make it explicitly for the user instead of make it work under the hood with not much aware but given the parameters that are needed we can implement what was suggested.

Default and CLI will start out the same, right? Might as well only allow "socket" and "cli", and document that "cli" is default.

yes

I would also propose to do multiple steps, where the first step is to make Jib's DockerClient part of the public API, to enable "bring your own client" 👍

this sounds a great approach

[nakamorichi]

right, no docker installation is required. It talks directly to the daemon that it is configured.

Can it talk directly to containerd/buildkitd so that it could work with Rancher Desktop in containerd mode (might require changes at containerd first, though; see here)?

Would it be possible to combine effort with containerd/nerdctl and Rancher Desktop teams to make jib work fully with RD without any Docker component?

[bademux]

@eddumelendez great work!
So can we now upload docker tar image without docker CLI or it is just preparation?