Use unix socket instead of opening up docker's port

[satoshi75nakamoto]

Use unix socket instead of opening up docker's port which is a giant security hole.

[satoshi75nakamoto]

My initial thoughts are that If we modify the internal HTTP GETs to tcp/4243 to some other port listening on the docker collector container and it acts like a proxy to provide the data requested to other instances of cluster insight with some sort of security mechanism and a white list of approved queries we should be okay.

[satoshi75nakamoto]

@EranGabber, @vasbala, @jackgr what do you think of the following modifications?

[jackgr]

I'm inclined to think there's an easier solution. If the container runs on every node, then the local instance on each node can use the Unix socket to communicate with its Docker daemon, and can expose the data it pulls through its own port. The instance on the master can then integrate the results from the local instances. No additional security would be required, since only collected configuration data is exposed by any instance.

[vasbala]

The ideal solution is to have the kubelet that is already present in every
node serve as a proxy for collecting Docker metadata. We've talked to
Brendan about this and he agreed this would be straightforward, but asked
us to send a pull request for it. Eran and I will discuss tomorrow and get
back to you - this is a priority for us.

On Monday, May 25, 2015, Jack Greenfield notifications@github.com wrote:

I'm inclined to think there's an easier solution. If the container runs on
every node, then the local instance on each node can use the Unix socket to
communicate with its Docker daemon, and can expose the data it pulls
through its own port. The instance on the master can then integrate the
results from the local instances. No additional security would be required,
since only collected configuration data is exposed by any instance.

—
Reply to this email directly or view it on GitHub
#62 (comment)
.

[vasbala]

Some clarifications on the nature of the problem reported by @preillyme, and the options to fix it:

The current implementation has 2 known problems: (a) poor usability: user has to modify with the Docker config file and restart the daemon for every minion node, and (b) sloppy security: 0.0.0.0/4243 means the master + all minions can access this port within the cluster. I would not characterize this as a "giant security hole" as @preillyme suggests, because this port is only accessible from within the cluster and not from outside.

We have 4 options to fix both problems:

1. Get the Kubernetes master to proxy Docker API requests through to the corresponding minion's kubelet which can then pull the Docker metadata locally and hand it back in the response. Pros: no changes to Docker daemon configs, no additional containers, no addition proxy servers on the minions. Cons: requires a (relatively small?) change to Kubernetes.
2. Start a Cluster Insight container instance on the master + every minion node. The master instance will communicate with the minion instances to assemble the context graph. The minion instances will use the local Unix socket to access the Docker daemon API. Pros: no changes to Docker daemon configs, minion instances can push (incremental) changes to the master instance for more efficient and scalable data collection. Cons: requires multiple Cluster Insight container instances running in the cluster, raising additional issues like dealing with their collective availability.
3. Run a proxy web serving container on each minion node that redirects port 4243 to the local Docker daemon's Unix socket. Pros and Cons are identical to (2). An additional Cons is we have one more piece of code to write, and one more Docker image (for the proxy) to maintain.

I have ordered the options by what I think is the cleanest design. If you want to order by how quickly it can be implemented the order would be 3, 2, 1. My personal choice would be option 2, because it is entirely under my team's control, but this may spill into next week to complete and test.

[EranGabber]

A fourth alternative is to enable HTTPS communication on the port, as explained in https://docs.docker.com/articles/https/. Search for "Protecting the Docker daemon Socket with HTTPS".

The plan is to generate the keys using the same setup script that we currently use to update the Docker configuration and restart the daemon in every node. The user (at least on GCP) will use the same setup script. Users on AWS will have to perform extra manual steps because I did not port my script to AWS yet.

[jackgr]

Thanks, Vas and Eran, for the investment you're making in helping to get
the Graph Tab into the release.

Option 2 looks the best to me, as proposed on issue #62
#62 (comment),
because:

• It's secure, since the cluster master and minion can communicate using
  SSH with the master's public key shared to the minions during deployment.
• It's fast. At this point, we are very close to missing the release
  train. We need a solution by Monday's triage meeting, or very shortly
  thereafter, in order to have any hope of making it. This option looks like
  it should be faster to implement than 3, if we don't try to make the minion
  instances more efficient at this point. The only code change should be
  aggregating the results from the minions in the master, and that should a
  small code change, if it's implemented brute force.

We should be able to secure high enough availability for this service,
since it isn't critical path for anything, using a replication controller.
We deploy the same number of pods as minions, and rely on the default
scheduling policy to spread them out as evenly as possible.

Jack Greenfield | Google

On Fri, May 29, 2015 at 6:37 AM, Eran Gabber notifications@github.com
wrote:

A fourth alternative is to enable HTTPS communication on the port, as
explained in https://docs.docker.com/articles/https/. Search for
"Protecting the Docker daemon Socket with HTTPS".

The plan is to generate the keys using the same setup script that we
currently use to update the Docker configuration and restart the daemon in
every node. The user (at least on GCP) will use the same setup script.
Users on AWS will have to perform extra manual steps because I did not port
my script to AWS yet.

—
Reply to this email directly or view it on GitHub
#62 (comment)
.

[EranGabber]

Implemented by #69.