Multiple private IPs - set advertise address

[neuroserve]

I don't know, whether this is a variant of issue #226 but in environments with containers having multiple nics with rfc1918 addresses, consul-agent cannot decide to which of the nics it should bind in order to talk to the consul-master/cluster:

"Error starting agent: Failed to get advertise address: Multiple private IPs found. Please configure one. [/usr/local/bin/consul agent -data-dir=/data -config-dir=/config -rejoin -retry-join consul.cluster -retry-max 10 -retry-interval 10s] exited: exit status 1"

Is there an easy way to determine the respective ip address and configure consul-agent accordingly?

[tgross]

@neuroserve we discovered this same problem in autopilotpattern/consul#17 and we provided some hooks in #145 #171 to fix that very problem.

We expose an environment variable CONTAINERPILOT_<SERVICE>_IP with the IP of each service. So you can use that environment variable as an argument to pass into the Consul agent co-process.

[neuroserve]

Hm. I understand, that I have to adapt the start of the consul-agent in my containers. Do I do that in the respective containerpilot.json file? Or in my docker-compose.yml? Or in my Dockerfile? Do you have an example with consul (instead of etcd)?

[tgross]

It's slightly tricky because you can't use the value inside the containerpilot.json file -- the value doesn't exist at the time we do the config loading because we haven't loaded the config yet! (We want to make this possible in a future ContainerPilot version.)

So what you can do is to have the preStart update the Consul configuration file and use that. There's an example of this in the autopilotpattern/vault blueprint. The ContainerPilot preStart calls a bash script which rewrites the Consul configuration in consul.json with plain ol' sed.

[tgross]

Looks like this was answered. Feel free to re-open if you need me to expand on it.

[lucj]

Hi @tgross, I'm following the procedure you detailed above. I'm using sed in the preStart to replace the SERVICE_IP placeholder within the consul.json file.
Within the container there is no env variable with the service's IP though. I was expected to get an env var like CONTAINERPILOT_API_IP (API is the name of the current service).
I'm not sure what I'm missing here.

[misterbisson]

@lucj can you confirm that you're using a recent version of ContainerPilot, and that you're running the scripts via ContainerPilot?

If you confirm both of the above and you find you're definitely not getting the expected env vars (please print out the env vars from your script or run env as a ContainerPilot job to confirm what vars you do have), that would be a different bug that deserves a new issue.

[lucj]

@misterbisson, all the services are run through ContainerPilot v3.3.0.
Below are examples for one of the service (api)

/app # ps aux
PID   USER     TIME   COMMAND
    1 root       0:00 containerpilot
    8 root       0:00 containerpilot
   14 root       0:00 {manage.sh} /bin/sh /app/manage.sh prestart
  142 root       0:00 sh
  316 root       0:00 /usr/local/bin/consul agent -data-dir=/data -config-dir=/config -log-level=err -rejoin -retry-join consul -retry-max 1
  336 root       0:00 sleep 5
  337 root       0:00 ps aux

/app # containerpilot --version
Version: 3.3.0
GitHash: e7fda21

Below is the result of the env command in the container of the api service:

~ $ docker exec -ti 64ed6cc21f39 sh
/app # env
HOSTNAME=64ed6cc21f39
SHLVL=1
HOME=/root
CONTAINERPILOT_VER=3.3.0
VERSION=v6.9.4
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
NPM_VERSION=3
LAST_UPDATED=20170515T152500
PWD=/app
CONTAINERPILOT=/etc/containerpilot.json5

In the case ContainerPilot set CONTAINERPILOT_SERVICE_IP, which IP is selected ? The IP of the container on the Overlay network it belongs to (for example within subnet 10.0.1.0/24) or the IP on the Docker0 bridge ?

[lucj]

My bad, when I check the correct env (the one available within the process running the whole thing), I have an IP for the services:
for instance CONTAINERPILOT_API_IP=10.0.1.19
Sorry for this.