Dynamic tags applied like health checks.

[fidian]

In issue #867 I suggested an idea to make tags that depend on the result of scripts, just like health checks.

I run mongo in the cloud with multiple machines all spun up from the same image. On boot they will query for mongodb.service.consul and join the cluster. That all works flawlessly. In being a good Ops person I have a cron job that will kill random machines in my infrastructure at random times. It will eventually hit the mongodb master, the system will hiccup and a slave will be promoted automatically. Life is fantastic.

In comes Legacy Software that must connect directly to the master mongodb instance. I would like to have master.mongodb.service.consul resolve to the one IP of the master in the cluster.

Current solution (runs via cron on all machines):

1. Get my service definition through API
2. Check the status of the cluster. This determines if we should or should not have a tag.
3. Determine if the service definition's tag list needs to be updated.
4. If an update is required, POST data back to the API.

Ideal solution:

1. Set up my service definition with dynamic tags.
2. Write a script that returns the status of the cluster, with an exit code of 0 meaning to apply the tag.
3. Let consul update itself automatically.

Sample JSON (one static tag, one dynamic tag):

{
    "service": {
        "name": "mongodb",
        "tags": [
            "fault-tolerant",
            {
                "name": "master",
                "script": "/usr/local/bin/mongo-is-master.sh",
                "interval": "10s"
            }
        ],
        "address": "127.0.0.1",
        "port": 8000,
        "checks": [
            {
                "script": "/usr/local/bin/mongo-health-check.sh",
                "interval": "10s"
            }
        ]
    }
}

This sort of solution could apply to issue #155 and #867, and possibly other.

[ryanuber]

Interesting idea. I think the work-around you mentioned is a decent way of doing this, but I'm going to leave this open as a thought ticket for now. Thanks!

[Kosta-Github]

@fidian with respect to your statement: "On boot they will query for mongodb.service.consul and join the cluster."

Can you describe this a bit more, since I want to setup something similar for a redis cluster. Do you use some handcrafted script (e.g., via consul-tenplate or the REST API) for querying for mongodb.service.consul to get all registered nodes for that service or are you relying on the DNS mechanism for that? At least one problem with solely relying on the DNS mechanism is, that if the node registeres itself (e.g., with registrator) within the consul cluster before it does the DNS lookup for mongodb.service.consul it might get back its own IP address, which would not be helpful to join the cluster... :-)

[walrusVision]

This would useful for services like zookeeper which dynamically elects a leader node among themselves every time a node joins or leaves the cluster and the leader has the setting on so that it no longer accepts client connections. Having dynamic tags like this via check would make so I could query consul for the non-leader nodes and not have a client trying to connect to the leader at all.

[fidian]

@Kosta-Github asked how I manage to auto cluster my mongo instances.

1. Consul is hooked up through dnsmasq.
2. Consul is started before mongo.
3. The health check fails unless mongo reports success and mongo is part of a cluster. This second part is vital - the health check fails until mongo is in a cluster.
4. The init script for mongo queries DNS for other members in the cluster. This will only report mongo instances that are already in a replica set.

• If IPs are found, become a slave and connect to the IP that we found.
• With no IPs, configure as a master and enable the replica set, which then makes the health check pass.

The only snag is that I must start one instance of mongo initially so it will bootstrap the replica set. Once it is running I am able to add and remove instances to my replica set.

[Kosta-Github]

@fidian thanks for the explanation; just one more question: how does your dnsmasq config look like? :-)

[fidian]

@Kosta-Github it looks like the following. I'd also answer questions off this issue. Feel free to email me directly at fidian@rumkin.com so we don't continue to pollute this thread.

server=/consul./127.0.0.1#8600

[igoratencompass]

+1 for this feature request

[eloycoto]

+1

[hugochinchilla]

+1

[xakraz]

+1

[jh409]

+1

[adbourne]

+1

[memelet]

This would be very very nice. There are all kinds of things for which clients need to connect the mast expliclty. A dynamic tag would be so elegant. So much better then a bunch of add scripts to tweak tags.

[danielbenzvi]

+1