_{HOSTNAME_COMMAND} does not work with KAFKA_JMX_OPTS

[mdedetrich]

So I currently I have a usecase where I need to set up a kafla cluster with multiple brokers (i.e. at least 2) to diagnose a performance issue with reassigning partitions. Since I am setting up multiple brokers I need to use the HOSTNAME_COMMAND to manually set the hostname for each broker.

This is the initial docker-compose.yaml that I used

version: '2'

services:
  zookeeper:
    image: wurstmeister/zookeeper:latest
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - "9092"
    depends_on:
      - zookeeper
    environment:
      # See https://stackoverflow.com/a/42729826
      HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
      KAFKA_DELETE_TOPIC_ENABLE: 'true'
      KAFKA_HEAP_OPTS: "-Xmx2G -Xms2G"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Now I need to actually set up a JMX port so that I can connect to the the broker instances using visualvm so I adjusted the docker-compose.yaml to look like the following

version: '2'

services:
  zookeeper:
    image: wurstmeister/zookeeper:latest
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - "9092"
      - "1099"
    depends_on:
      - zookeeper
    environment:
      # See https://stackoverflow.com/a/42729826
      HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
      KAFKA_DELETE_TOPIC_ENABLE: 'true'
      KAFKA_HEAP_OPTS: "-Xmx2G -Xms2G"
      KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=_{HOSTNAME_COMMAND} -Dcom.sun.management.jmxremote.rmi.port=1099"
      JMX_PORT: 1099
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

The main issue here is using the _{HOSTNAME_COMMAND} in -Djava.rmi.server.hostname for the KAFKA_JMX_OPTS. With these changes and using docker-compose up -d --scale kafka=2 there seems to be some interpolation error when kafka tries to start, I get the following error

/usr/bin/start-kafka.sh: line 59: export: `-Dcom.sun.management.jmxremote.authenticate=false': not a valid identifier
/usr/bin/start-kafka.sh: line 59: export: `-Dcom.sun.management.jmxremote.ssl=false': not a valid identifier
/usr/bin/start-kafka.sh: line 59: export: `-Djava.rmi.server.hostname=192.168.160.1': not a valid identifier
/usr/bin/start-kafka.sh: line 59: export: `-Dcom.sun.management.jmxremote.rmi.port=1099': not a valid identifier

so the actual substitution for _{HOSTNAME_COMMAND} seems to work since 192.168.160.1 is in the error but it appears that the substitution method is somehow breaking the -D java settings that get sent to kafka. Note that if I manually put in 192.168.160.1 into the docker-compose, i.e.

version: '2'

services:
  zookeeper:
    image: wurstmeister/zookeeper:latest
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - "9092"
      - "1099"
    depends_on:
      - zookeeper
    environment:
      # See https://stackoverflow.com/a/42729826
      HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
      KAFKA_DELETE_TOPIC_ENABLE: 'true'
      KAFKA_HEAP_OPTS: "-Xmx2G -Xms2G"
      KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=192.168.160.1 -Dcom.sun.management.jmxremote.rmi.port=1099"
      JMX_PORT: 1099
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

The cluster works fine but I then have issues to the host using visualvm, i.e. after a run of docker-compose up -d --scale kafka=2 and then doing docker ps -a I get

a5c10f9bd8bb   wurstmeister/kafka:latest       "start-kafka.sh"         25 seconds ago   Up 24 seconds   0.0.0.0:49222->1099/tcp, 0.0.0.0:49221->9092/tcp     docker-compose_kafka_1
beec603b54cc   wurstmeister/kafka:latest       "start-kafka.sh"         25 seconds ago   Up 24 seconds   0.0.0.0:49220->1099/tcp, 0.0.0.0:49219->9092/tcp     docker-compose_kafka_2
d414d7e5d557   wurstmeister/zookeeper:latest   "/bin/sh -c '/usr/sb…"   26 seconds ago   Up 25 seconds   22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   docker-compose_zookeeper_1

The first brokers JXM host should be localhost:49220 however in visualvm I don't seem to be able to connect to this