hazelcast:3.9.2-os docker image cluster does not create a single cluster.

[mattyarbrough]

When using the official dotcms/hazelcast:3.9.2-os image in a Deployment/replicaset with more than one replica, the instances of hazelcast do not create a single cluster.

To reproduce:
This can be reproduced locally via docker-compose for simplicity or in a kubernetes cluster.

Via docker-compose:

1. Using the docker-compose.yml from here: https://github.com/dotCMS/docker/blob/master/referenceImplementations/03-full_stack/docker-compose.yml
2. Run docker-compose up -d --scale dotcms=0 --scale db=0 --scale elasticsearch=0 --scale elasticsearch-statis=0 --scale hazelcast=3
3. Then exec into the pod by running docker-compose exec hazelcast sh
4. From inside the pod run wget -O- hazelcast:5701/hazelcast/rest/cluster several times.
5. Notice the member size reported is always 1. Notice the member reported is always marked with "this".

Via kubernetes:
Given the attached file hazelcast-deployment.yml, which is a stripped down version of
https://github.com/brentgriffin/2018BootcampK8sDemo/blob/master/deployment.yaml with replicas set to 3. And a Kubernetes cluster:

1. Run kubectl apply -f hazelcast-deployment.yml to deploy hazelcast to kubernetes cluster
2. After deploy, find the pod names for hazelcast and pick one: kubectl get pods
3. Run kubectl exec -it {podname} sh
4. From inside the pod run wget -O- hazelcast-svc:5701/hazelcast/rest/cluster several times.
5. Notice the member size reported is always 1. Notice the member reported is always marked with "this".

[wezell]

@mattyarbrough I am not sure but dotCMS may act as the nexus for the Hazelcast servers to see each other. Does the HZ cluster work if you bring up a dotCMS into the mix?

[pmooneyarchitech]

@mattyarbrough @wezell I've tried bringing up dotCMS as well but no luck.
I've tested this with the modified docker-compose method from the OP:

1. Using the docker-compose.yml from here: https://github.com/dotCMS/docker/blob/master/referenceImplementations/03-full_stack/docker-compose.yml
2. Add license.zip to 03-full_stack folder
3. Run docker-compose up -d --scale hazelcast=3
4. Then exec into the pod by running docker-compose exec hazelcast sh
5. From inside the pod run wget -O- hazelcast:5701/hazelcast/rest/cluster several times.

I get the same results, each hazelcast instance reports a cluster size of 1 with only itself as the member.

[wezell]

@pmooneyarchitech -
Not sure why they are not connecting - Maybe try setting up a hazelcast cluster as described here and then attaching your dotCMS to it - I would try to use the same version as what we are running

https://github.com/hazelcast/hazelcast-code-samples/tree/master/hazelcast-integration/docker-compose