Usage with testcontainers and jenkins in docker

[ash-sykes-acn]

Hi All - bit of a niche question - I'm looking to use testcontainers to create the Redis cluster. Locally I can configure my tests to talk to the cluster and everything works fine. The issue is that when the same tests are run on a build slave, the slave is itself a container. This means we need to use the docker wormhole pattern to access the container. Testcontainers provides a method (getHost) that correctly resolves the IP of the container running the cluster. The issue is that the cluster registers all nodes under 127.0.0.1 so when the test code attempts to make use of the cluster, the cluster tries to access nodes on 127.0.0.1 which is obviously wrong as it needs to communicate using the hostIp. I know this isn't specific to this image but don't suppose anyone has come across this before and knows how to resolve it?

[Grokzen]

@ash-sykes-acn I dont know if it really solves the issue, but in 7.0-RC1 they changed this * Cluster: Support for hostnames, instead of IP addresses only (#9530) this was renamed to * cluster-announce-hostnameand cluster-preferred-endpoint-type (#9530) in the final 7.0 release, that might be a possible solution. However this is still a problem because of network routing because you might still have an issue where you nodes want the inside ip but your client sometime needs the local ip but sometime the external ip.You can technically mitigate this by manually manage /etc/hosts file in the different locations to sort it out i guess. I would try to build with a 7.0 release and see if you have any further options there or not. I know this is a common issue when running all nodes on the same machine, technically it is not recommended to run multiple redis nodes per VM or docker container but they should be bound to a single node like if you run it within docker-compose and get 3 different containers talking to each other and they are capable of being reached by a unique dns name and a unique IP address.
What i always do in reality when i run a cluster is to even in a container case to bind the containers network to the host network and just sort out the cluster shards with different ports and the host machine IP is reachable across the entire network as expected.

[dcole-gsn]

Hi Grokzen,
Was wondering if you could provide a little more context to your final point

What i always do in reality when i run a cluster is to even in a container case to bind the containers network to the host network and just sort out the cluster shards with different ports and the host machine IP is reachable across the entire network as expected.

Like @ash-sykes-acn I'm trying to run your redis-cluster image within testcontainers. I'm doing the following, does this look like what your suggesting?

GenericContainer redisContainer;
static final Set<Integer> clusterPorts = new HashSet<>(Arrays.asList(7000, 7001, 7002, 7003, 7004, 7005));

redisContainer = new GenericContainer<>(
                DockerImageName.parse("grokzen/redis-cluster:latest"))
                .withExposedPorts(clusterPorts.toArray(new Integer[0]))
                .withAccessToHost(true)
                .waitingFor(Wait.forLogMessage(".*Ready to accept connections.*\\n", 1));

redisContainer.start();

final String hostIP = "123.234.123.234";   // Host local IP or do you mean localhost/127.0.0.1?
final Set<HostAndPort> hostPortSet = new HashSet();
final Integer[] portArray = clusterPorts.toArray(new Integer[0]);
for(Integer port : portArray) {
	hostPortSet.add(new HostAndPort(hostIP, redisContainer.getMappedPort(port)));
}

final String testKey = "test";
final JedisCluster jedisCluster = new JedisCluster(hostPortSet);
final Map<String, ConnectionPool> clusterNodes = jedisCluster.getClusterNodes();
assertTrue(clusterNodes.size() > 0);
try {
	final String result = jedisCluster.set(testKey, "value");    // <------- Throws "Cluster retry deadline exceeded" exception
	assertTrue(Objects.equals(result, "OK"));
} catch(Exception ex) {
	fail(ex.getLocalizedMessage());
}

Reason I ask, when I do the set, it throws an exception

Cluster retry deadline exceeded.
java.lang.AssertionError: Cluster retry deadline exceeded.
at org.junit.Assert.fail(Assert.java:89)

Perhaps I misunderstood what you were saying. Do you have any suggestions I could try to resolve the issue?
Kind regards.

[Grokzen]

@dcole-gsn When you run this docker image by default w/o any docker flags to the run command, it runs with an internal network and all redis nodes will either use 127.0.0.1 or the docker specific IP as the IP it wants to publish to clients connecting into it. When i mean i run on a host network with docker is that the IP is the same IP as i get outside on my laptop in my case so that the IP addresses that Redis picks up and publishes to the clients will be like i ran all redis nodes directly on my laptop, even if they really belongs or lives inside a docker container.

I don't know Jedis enough to be able to help you. Usually what i recommend in general is that from the location you are running your code, install redis-cli and figure out how to connect it to your redis cluster nodes, what IP:port etc. You also usually connect in and run CLUSTER INFO and CLUSTER SLOTS if i remember correct to get the current cluster state that your code client will use. Then use those settings within your client and connecting should work out totally fine.

[dcole-gsn]

Hi Grokzen,
Thanks for your response. I tried what you suggested and I'm able to connect using Jedis directly to the cluster running within docker desktop.
So I'm guessing it's an issue with the port mapping for testcontainer, I'm just not sure how to expose the correct ports to jedis from testcontainer.
I'll do more digging, thanks for your time.
Kind regards.

[dcole-gsn]

Hi,
I managed to get this working.
To anyone else wanting to accomplish this, I ended up implementing my own ConnectionProvider, extending ClusterConnectionProvider. The new ConnectionProvider maps the internal redis node ips and ports to localhost and the exposed ports.

Then I implemented the following methods

public Connection getConnectionFromSlot(int slot)
public Connection getConnection()

to return the necessary Connection from the available localhost:exposedPort connection pools.

eg:

@Override
  public Connection getConnectionFromSlot(int slot) {
    return connectionPoolMap.get(slotHostMap.get(slot)).getResource();
  }

and finally create the JedisCluster like

public JedisCluster getJedisCluster() {
        final Set<HostAndPort> hostPortSet = new HashSet();
        final Integer[] portArray = clusterPorts.toArray(new Integer[0]);
        final Map<Integer, Integer> portMapping = new HashMap();
        for(Integer port : portArray) {
            final Integer containerPort = redisContainer.getMappedPort(port);
            hostPortSet.add(new HostAndPort(redisContainer.getHost(),  containerPort));
            portMapping.put(port, containerPort);
        }

        final TestContainerClusterConnectionProvider clusterConnectionProvider = new TestContainerClusterConnectionProvider(hostPortSet, portMapping, DefaultJedisClientConfig.builder().build());
        return new JedisCluster(clusterConnectionProvider, 5, Duration.ofMillis(100));
    }

Not sure if this helps anyone, but hopefully it does.
Grokzen once again, thanks for your help.
Kind regards.

[Grokzen]

@dcole-gsn Ahh, i see you have to implement something that i implemented inside redis-py-cluster, we had this issue many years ago where people wanted to be able to remap what redis gave back to something custom that redis not always could pick up properly. That rebind method you implemented basically does the same thing.

The simplest solution is really to make redis DNS aware instead of only IP aware becuase it is much simpler to sort out your IP:s and routing on a DNS level in different places to avoid having to do this rebind hacks. When running managed redis clusters in the cloud, you get this kinda for free there as they provide dns names that is managed for each node and stuff

[ash-sykes-acn]

If this helps anyone here is some, non productionised test code, for doing this with Lettuce (not Spring Data)

        Integer port1 = EnvUtil.getInteger("7000");
        if (port1 == null) {
            RedisConnector.LOGGER.info("Redis port doesn't exist locally. Reading from AWS parameter store...");
            port1 = Integer.parseInt(RedisConnector.paramStore.getRequiredString(portName));
        }

        Integer port2 = EnvUtil.getInteger("7001");
        if (port2 == null) {
            RedisConnector.LOGGER.info("Redis port doesn't exist locally. Reading from AWS parameter store...");
            port2 = Integer.parseInt(RedisConnector.paramStore.getRequiredString(portName));
        }

        Integer port3 = EnvUtil.getInteger("7002");
        if (port3 == null) {
            RedisConnector.LOGGER.info("Redis port doesn't exist locally. Reading from AWS parameter store...");
            port3 = Integer.parseInt(RedisConnector.paramStore.getRequiredString(portName));
        }

        final ClientResources clientResources = DefaultClientResources.builder().build();

        final List<RedisURI> redisUris = Stream.of(port1, port2, port3)
                .map(p -> RedisURI.create(hostname, p))
                .collect(Collectors.toList());

        final RedisClusterClient client = RedisClusterClient.create(clientResources, redisUris);

        // final RedisURI uri = RedisURI.create(host, port);
        // final RedisClusterClient client = RedisClusterClient.create(uri);

        final ClusterTopologyRefreshOptions options = ClusterTopologyRefreshOptions.builder()
                .dynamicRefreshSources(false)
                .build();

        RedisClusterNode node1 = new RedisClusterNode();
        node1.setUri(RedisURI.create(hostname, port1));
        node1.setSlots(IntStream.range(1, 5461).boxed().collect(Collectors.toList()));
        RedisClusterNode node2 = new RedisClusterNode();
        node2.setUri(RedisURI.create(hostname, port2));
        node2.setSlots(IntStream.range(5462, 10923).boxed().collect(Collectors.toList()));
        RedisClusterNode node3 = new RedisClusterNode();
        node3.setUri(RedisURI.create(hostname, port3));
        node3.setSlots(IntStream.range(10924, 16384).boxed().collect(Collectors.toList()));
        Partitions partitions = new Partitions();
        partitions.add(node1);
        partitions.add(node2);
        partitions.add(node3);

        client.setPartitions(partitions);
        client.setOptions(ClusterClientOptions.builder()
                .pingBeforeActivateConnection(true)
                .topologyRefreshOptions(options)
                .build());
        return client.connect();

[dkthezero]

Here is my solution for the test container to run with dynamic ports, use Redisson on Kotlin and tested with tag latest:

1. Create Container

class RedisClusterContainer<SelfT : RedisClusterContainer<SelfT>> @JvmOverloads constructor(
  imageName: String = "grokzen/redis-cluster:latest"
) : GenericContainer<SelfT>(imageName) {
  companion object {
    val ports = listOf(
      HttpTestUtils.findRandomOpenPort(),
      HttpTestUtils.findRandomOpenPort(),
      HttpTestUtils.findRandomOpenPort(),
      HttpTestUtils.findRandomOpenPort(),
      HttpTestUtils.findRandomOpenPort(),
      HttpTestUtils.findRandomOpenPort(),
    ).toTypedArray()
    const val local = "0.0.0.0"
    val environments = mutableMapOf(
      "INITIAL_PORT" to ports[0].toString(),
      "MASTERS" to "3",
      "SLAVES_PER_MASTER" to "1",
      "SENTINEL" to "false",
      "REDIS_CLUSTER_IP" to local,
      "IP" to local,
      "BIND_ADDRESS" to local
    )
  }


  override fun configure() {
    startupAttempts = 3
    withExposedPorts(*ports)
    withEnv(environments)
    waitingFor(LogMessageWaitStrategy().withRegEx("(?s).*==> /var/log/supervisor/redis.*$"))
  }
}

2. Create a NAT Config Class

class ClusterContainerNatMapper(
  private val host: String,
  private val mappedPort: Map<Int, Int>,
) : NatMapper {
  private val mappedAddress = mutableMapOf<Int, RedisURI>()

  override fun map(uri: RedisURI): RedisURI {
    val mappedPort = mappedPort[uri.port]

    return if (mappedPort == null) {
      uri
    } else {
      val address = mappedAddress[uri.port]

      if (address != null) {
        address
      } else {
        val mappedUri = RedisURI("redis://${host}:$mappedPort")
        mappedAddress.putIfAbsent(mappedPort, mappedUri)
        mappedUri
      }
    }
  }
}

3. Get mapped ports

val mappedPorts = RedisClusterContainer.ports.associateWith { port ->
        val mappedPort = redisClusterContainer?.getMappedPort(port)
        mappedPort ?: port
      }

  val addresses = mappedPorts.values.map { "redis://${host}:${it}" }.toList()

4. Setup config

  val clusterConfig = config
          .setCodec(JsonJacksonCodec())
          .useClusterServers()

    clusterConfig.nodeAddresses = address
    clusterConfig.natMapper = ClusterContainerNatMapper(host, mappedPorts)