allow containers to connect to the host machine

[n1ru4l]

In one of my integration tests, I am starting a webserver (via jest) and check whether a webhook on that webserver is called by a service in a docker container.

There are several options for this: https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mach

It would be awesome if this could be easily achieved with this library!

Edit:

Some more thoughts:

On platforms like AWS Codebuild, the code is already built inside a container. For starting additional containers the docker socket is injected into the container. That means if you start a new container and want it to be able to connect to the main "test" container you will have to set up a network between those containers.

[n1ru4l]

First Idea regarding tackling this issue:

Need to find out whether dockest is run inside a docker container or not.

In case it is run inside a container we have to create a network for linking all runner containers to the dockest container.

# create network
docker network create --driver bridge dockest_bridge_network
# add container that runs dockest to network
docker network connect dockest_bridge_network "$HOSTNAME"
# add remaining runner containers to the network
docker network connect dockest_bridge_network service-foobars
docker network connect dockest_bridge_network service-foobars
docker network connect dockest_bridge_network service-abcdef

I tested this locally and it seems to work fine.

The next problem is that connection check against ports must be done against the containerId:port instead of localhost:port. Docker uses the containerId as a hostname.

A quick solution would be to allow the following (or at least fallback to do this instead of using localhost as the default host in case dockest inside a container is detected):

const runner = new runners.GeneralPurposeRunner({
  service: "website",
  build: "./app",
  ports: {
    "9000": "9000"
  },
  responsivenessTimeout: 30 * 1000,
  connectionTimeout: 30 * 1000,
  // return containerId as host
  host: containerId => containerId
});

I hacked this into my testing fork (n1ru4l@de5bbfe)

It would then also be helpful to add runtime helper that allows resolving the hostname of a given service at runtime.

test("asdadasd", async () => {
  // throws in case service does not exist/ is not up
  const hostname = await getServiceHostname("website") 
  fetch(hostname)  // ... do sth with the hostname
})

@erikengervall Please let me know what you think.

[erikengervall]

Need to find out whether dockest is run inside a docker container or not.

Initially this responsibility could be placed on the user (i.e. via configuration) until a reliable solution is found.

In case it is run inside a container we have to create a network for linking all runner containers to the dockest container.

Hm, is it safe to assume that all projects wants all services to be linked to one another? The default mode could definitely hook everything up to each other and down the line an option could be introduced where the networking can be managed more granularly.

It would then also be helpful to add runtime helper that allows resolving the hostname of a given service at runtime.

Yeah definitely. If it's possible to globalize this type of information I'd be thrilled!

[n1ru4l]

Initially this responsibility could be placed on the user (i.e. via configuration) until a reliable solution is found.

is-docker seems like a widely used solution (almost 700,000 weekly downloads). Of course, we could still allow overwriting this (dunno about other peoples setup).

Hm, is it safe to assume that all projects wants all services to be linked to one another? The default mode could definitely hook everything up to each other and down the line an option could be introduced where the networking can be managed more granularly.

We could either go that route or generate a network for each pair (dockest container and service container). I agree for keeping it simple and add a more granular configuration option in the future.

Yeah definitely. If it's possible to globalize this type of information I'd be thrilled!

I will do some research!

[erikengervall]

is-docker seems like a widely used solution (almost 700,000 weekly downloads). Of course, we could still allow overwriting this (dunno about other peoples setup).

Looks like a solid library, could definitely use it in the first iteration of this feature.

I will do some research!

Awesome, thanks!

[n1ru4l]

It would then also be helpful to add runtime helper that allows resolving the hostname of a given service at runtime.

Yeah definitely. If it's possible to globalize this type of information I'd be thrilled!

There is no simple way of doing this. jestjs/jest#7184

We could, however, export a module from "dockest" that could be used for doing such stuff. It could be an abstraction above the docker and/or docker-compose cli for retrieving info such as the hostname etc.

example api:

import { getServiceHost } from "dockest/jest"

test("my test", async () => {
  const host = await getServiceHost("website")
})

The remaining problem would be the following: how does the dockest/jest know the used services? It would need to somehow first figure out which docker-compose.dockest-generated.yml file should be used and then parse it.

---

Alternative approach:

Give the containers that you connect to the container an alias that is equivalent to the one in the docker-compose.dockest-generated.yml file (= the service name defined in the runner config. That way we do not need a jest helper library (for now). Users just have to follow the convention of using the docker-compose service name as the hostname as they would actually also do in your application code (assuming they use docker-swarm or docker-compose in production). So this might be the better way (at least for now)?

import Dockest, { runners, logLevel } from "dockest";

const runner = new runners.GeneralPurposeRunner({
  service: "website",
  build: "./app",
  ports: {
    "9000": "9000"
  },
  responsivenessTimeout: 30 * 1000,
  connectionTimeout: 30 * 1000,
  // return continerId as host
  host: containerId => containerId
});

const dockest = new Dockest({
  opts: {
    logLevel: logLevel.DEBUG
  }
});

dockest.attachRunners([runner]);

dockest.run();

docker network connect dockest_bridge_network --alias website service-host

---

The ports config option is not relevant anymore as no port is bound to the dockest container. All container ports are accessible in the network. This, however, would break some of the current readiness checks.

[n1ru4l]

I got a first docker-in-docker solution working over here: n1ru4l#1. Will create a pull request over here once my other ones are merged.

The dockest in docker approach is working fine!

However, there are some complications with connecting to the host machine. On Mac OSX and windows, you can reach the docker host trough the hostname host.docker.internal. On Linux, there is no official way (docker/for-linux#264).

However, there are various hacks available, the easiest seemed to be the following:

check whether process.env is linux and manually add the hostname to /etc/hosts

if (process.platform === 'linux' && !runner.isBridgeNetworkMode) {
    const command = ` \
      docker exec ${runner.containerId} \
        /bin/sh -c "ip -4 route list match 0/0 | awk '{print \\$3\\" host.docker.internal\\"}' >> /etc/hosts"
    `
    await execa(command)
  }

The next issue I have is that ideally, I want to run yarn test without having to write some custom logic for checking whether I have to connect to host.docker.internal (dockest on host machine mode) or a container id (dockest inside docker container mode). Which is only solvable by providing test runtime helpers (n1ru4l@fdce244#diff-de61c684d5eed40fdbba538134d5258f).

We can share the docker-compose config with the test specs by setting them to process.env as a string. jest.runCLI will then forward the environment variable to the workers that run the tests.