Newt dials an unreachable IP when the target container has several networks shared with Newt

[GeoffreyCoulaud]

Describe the Bug

Issue

When a container is attached to more than one Docker network that Newt is also attached to, Newt resolves the destination hostname via Docker DNS and dials one of the returned A records without checking whether it is usable.

If it picks a network with enable_icc=false, every request fails with dial tcp <ip>:<port>: i/o timeout and the client gets a 502 after ~30s, even though a valid IP exists on another shared network.
DOCKER_ENFORCE_NETWORK_VALIDATION=true does not prevent this, Newt still targets the unusable IP.

On my server, I have multiple internal docker bridge networks and an internet (but no ICC) bridge netwok.
This is how I encountered the issue.

Workaround

My workaround for this has been to use unique container aliases specific to the network I want Newt and the container to use:

    networks:
      bridge-newt:
        aliases: [broken-service.newt]
# ....
      - pangolin.private-resources.broken-service.destination=broken-service.newt

Suggestion

I'd suggest adding an explicit docker network to use, like Traefik does with its labels, something like pangolin.private-resources.<id>.docker-network.

Environment

• OS Type & Version: OpenMediaVault 7.7.24-7
• Pangolin Version: 1.22.0
• Edition (Community or Enterprise): CE
• Gerbil Version: 1.5.0
• Traefik Version: 3.7
• Newt Version: 1.16.0
• Client Version: 0.16.0 (Linux CLI)

To Reproduce

I'm assuming you have a Pangolin instance running.

Example docker compose:

name: newt-multinet-repro

networks:
  bridge-newt:
    driver: bridge
  bridge-noicc:
    driver: bridge
    internal: true
    driver_opts:
      com.docker.network.bridge.enable_icc: "false"
  bridge-something:
    driver: bridge
    internal: true

services:
  newt:
    image: fosrl/newt:1.16.0
    restart: unless-stopped
    environment:
      - PANGOLIN_ENDPOINT=...
      - NEWT_ID=...
      - NEWT_SECRET=...
      - DOCKER_SOCKET=/var/run/docker.sock
      - DOCKER_ENFORCE_NETWORK_VALIDATION=true
      - LOG_LEVEL=DEBUG
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      bridge-newt:
      bridge-noicc:

  # Reachable: shares only bridge-newt with newt
  ok-service:
    image: traefik/whoami
    networks:
      bridge-newt:
      bridge-something:
    labels:
      - pangolin.private-resources.ok-service.name=OK Service
      - pangolin.private-resources.ok-service.mode=http
      - pangolin.private-resources.ok-service.full-domain=ok.${INTRANET_DOMAIN}
      - pangolin.private-resources.ok-service.destination=ok-service
      - pangolin.private-resources.ok-service.destination-port=80
      - pangolin.private-resources.ok-service.scheme=http

  # Unreachable: also shares bridge-noicc with newt
  broken-service:
    image: traefik/whoami
    networks:
      bridge-newt:
      bridge-noicc:
      bridge-something:
    labels:
      - pangolin.private-resources.broken-service.name=Broken Service
      - pangolin.private-resources.broken-service.mode=http
      - pangolin.private-resources.broken-service.full-domain=broken.${INTRANET_DOMAIN}
      - pangolin.private-resources.broken-service.destination=broken-service
      - pangolin.private-resources.broken-service.destination-port=80
      - pangolin.private-resources.broken-service.scheme=http

Reproduce with the compose above ok-service works, broken-service times out.
The two containers are identical apart from bridge-noicc.

Newt log:

INFO:  HTTP handler: GET / -> http://broken-service:80
ERROR: HTTP handler: upstream error (GET / -> http://broken-service:80): dial tcp 172.22.0.11:80: i/o timeout

The service itself is fine, curling it from another container succeeds.

Expected Behavior

When Newt and a container share a docker network, referencing the container via its container name or service name should reach it.

[AstralDestiny]

So you have within the docker network multiple networks each having the same service name for different services?

[GeoffreyCoulaud]

No, I have multiple services, with each a unique service name.

The issue is that service broken-service is on bridge-noicc with IP 10.0.0.2 and on bridge-newt with IP 10.1.0.2.
When newt wants to join broken-service by name, it asks the internal docker DNS, and gets two A records, 10.0.0.2 and 10.1.0.2.
Newt doesn't discriminate between those IPs since it's also on both of these networks, and picks (probably at random), the one on bridge-noicc which won't be able to join the service, since inter container communication is off on that docker network.

The workaround disembiguates by having the name broken-service.newt pointing only to 10.1.0.2, since that alias is only on the bridge-newt network.

[AstralDestiny]

Still strange that it's getting an ip near the start of the 10.x.x.x either way but if both reach it shouldn't matter.. I'm more questioning on the deployment of the docker networks..

Even if I take into account your noicc you got ICC/Internal block container communication + don't make routes to allow it to reach the host. However broken service can still reach ok-service and newt and any other containers bridge-newt is in. So the noicc isn't giving it any extra security at all as it can always still reach the rest of the network and containers via it's shared network via bridge-newt. So not fully sure on the architecture of this honestly even as an example. Even for a prod workload.. Unless you were putting in rules to the container to force unidirectional connections.

broken-service
      │
      ├── noicc/internal ── Can't path to peers
      │
      └── bridge-newt ───── Can path to peers
                              │
                              ├── newt
                              ├── ok-service
                              └── other containers

You could use aliases but the topology isn't security so hopefully it's just a test example.. would not use in prod. though for something like where I need icc disabled and internal I'd have to sidecar a container to write iptable entries to those containers to just enable unidirectional connections.. else if I don't the icc+internal is security theater as it has a route via your other interface.

But the option to select networks would be good either way will see about it and how far away it might be if any. My own preferable usecase would be for when there is ipvlan's or a overlay network attached alongside as might have the same container name the same.

[GeoffreyCoulaud]

This is a test example, don't worry I understand that bridge-newt would enable services to join each-other. The 10.x.x.x IPs are for the example too, to be more legible at a glance.

A simpler and plausible use case would be having the compose project's default bridge internal to only selectively give access to the internet to some specific containers, and have a noicc network with internet access. In that case, the issue would arise exactly the same.

Really sorry to have sent you for a spin on this explanation, I should have picked a better example. It will be helpful to anyone passing by anyway.
Point is: Selecting networks would be useful.

[AstralDestiny]

No no I meant to send it earlier, Figured I'd say it either way as some might have tried to use it or whatnot.