Bug: VPN Port forwarding doesn't work always after restart and doesn't try again

[christianmerges]

What's the feature 🧐

Currently the VPN-Provider assigns a random port. But my Application is listening on a fixed listen port. So I have to create another container and change the port manually before starting the container:

tcp-app:
image: tecnativa/tcp-proxy
container_name: tcp-app
environment:
LISTEN: ":35386" #here I have to put the random port from the vpn provider
TALK: "172.18.0.1:12345" #this is where the application runs
network_mode: "service:glue-app"

It totally works like that. I'm just looking for an easier and more automated way. Maybe TCP Proxy can be implemented within the gluetun container like we already have for shadowsocks and http proxy.

Extra information and references

No response

[github-actions]

@qdm12 is more or less the only maintainer of this project and works on it in his free time.
Please:

• do not ask for updates, be patient
• 👍 the issue to show your support instead of commenting
  @qdm12 usually checks issues at least once a week, if this is a new urgent bug,
  revert to an older tagged container image

[qdm12]

Doesn't VPN_PORT_FORWARDING_LISTENING_PORT solve this (see https://github.com/qdm12/gluetun-wiki/blob/main/setup/options/port-forwarding.md)?

If it does, what wiki page did you visit? Maybe that information is not mentioned that much and could use some improvement.

[christianmerges]

I tried this now and it didn't work. But I don't know why. I shutdown the tcp proxy container and added this to the gluetun container as environment variable:
VPN_PORT_FORWARDING_LISTENING_PORT=12345
The application was not reachable on the wanip with the random port of the tunnel.

Maybe additional firewall rule needed?

[qdm12]

Please double check using those steps:

#2354 (comment)

Note this was fixed a few days ago only, so make sure you use the latest image (re-pull it).

[christianmerges]

I can see the difference. In my scenario the application container is also using gluetun as docker network for outbound connections. Maybe this is why it doesn't work for me. I will now replace my application with the Portcheck container and give it a trial.

Also with proton, sometimes I don't get a port (probably wrong Server without P2P), can gluetun in this scenario try to connect to another Server on it's own? At the moment I restart the container several times till I'm lucky to get a port. And I also have to restart applications because it breaks the docker network.

[qdm12]

In my scenario the application container is also using gluetun as docker network for outbound connections

That's irrelevant to the VPN server side port forwarding though, right? Outbound connections can go freely through the VPN, I don't see how this matters here.

Also with proton, sometimes I don't get a port (probably wrong Server without P2P), can gluetun in this scenario try to connect to another Server on it's own?

This should not be the case since commit (part of v3.39 release and latest) 5d75bbc

What error are you getting in your logs?

[christianmerges]

Running version latest built on 2024-08-25T07:04:32.409Z (commit 01fa993)

🔧 Need help? ☕ Discussion? https://github.com/qdm12/gluetun/discussions/new/choose
🐛 Bug? ✨ New feature? https://github.com/qdm12/gluetun/issues/new/choose
💻 Email? quentin.mcgaw@gmail.com
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2024-08-29T13:39:08Z INFO [routing] default route found: interface eth0, gateway 172.18.0.1, assigned IP 172.18.0.2 and family v4
2024-08-29T13:39:08Z INFO [routing] local ethernet link found: eth0
2024-08-29T13:39:08Z INFO [routing] local ipnet found: 172.18.0.0/16
2024-08-29T13:39:08Z INFO [firewall] enabling...
2024-08-29T13:39:09Z INFO [firewall] enabled successfully
2024-08-29T13:39:09Z INFO [storage] creating /gluetun/servers.json with 20480 hardcoded servers
2024-08-29T13:39:09Z INFO Alpine version: 3.20.2
2024-08-29T13:39:09Z INFO OpenVPN 2.5 version: 2.5.10
2024-08-29T13:39:09Z INFO OpenVPN 2.6 version: 2.6.11
2024-08-29T13:39:09Z INFO IPtables version: v1.8.10
2024-08-29T13:39:09Z INFO Settings summary:
├── VPN settings:
|   ├── VPN provider settings:
|   |   ├── Name: protonvpn
|   |   ├── Server selection settings:
|   |   |   ├── VPN type: openvpn
|   |   |   ├── Countries: Cyprus
|   |   |   ├── Cities: Limassol
|   |   |   ├── Port forwarding only servers: yes
|   |   |   └── OpenVPN server selection settings:
|   |   |       └── Protocol: UDP
|   |   └── Automatic port forwarding settings:
|   |       ├── Redirection listening port: 32400
|   |       ├── Use code for provider: protonvpn
|   |       ├── Forwarded port file path: /tmp/gluetun/forwarded_port
|   |       └── Credentials:
|   |           ├── Username: 25...+nr+f2+pmp
|   |           └── Password: n7...9dK
|   └── OpenVPN settings:
|       ├── OpenVPN version: 2.6
|       ├── User: [set]
|       ├── Password: n7...9dK
|       ├── Network interface: tun0
|       ├── Run OpenVPN as: root
|       └── Verbosity level: 1
├── DNS settings:
|   ├── Keep existing nameserver(s): no
|   ├── DNS server address to use: 127.0.0.1
|   └── DNS over TLS settings:
|       ├── Enabled: yes
|       ├── Update period: every 24h0m0s
|       ├── Upstream resolvers:
|       |   └── cloudflare
|       ├── Caching: yes
|       ├── IPv6: no
|       └── DNS filtering settings:
|           ├── Block malicious: yes
|           ├── Block ads: no
|           ├── Block surveillance: no
|           └── Blocked IP networks:
|               ├── 127.0.0.1/8
|               ├── 10.0.0.0/8
|               ├── 172.16.0.0/12
|               ├── 192.168.0.0/16
|               ├── 169.254.0.0/16
|               ├── ::1/128
|               ├── fc00::/7
|               ├── fe80::/10
|               ├── ::ffff:127.0.0.1/104
|               ├── ::ffff:10.0.0.0/104
|               ├── ::ffff:169.254.0.0/112
|               ├── ::ffff:172.16.0.0/108
|               └── ::ffff:192.168.0.0/112
├── Firewall settings:
|   ├── Enabled: yes
|   └── Outbound subnets:
|       └── 100.64.0.0/10
├── Log settings:
|   └── Log level: info
├── Health settings:
|   ├── Server listening address: 127.0.0.1:9999
|   ├── Target address: cloudflare.com:443
|   ├── Duration to wait after success: 5s
|   ├── Read header timeout: 100ms
|   ├── Read timeout: 500ms
|   └── VPN wait durations:
|       ├── Initial duration: 6s
|       └── Additional duration: 5s
├── Shadowsocks server settings:
|   └── Enabled: no
├── HTTP proxy settings:
|   └── Enabled: no
├── Control server settings:
|   ├── Listening address: :8000
|   └── Logging: yes
├── Storage settings:
|   └── Filepath: /gluetun/servers.json
├── OS Alpine settings:
|   ├── Process UID: 1000
|   └── Process GID: 1000
├── Public IP settings:
|   ├── Fetching: every 12h0m0s
|   ├── IP file path: /tmp/gluetun/ip
|   └── Public IP data API: ipinfo
└── Version settings:
    └── Enabled: yes
2024-08-29T13:39:09Z INFO [openvpn] [node-cy-01.protonvpn.net] Peer Connection Initiated with [AF_INET]85.132.252.34:1194
2024-08-29T13:39:22Z ERROR [vpn] starting port forwarding service: port forwarding for the first time: getting external IPv4 address: executing remote procedure call: reading from udp connection: read udp 10.22.0.3:51944->10.22.0.1:5351: recvfrom: connection refused - make sure you have +pmp at the end of your OpenVPN username

Only restarting several times gave me randomly a server, which accepted my request.

But I can confirm the portforwarding directly to my application to be working. Maybe last time i was not waiting enough time till the webserver was ready. Should i open another issuie regarding to still existing bug with not getting a port and not trying another server from the pool?

[qdm12]

But I can confirm the portforwarding directly to my application to be working. Maybe last time i was not waiting enough time till the webserver was ready.

✅

---

The two logs line here:

2024-08-29T13:39:09Z INFO [openvpn] [node-cy-01.protonvpn.net] Peer Connection Initiated with [AF_INET]85.132.252.34:1194
2024-08-29T13:39:22Z ERROR [vpn] starting port forwarding service: port forwarding for the first time: getting external IPv4 address: executing remote procedure call: reading from udp connection: read udp 10.22.0.3:51944->10.22.0.1:5351: recvfrom: connection refused - make sure you have +pmp at the end of your OpenVPN username

Did you remove some log lines ⁉️ Because the VPN connection is not up yet (need to wait for Initialization completed), and so port forwarding should not trigger before this. It fails with connection refused because the firewall blocks it because the VPN connection is not working yet.

On top of that, if you search IP address 85.132.252.34 in the servers.json it is marked as supporting port forwarding "port_forward": true, so the server is correct to use port forwarding. You can also check from https://api.protonmail.ch/vpn/logicals search for that same IP address, to which you can find "Features":28 which means it supports port forwarding (see https://github.com/ProtonVPN/protonvpn-nm-lib/blob/31d5f99fbc89274e4e977a11e7432c0eab5a3ef8/protonvpn_nm_lib/enums.py#L47)

[christianmerges]

i have multiple gluetun containers running, because one container can only forward one port. Maybe there is a conflict between the containers. There is two problems. One is that the port becomes unavailable after some time, but this can be managed by restarting the tunnel. So usually i restart the tunnel with this command, because then I don't have to restart the container behind gluetun:

docker exec applicationBehindGluetunContainer curl -X PUT http://localhost:8000/v1/openvpn/status
-H "Content-Type: application/json"
-d '{"status":"stopped"}'

Then the following logfile shows the console of the the gluetun container when it receives the stop signal.

2024-09-18T22:53:23.680336948Z 2024-09-18T22:53:23Z INFO [http server] 200 GET /portforwarded wrote 15B to [::1]:35488 in 192.643µs
2024-09-18T22:54:23.725473078Z 2024-09-18T22:54:23Z INFO [http server] 200 GET /portforwarded wrote 15B to [::1]:45482 in 185.164µs
2024-09-18T22:55:23.786942454Z 2024-09-18T22:55:23Z INFO [http server] 200 GET /portforwarded wrote 15B to [::1]:37720 in 22.819µs
2024-09-18T22:56:23.844802848Z 2024-09-18T22:56:23Z INFO [http server] 200 GET /portforwarded wrote 15B to [::1]:42686 in 371.898µs
2024-09-18T22:57:23.914993871Z 2024-09-18T22:57:23Z INFO [http server] 200 GET /portforwarded wrote 15B to [::1]:53482 in 379.167µs
2024-09-18T22:58:23.955369061Z 2024-09-18T22:58:23Z INFO [http server] 200 GET /portforwarded wrote 15B to [::1]:44968 in 283.781µs
2024-09-18T22:59:24.019107509Z 2024-09-18T22:59:24Z INFO [http server] 200 GET /portforwarded wrote 15B to [::1]:57336 in 418.067µs
2024-09-18T23:00:24.074240849Z 2024-09-18T23:00:24Z INFO [http server] 200 GET /portforwarded wrote 15B to [::1]:36266 in 23.25µs
2024-09-18T23:00:38.771799778Z 2024-09-18T23:00:38Z INFO [vpn] stopping
2024-09-18T23:00:38.772311421Z 2024-09-18T23:00:38Z INFO [port forwarding] stopping
2024-09-18T23:00:38.772365819Z 2024-09-18T23:00:38Z INFO [firewall] removing allowed port 57086...
2024-09-18T23:00:38.841382024Z 2024-09-18T23:00:38Z INFO [port forwarding] removing port file /tmp/gluetun/forwarded_port
2024-09-18T23:00:38.863762049Z 2024-09-18T23:00:38Z INFO [http server] 200 PUT /status wrote 22B to [::1]:35988 in 92.418717ms
2024-09-18T23:00:50.292757125Z 2024-09-18T23:00:50Z INFO [healthcheck] program has been unhealthy for 6s: restarting VPN
2024-09-18T23:00:50.292861622Z 2024-09-18T23:00:50Z INFO [healthcheck] 👉 See https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md
2024-09-18T23:00:50.292885391Z 2024-09-18T23:00:50Z INFO [healthcheck] DO NOT OPEN AN ISSUE UNLESS YOU READ AND TRIED EACH POSSIBLE SOLUTION
2024-09-18T23:00:50.292905830Z 2024-09-18T23:00:50Z INFO [vpn] starting
2024-09-18T23:00:50.293522920Z 2024-09-18T23:00:50Z INFO [firewall] allowing VPN connection...
2024-09-18T23:00:50.369502562Z 2024-09-18T23:00:50Z INFO [openvpn] OpenVPN 2.6.11 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
2024-09-18T23:00:50.369566660Z 2024-09-18T23:00:50Z INFO [openvpn] library versions: OpenSSL 3.3.1 4 Jun 2024, LZO 2.10
2024-09-18T23:00:50.371282985Z 2024-09-18T23:00:50Z INFO [openvpn] TCP/UDP: Preserving recently used remote address: [AF_INET]149.88.102.33:1194
2024-09-18T23:00:50.371340823Z 2024-09-18T23:00:50Z INFO [openvpn] UDPv4 link local: (not bound)
2024-09-18T23:00:50.371363563Z 2024-09-18T23:00:50Z INFO [openvpn] UDPv4 link remote: [AF_INET]a.b.c.d:1194
2024-09-18T23:00:50.407002824Z 2024-09-18T23:00:50Z INFO [openvpn] [node-de-21.protonvpn.net] Peer Connection Initiated with [AF_INET]149.88.102.33:1194
2024-09-18T23:00:51.425658199Z 2024-09-18T23:00:51Z INFO [openvpn] setsockopt TCP_NODELAY=1 failed
2024-09-18T23:00:51.426025288Z 2024-09-18T23:00:51Z INFO [openvpn] TUN/TAP device tun0 opened
2024-09-18T23:00:51.426090846Z 2024-09-18T23:00:51Z INFO [openvpn] /sbin/ip link set dev tun0 up mtu 1500
2024-09-18T23:00:51.432178461Z 2024-09-18T23:00:51Z INFO [openvpn] /sbin/ip link set dev tun0 up
2024-09-18T23:00:51.433945605Z 2024-09-18T23:00:51Z INFO [openvpn] /sbin/ip addr add dev tun0 10.16.0.4/16
2024-09-18T23:00:51.440865494Z 2024-09-18T23:00:51Z INFO [openvpn] UID set to nonrootuser
2024-09-18T23:00:51.440888583Z 2024-09-18T23:00:51Z INFO [openvpn] Initialization Sequence Completed
2024-09-18T23:00:52.454233767Z 2024-09-18T23:00:52Z INFO [healthcheck] healthy!
2024-09-18T23:00:52.679819487Z 2024-09-18T23:00:52Z INFO [ip getter] Public IP address is a.b.c.d (Germany, Hesse, Frankfurt am Main)
2024-09-18T23:00:52.679968372Z 2024-09-18T23:00:52Z INFO [port forwarding] starting
2024-09-18T23:00:52.686620080Z 2024-09-18T23:00:52Z ERROR [vpn] starting port forwarding service: port forwarding for the first time: getting external IPv4 address: executing remote procedure call: reading from udp connection: read udp 10.16.0.4:43870->10.16.0.1:5351: recvfrom: connection refused - make sure you have +pmp at the end of your OpenVPN username

The gluetun container now doesn't try to do anything. It is staying at this forever. Then i invoke the command

docker exec applicationBehindGluetunContainer curl -X PUT http://localhost:8000/v1/openvpn/status
-H "Content-Type: application/json"
-d '{"status":"stopped"}'

several times till i get a working port forwarding. Also i see sometimes in the log this error:

2024-09-17T15:24:06.984481650Z 2024-09-17T15:24:06Z ERROR [openvpn] OpenVPN tried to add an IP route which already exists (RTNETLINK answers: File exists)

The not retrying after failed port forwarding is the second problem.