Bug: Openvpn 2.6 causes high CPU usage

[engageub]

Is this urgent?

None

Host OS

Ubuntu 22

CPU arch

x86_64

VPN service provider

OPENVPN

What are you using to run the container

docker run

What is the version of Gluetun

Running version latest built on 2024-05-18T18:08:57.405Z (commit 4218dba)

What's the problem 🤔

When the docker image qmcgaw/gluetun is used. The CPU utilization of the container goes to about 100% of 1 CORE where as the following image qmcgaw/gluetun:v3.37.0 uses less than 1% of 1 CORE.
Could you please look into the latest version and compare it with v3.37.0.

Share your logs (at least 10 lines)

root@vmi1921324:~/InternetIncome-test# sudo docker container logs gluetuntest
========================================
========================================
=============== gluetun ================
========================================
=========== Made with ❤️ by ============
======= https://github.com/qdm12 =======
========================================
========================================

Running version latest built on 2024-05-18T18:08:57.405Z (commit 4218dba)

🔧 Need help? https://github.com/qdm12/gluetun/discussions/new
🐛 Bug? https://github.com/qdm12/gluetun/issues/new
✨ New feature? https://github.com/qdm12/gluetun/issues/new
☕ Discussion? https://github.com/qdm12/gluetun/discussions/new
💻 Email? quentin.mcgaw@gmail.com
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2024-06-07T07:37:19Z INFO [routing] default route found: interface eth0, gateway 172.17.0.1, assigned IP 172.17.0.8 and family v4
2024-06-07T07:37:19Z INFO [routing] local ethernet link found: eth0
2024-06-07T07:37:19Z INFO [routing] local ipnet found: 172.17.0.0/16
2024-06-07T07:37:19Z INFO [firewall] enabling...
2024-06-07T07:37:19Z DEBUG [firewall] iptables-legacy --policy INPUT DROP
2024-06-07T07:37:19Z DEBUG [firewall] iptables-legacy --policy OUTPUT DROP
2024-06-07T07:37:19Z DEBUG [firewall] iptables-legacy --policy FORWARD DROP
2024-06-07T07:37:19Z DEBUG [firewall] ip6tables --policy INPUT DROP
2024-06-07T07:37:19Z DEBUG [firewall] ip6tables --policy OUTPUT DROP
2024-06-07T07:37:19Z DEBUG [firewall] ip6tables --policy FORWARD DROP
2024-06-07T07:37:19Z DEBUG [firewall] iptables-legacy --append INPUT -i lo -j ACCEPT
2024-06-07T07:37:19Z DEBUG [firewall] ip6tables --append INPUT -i lo -j ACCEPT
2024-06-07T07:37:19Z DEBUG [firewall] iptables-legacy --append OUTPUT -o lo -j ACCEPT
2024-06-07T07:37:19Z DEBUG [firewall] ip6tables --append OUTPUT -o lo -j ACCEPT
2024-06-07T07:37:19Z DEBUG [firewall] iptables-legacy --append OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2024-06-07T07:37:19Z DEBUG [firewall] ip6tables --append OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2024-06-07T07:37:19Z DEBUG [firewall] iptables-legacy --append INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2024-06-07T07:37:19Z DEBUG [firewall] ip6tables --append INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2024-06-07T07:37:19Z DEBUG [firewall] iptables-legacy --append OUTPUT -o eth0 -s 172.17.0.8 -d 172.17.0.0/16 -j ACCEPT
2024-06-07T07:37:19Z DEBUG [firewall] ip6tables --append OUTPUT -o eth0 -d ff02::1:ff/104 -j ACCEPT
2024-06-07T07:37:19Z DEBUG [firewall] iptables-legacy --append INPUT -i eth0 -d 172.17.0.0/16 -j ACCEPT
2024-06-07T07:37:19Z INFO [firewall] enabled successfully
2024-06-07T07:37:20Z INFO [storage] creating /gluetun/servers.json with 19425 hardcoded servers
2024-06-07T07:37:20Z DEBUG [netlink] IPv6 is not supported after searching 0 routes
2024-06-07T07:37:20Z INFO Alpine version: 3.19.1
2024-06-07T07:37:20Z INFO OpenVPN 2.5 version: 2.5.8
2024-06-07T07:37:20Z INFO OpenVPN 2.6 version: 2.6.8
2024-06-07T07:37:20Z INFO Unbound version: 1.20.0
2024-06-07T07:37:20Z INFO IPtables version: v1.8.10
2024-06-07T07:37:20Z INFO Settings summary:
├── VPN settings:
|   ├── VPN provider settings:
|   |   ├── Name: custom
|   |   └── Server selection settings:
|   |       ├── VPN type: openvpn
|   |       └── OpenVPN server selection settings:
|   |           ├── Protocol: UDP
|   |           └── Custom configuration file: /gluetun/custom.conf
|   └── OpenVPN settings:
|       ├── OpenVPN version: 2.6
|       ├── User: [set]
|       ├── Password: [set]
|       ├── Custom configuration file: /gluetun/custom.conf
|       ├── Network interface: tun0
|       ├── Run OpenVPN as: root
|       └── Verbosity level: 1
├── DNS settings:
|   ├── Keep existing nameserver(s): no
|   ├── DNS server address to use: 127.0.0.1
|   └── DNS over TLS settings:
|       └── Enabled: no
├── Firewall settings:
|   └── Enabled: yes
├── Log settings:
|   └── Log level: debug
├── Health settings:
|   ├── Server listening address: 127.0.0.1:9999
|   ├── Target address: cloudflare.com:443
|   ├── Duration to wait after success: 5s
|   ├── Read header timeout: 100ms
|   ├── Read timeout: 500ms
|   └── VPN wait durations:
|       ├── Initial duration: 6s
|       └── Additional duration: 5s
├── Shadowsocks server settings:
|   └── Enabled: no
├── HTTP proxy settings:
|   └── Enabled: no
├── Control server settings:
|   ├── Listening address: :8000
|   └── Logging: yes
├── OS Alpine settings:
|   ├── Process UID: 1000
|   └── Process GID: 1000
├── Public IP settings:
|   ├── Fetching: every 12h0m0s
|   ├── IP file path: /tmp/gluetun/ip
|   └── Public IP data API: ipinfo
└── Version settings:
    └── Enabled: yes
2024-06-07T07:37:20Z INFO [routing] default route found: interface eth0, gateway 172.17.0.1, assigned IP 172.17.0.8 and family v4
2024-06-07T07:37:20Z DEBUG [routing] ip rule add from 172.17.0.8/32 lookup 200 pref 100
2024-06-07T07:37:20Z INFO [routing] adding route for 0.0.0.0/0
2024-06-07T07:37:20Z DEBUG [routing] ip route replace 0.0.0.0/0 via 172.17.0.1 dev eth0 table 200
2024-06-07T07:37:20Z INFO [firewall] setting allowed subnets...
2024-06-07T07:37:20Z INFO [routing] default route found: interface eth0, gateway 172.17.0.1, assigned IP 172.17.0.8 and family v4
2024-06-07T07:37:20Z DEBUG [routing] ip rule add to 172.17.0.0/16 lookup 254 pref 98
2024-06-07T07:37:20Z INFO [dns] using plaintext DNS at address 8.8.8.8
2024-06-07T07:37:20Z INFO [http server] http server listening on [::]:8000
2024-06-07T07:37:20Z INFO [firewall] allowing VPN connection...
2024-06-07T07:37:20Z DEBUG [firewall] iptables-legacy --append OUTPUT -d 211.104.231.58 -o eth0 -p tcp -m tcp --dport 1489 -j ACCEPT
2024-06-07T07:37:20Z INFO [healthcheck] listening on 127.0.0.1:9999
2024-06-07T07:37:20Z DEBUG [firewall] iptables-legacy --append OUTPUT -o tun0 -j ACCEPT
2024-06-07T07:37:20Z DEBUG [firewall] ip6tables --append OUTPUT -o tun0 -j ACCEPT
2024-06-07T07:37:20Z INFO [openvpn] DEPRECATED OPTION: --cipher set to 'AES-128-CBC' but missing in --data-ciphers (AES-256-GCM:AES-1        28-GCM:CHACHA20-POLY1305). OpenVPN ignores --cipher for cipher negotiations.
2024-06-07T07:37:20Z INFO [openvpn] OpenVPN 2.6.8 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
2024-06-07T07:37:20Z INFO [openvpn] library versions: OpenSSL 3.1.4 24 Oct 2023, LZO 2.10
2024-06-07T07:37:20Z WARN [openvpn] No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mi        tm for more info.
2024-06-07T07:37:20Z INFO [openvpn] TCP/UDP: Preserving recently used remote address: [AF_INET]211.104.231.58:1489
2024-06-07T07:37:20Z INFO [openvpn] Attempting to establish TCP connection with [AF_INET]211.104.231.58:1489
2024-06-07T07:37:20Z INFO [openvpn] TCP connection established with [AF_INET]211.104.231.58:1489
2024-06-07T07:37:20Z INFO [openvpn] TCPv4_CLIENT link local: (not bound)
2024-06-07T07:37:20Z INFO [openvpn] TCPv4_CLIENT link remote: [AF_INET]211.104.231.58:1489
2024-06-07T07:37:21Z INFO [openvpn] [opengw.net] Peer Connection Initiated with [AF_INET]211.104.231.58:1489
2024-06-07T07:37:22Z INFO [openvpn] OPTIONS ERROR: failed to negotiate cipher with server.  Add the server's cipher ('AES-128-CBC') t        o --data-ciphers (currently 'AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305') if you want to connect to this server.
2024-06-07T07:37:22Z ERROR [openvpn] Failed to apply push options
2024-06-07T07:37:22Z INFO [openvpn] Failed to open tun/tap interface
2024-06-07T07:37:22Z INFO [openvpn] SIGUSR1[soft,process-push-msg-failed] received, process restarting
2024-06-07T07:37:26Z INFO [healthcheck] program has been unhealthy for 6s: restarting VPN
2024-06-07T07:37:26Z INFO [healthcheck] 👉 See https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md
2024-06-07T07:37:26Z INFO [healthcheck] DO NOT OPEN AN ISSUE UNLESS YOU READ AND TRIED EACH POSSIBLE SOLUTION
2024-06-07T07:37:26Z INFO [vpn] stopping
2024-06-07T07:37:26Z INFO [vpn] starting
2024-06-07T07:37:26Z INFO [firewall] allowing VPN connection...
2024-06-07T07:37:26Z INFO [openvpn] DEPRECATED OPTION: --cipher set to 'AES-128-CBC' but missing in --data-ciphers (AES-256-GCM:AES-1        28-GCM:CHACHA20-POLY1305). OpenVPN ignores --cipher for cipher negotiations.
2024-06-07T07:37:26Z INFO [openvpn] OpenVPN 2.6.8 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
2024-06-07T07:37:26Z INFO [openvpn] library versions: OpenSSL 3.1.4 24 Oct 2023, LZO 2.10
2024-06-07T07:37:26Z WARN [openvpn] No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mi        tm for more info.
2024-06-07T07:37:26Z INFO [openvpn] TCP/UDP: Preserving recently used remote address: [AF_INET]211.104.231.58:1489
2024-06-07T07:37:26Z INFO [openvpn] Attempting to establish TCP connection with [AF_INET]211.104.231.58:1489
2024-06-07T07:37:26Z INFO [openvpn] TCP connection established with [AF_INET]211.104.231.58:1489
2024-06-07T07:37:26Z INFO [openvpn] TCPv4_CLIENT link local: (not bound)
2024-06-07T07:37:26Z INFO [openvpn] TCPv4_CLIENT link remote: [AF_INET]211.104.231.58:1489
2024-06-07T07:37:27Z INFO [openvpn] [opengw.net] Peer Connection Initiated with [AF_INET]211.104.231.58:1489
2024-06-07T07:37:28Z INFO [openvpn] OPTIONS ERROR: failed to negotiate cipher with server.  Add the server's cipher ('AES-128-CBC') t        o --data-ciphers (currently 'AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305') if you want to connect to this server.
2024-06-07T07:37:28Z ERROR [openvpn] Failed to apply push options
2024-06-07T07:37:28Z INFO [openvpn] Failed to open tun/tap interface
2024-06-07T07:37:28Z INFO [openvpn] SIGUSR1[soft,process-push-msg-failed] received, process restarting

Share your configuration

-e LOG_LEVEL=debug -e VPN_SERVICE_PROVIDER=custom -e VPN_TYPE=openvpn -e OPENVPN_USER=vpn -e OPENVPN_PASSWORD=vpn -v $volume -e OPENVPN_CUSTOM_CONFIG=/gluetun/custom.conf -v '/dev/net/tun:/dev/net/tun' --cap-add=NET_ADMIN -e DOT=off -e DOT_PROVIDERS=google,cloudflare -e DOT_CACHING=off -e BLOCK_MALICIOUS=off qmcgaw/gluetun

[github-actions]

@qdm12 is more or less the only maintainer of this project and works on it in his free time.
Please:

• do not ask for updates, be patient
• 👍 the issue to show your support instead of commenting
  @qdm12 usually checks issues at least once a week, if this is a new urgent bug,
  revert to an older tagged container image

[elmagow]

Same for me on latest Fedora.
Not as high usage, but still on top of my cpu usage with my server doing nothing

[qdm12]

OpenVPN version: 2.6 is the difference. Try using OPENVPN_VERSION=2.5?

[engageub]

Hi,
Thank you for the response. Yes, OPENVPN_VERSION=2.5 reduces the CPU similar to v3.37.0. However, there is a problem with consistency in the latest version.
When I run the command sudo docker stats <gluetun_container_name> for about a minute to get the stats, the CPU utilization suddenly spikes to 100% and comes back to normal, whereas with v3.37.0 this is not the case.

Thank you

[qdm12]

reduces the CPU similar to v3.37.0.

Do you also have the problem with v3.38.0?

Anyway if Openvpn 2.6 is at fault, there isn't much I can do as far as I know, nothing changed except the openvpn version. Still a strange issue... It might be worth reporting it to the OpenVPN dev team? 🤔

the CPU utilization suddenly spikes to 100% and comes back to normal, whereas with v3.37.0 this is not the case.

I'm not sure I understand this fully, the CPU spikes to 100% for Gluetun only, or for the entire machine, and for how long? Does it happen only when querying docker stats?

[engageub]

Do you also have the problem with v3.38.0?

v3.38.0 is slightly better than v3.37.0 when compared to memory usage. v3.38.0 was consuming about 56 MB where as v3.37.0 was consuming about 64 MB.
CPU is normal in this version.

Anyway if Openvpn 2.6 is at fault, there isn't much I can do as far as I know, nothing changed except the openvpn version. Still a strange issue... It might be worth reporting it to the OpenVPN dev team? 🤔

If OpenVPN version is the only problem, then it is supposed to be informed to them to resolve the issue.

I'm not sure I understand this fully, the CPU spikes to 100% for Gluetun only, or for the entire machine, and for how long? Does it happen only when querying docker stats?

I started the container with --cpus=1 options in 4 core machine. The CPU is 100% only for gluetun container displayed by docker stats command.
This can also be tested on Play with Docker website directly without using --cpus option.

[engageub]

In v3.37.0, I also observed that the default health checks were consuming more CPU. A 5-second interval ping is too aggressive for health checks. By disabling health checks and removing the exposed ports, CPU utilization has drastically reduced. I was only able to run 100 containers earlier, which reached about 90% CPU utilization. After removing health checks and deleting the ports, 100 containers are now using only about 10% CPU in total. This shows a significant variation in CPU usage with health checks enabled. Memory utilization has also dropped below 20 MB for each container after removing health checks and exposed ports.

Thank you

[qdm12]

By removing the exposed ports

Do you mean by removing the EXPOSE instruction in the Dockerfile?