Bug: Still seeing high memory usage when using block malicious flag

[evandev]

Is this urgent?

No

Host OS

Amazon Linux 2

CPU arch

x86_64

VPN service provider

Surfshark

What are you using to run the container

docker-compose

What is the version of Gluetun

latest 3.27.0

What's the problem 🤔

I am still seeing very high memory using, I am attempting to run multiple instances of this container. Memory usage is over 300MB for each container.

Share your logs

`docker stats:

CONTAINER ID   NAME                     CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O         PIDS
e12518276347   openvpn_us-chicago_1     22.72%    304.5MiB / 3.829GiB   7.77%     7.05MB / 250kB   2.62MB / 2.64MB   12
4ae73e25a649   openvpn_us-buffalo_1     0.00%     305.2MiB / 3.829GiB   7.78%     5.91MB / 247kB   2.62MB / 2.65MB   12
7aaf9c4da775   openvpn_us-bend_1        0.00%     302.1MiB / 3.829GiB   7.71%     8.08MB / 274kB   2.62MB / 2.64MB   13
6bb3f860ab17   openvpn_us-dallas_1      0.00%     304.2MiB / 3.829GiB   7.76%     5.5MB / 225kB    2.69MB / 2.67MB   12
b7e747f1de17   openvpn_us-charlotte_1   0.00%     302.1MiB / 3.829GiB   7.71%     5.75MB / 232kB   2.62MB / 2.64MB   12
b19c9e5904c8   openvpn_us-ashburn_1     0.00%     303.3MiB / 3.829GiB   7.73%     5.68MB / 274kB   2.62MB / 2.64MB   12
145d862e97fe   openvpn_us-boston_1      0.34%     306.6MiB / 3.829GiB   7.82%     6.34MB / 262kB   4.11MB / 2.64MB   12`

Share your configuration

`version: "3"
services:
  us-ashburn:
    image: qmcgaw/gluetun
    cap_add:
      - NET_ADMIN
    ports:
      - 9000:8888/tcp # HTTP proxy
    volumes:
      - ./data/us-ashburn:/gluetun
    environment:
      - VPNSP=surfshark
      - OPENVPN_USER=[user]
      - OPENVPN_PASSWORD=[pass]
      - SERVER_HOSTNAME=us-ash.prod.surfshark.com
      - HTTPPROXY=on
      - HTTPPROXY_PORT=8888
      - BLOCK_MALCIOUS=off
   `

[qdm12]

Hey @evandev !

Try turning DOT_CACHING=off? Caching might play a role there for now.

What you can try to debug yourself relatively easily is:

docker exec -it <your-gluetun-container-name> apk add htop && htop

You can then click on the RES column header to sort by reserved memory usage.
Press CTRL + C to stop htop. That should show if it's Unbound, gluetun or OpenVPN at fault.

I'll soon fix #137 which should offer finer grain control of performance vs memory for DNS related stuff.

I'm away from keyboard right now, but I'll try to investigate it myself soon.

And also 🥇 for the sponsorship 😉

[evandev]

Thanks for the response. I set DOT_CACHING=off for Ashburn, Bend, and Boston. The other containers I left it on. Here are the results:

57a46cd147e7   openvpn_us-dallas_1      0.05%     301.9MiB / 3.829GiB   7.70%     6.4MB / 260kB    42.6MB / 2.64MB   13
ce0e8a696ebc   openvpn_us-buffalo_1     0.08%     302.3MiB / 3.829GiB   7.71%     6.72MB / 302kB   49.3MB / 2.65MB   14
221f24aa5082   openvpn_us-boston_1      0.05%     299.7MiB / 3.829GiB   7.64%     6.9MB / 296kB    16.8MB / 2.64MB   12
1a95cb6b1349   openvpn_us-chicago_1     0.04%     299MiB / 3.829GiB     7.63%     6.73MB / 291kB   17.3MB / 2.64MB   13
da21d9b9ec26   openvpn_us-bend_1        10.88%    305MiB / 3.829GiB     7.78%     7.37MB / 291kB   33.1MB / 2.64MB   22
395875529b38   openvpn_us-charlotte_1   0.04%     299.5MiB / 3.829GiB   7.64%     11.7MB / 500kB   141MB / 2.66MB    12
a58f6c97864f   openvpn_us-denver_1      0.05%     303MiB / 3.829GiB     7.73%     12.4MB / 477kB   216MB / 35.7MB    13
7d84e5658e63   openvpn_us-ashburn_1     0.04%     300MiB / 3.829GiB     7.65%     6.56MB / 307kB   55.2MB / 2.64MB   13```

[qdm12]

That's strange, did you pull the latest image? Check with htop (either in the container or on your host) if it's openvpn or unbound or /gluetun-entrypoint taking memory? For me it stays at around 50MB 🤔

[evandev]

docker pull qmcgaw/gluetun
Using default tag: latest
latest: Pulling from qmcgaw/gluetun
Digest: sha256:1b2303edd2f9f7d609f05328cae1861b49593b971014045f68a6abaee02c2356
Status: Image is up to date for qmcgaw/gluetun:latest
docker.io/qmcgaw/gluetun:latest

CONTAINER ID   NAME                     CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O         PIDS
ce0e8a696ebc   openvpn_us-buffalo_1     0.00%     445.2MiB / 3.829GiB   11.35%    6.27MB / 290kB   86kB / 35.6MB     12
221f24aa5082   openvpn_us-boston_1      0.00%     441.5MiB / 3.829GiB   11.26%    6.35MB / 254kB   8.19kB / 35.6MB   12
1a95cb6b1349   openvpn_us-chicago_1     0.00%     437.7MiB / 3.829GiB   11.16%    7.68MB / 268kB   713kB / 35.6MB    13
da21d9b9ec26   openvpn_us-bend_1        0.53%     427MiB / 3.829GiB     10.89%    6.61MB / 274kB   172kB / 35.6MB    13
395875529b38   openvpn_us-charlotte_1   0.00%     462MiB / 3.829GiB     11.78%    7.57MB / 285kB   1.43MB / 35.6MB   12
a58f6c97864f   openvpn_us-denver_1      122.90%   441.1MiB / 3.829GiB   11.25%    6.71MB / 253kB   2.03MB / 2.64MB   13
7d84e5658e63   openvpn_us-ashburn_1     0.00%     459.1MiB / 3.829GiB   11.71%    6MB / 252kB      8.19kB / 35.6MB   12

Htop is showing Unbound as using most of the memory. I am running 7 containers, htop shows 14 entries for unbound at about 283MB each.

[qdm12]

Are you sure you are running with both DOT_CACHING=off and BLOCK_MALICIOUS=off? 🤔 Maybe have a look at the settings tree logged out by one of your containers? Are you doing anything network-heavy on your VPN containers, that could explain the ram usage of Unbound? 🤔 Anyway I'll merge github.com/qdm12/dns in there rather soon to replace Unbound.

[evandev]

version: "3"
services:
  us-ashburn:
    image: qmcgaw/gluetun
    cap_add:
      - NET_ADMIN
    ports:
      - 9000:8888/tcp # HTTP proxy
    volumes:
      - ./data/us-ashburn:/gluetun
    environment:
      - VPNSP=surfshark
      - OPENVPN_USER=[user]
      - OPENVPN_PASSWORD=[password]
      - SERVER_HOSTNAME=us-ash.prod.surfshark.com
      - HTTPPROXY=on
      - HTTPPROXY_PORT=8888
      - BLOCK_MALCIOUS=off
      - DOT_CACHING=off

40c3a52011f7   openvpn_us-ashburn_1     0.00%     459.1MiB / 3.829GiB   11.71%    6.79MB / 281kB   2.62MB / 35.6MB   1

── VPN settings:
|   ├── VPN provider settings:
|   |   ├── Name: surfshark
|   |   └── Server selection settings:
|   |       ├── VPN type: openvpn
|   |       ├── Hostnames: us-ash.prod.surfshark.com
|   |       └── OpenVPN server selection settings:
|   |           └── Protocol: UDP
|   └── OpenVPN settings:
|       ├── OpenVPN version: 2.5
|       ├── User: [set]
|       ├── Password: [set]
|       ├── Tunnel IPv6: no
|       ├── Network interface: tun0
|       ├── Run OpenVPN as: root
|       └── Verbosity level: 1
├── DNS settings:
|   ├── DNS server address to use: 127.0.0.1
|   ├── Keep existing nameserver(s): no
|   └── DNS over TLS settings:
|       ├── Enabled: yes
|       ├── Update period: every 24h0m0s
|       ├── Unbound settings:
|       |   ├── Authoritative servers:
|       |   |   └── cloudflare
|       |   ├── Caching: no
|       |   ├── IPv6: no
|       |   ├── Verbosity level: 1
|       |   ├── Verbosity details level: 0
|       |   ├── Validation log level: 0
|       |   ├── System user: root
|       |   └── Allowed networks:
|       |       ├── 0.0.0.0/0
|       |       └── ::/0
|       └── DNS filtering settings:
|           ├── Block malicious: yes
|           ├── Block ads: no
|           ├── Block surveillance: no
|           └── Blocked IP networks:
|               ├── 127.0.0.1/8
|               ├── 10.0.0.0/8
|               ├── 172.16.0.0/12
|               ├── 192.168.0.0/16
|               ├── 169.254.0.0/16
|               ├── ::1/128
|               ├── fc00::/7
|               ├── fe80::/10
|               ├── ::ffff:7f00:1/104
|               ├── ::ffff:a00:0/104
|               ├── ::ffff:a9fe:0/112
|               ├── ::ffff:ac10:0/108
|               └── ::ffff:c0a8:0/112
├── Firewall settings:
|   └── Enabled: yes
├── Log settings:
|   └── Log level: INFO
├── Health settings:
|   ├── Server listening address: 127.0.0.1:9999
|   ├── Address to ping: github.com
|   └── VPN wait durations:
|       ├── Initial duration: 5s
|       └── Additional duration: 5s
├── Shadowsocks server settings:
|   └── Enabled: no
├── HTTP proxy settings:
|   ├── Enabled: yes
|   ├── Listening address: :8888
|   ├── User: 
|   ├── Password: [not set]
|   ├── Stealth mode: no
|   └── Log: no
├── Control server settings:
|   ├── Listening port: 8000
|   └── Logging: yes
├── OS Alpine settings:
|   ├── Process UID: 1000
|   └── Process GID: 1000
├── Public IP settings:
|   ├── Fetching: every 12h0m0s
|   └── IP file path: /tmp/gluetun/ip
└── Version settings:
    └── Enabled: yes

Double checked, posted my docker-compose snippet for you.

I'm not doing anything heavy yet, actually these stats above are minutes after a fresh start and before any traffic has been sent through.

The denver container has only 20mb of memory usage, however the logs show that there was a problem establishing the vpn connection in there, so probably irrelevant:

2022/01/27 12:45:09 WARN openvpn: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
🚒🚒🚒🚒🚒🚨🚨🚨🚨🚨🚨🚒🚒🚒🚒🚒
That error usually happens because either:

1. The VPN server IP address you are trying to connect to is no longer valid 🔌
   Update your server information using https://github.com/qdm12/gluetun/wiki/Updating-Servers

2. The VPN server crashed 💥, try changing your VPN servers filtering options such as REGION

3. Your Internet connection is not working 🤯, ensure it works

4. Something else ➡️ https://github.com/qdm12/gluetun/issues/new/choose
2022/01/27 12:45:09 INFO openvpn: TLS Error: TLS handshake failed
2022/01/27 12:45:09 INFO openvpn: SIGTERM received, sending exit notification to peer
2022/01/27 12:45:09 INFO openvpn: SIGTERM[soft,tls-error] received, process exiting
2022/01/27 12:45:09 INFO vpn: retrying in 15s
2022/01/27 12:45:09 INFO healthcheck: program has been unhealthy for 1m0s: restarting VPN
2022/01/27 12:45:24 INFO firewall: setting VPN connection through firewall...
2022/01/27 12:45:24 INFO openvpn: OpenVPN 2.5.4 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Nov 15 2021
2022/01/27 12:45:24 INFO openvpn: library versions: OpenSSL 1.1.1l  24 Aug 2021, LZO 2.10
2022/01/27 12:45:24 INFO openvpn: TCP/UDP: Preserving recently used remote address: [AF_INET]66.115.177.141:1194
2022/01/27 12:45:24 INFO openvpn: UDP link local: (not bound)
2022/01/27 12:45:24 INFO openvpn: UDP link remote: [AF_INET]66.115.177.141:119

[frepke]

Hi @evandev,

You have a typo in your compose-file - BLOCK_MALCIOUS=off is missing an I

[evandev]

@frepke Oops! Thank you, I am fixing this now.

[evandev]

That probably fixed my issue. Will continue testing.

b0cd2fa26dc2   openvpn_us-ashburn_1     0.06%     31.93MiB / 3.829GiB   0.81%     136kB / 46.6kB   0B / 2.63MB   12

[qdm12]

Awesome @frepke I really didn't spot this, good 👀 !!

Let's close the issue for now, feel free to comment here more though. Since this issue is linked to the DNS over HTTPs Go implementation, I'll probably comment here once it's done. I just ran my DNS container with block malicious and it uses 125MB of ram. Not amazing, but still twice better than Unbound! 😉