"open tun" Operation not permitted

[hugoghx]

First of all, thank you for putting this together. It's a great piece of software! ❤️

I've been using warp-docker for a while now in tunnel_only mode and it's all been great until a couple of days ago when connections relying on this proxy started failing.

I started looking into this and the first thing I saw was this error in the warp-docker logs, from warp-svc:

ERROR main_loop: warp::warp_service: Connection error error=IoError(AnnotatedIoError { err: Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" }, context: "open tun" })

I thought initially this was a permissions issue trying to interact with /dev/net/tun, so I went and had a look: crw------- 1 root root 10, 200 Dec 7 16:53 /dev/net/tun

Looked fine. The warp user is a sudoer, but for sanity's sake I chown'd /dev/net/tun to warp and even chmod'd it to 666. Same issue.

I tried a bunch of other combinations of settings and docker configurations (including using MASQUE and WG for the tunnel, and different modes too), but I wasn't able to resolve the issue or find a root cause.

For context, here's my warp-docker Docker Compose definition:

  warproxy:
    image: "caomingjun/warp"
    container_name: "warproxy"
    init: true
    cap_add:
      - "NET_ADMIN"
    environment:
      GOST_ARGS: "-L :42069"
    networks:
      - "warproxy_nw"
    sysctls:
      - "net.ipv6.conf.all.disable_ipv6=0"
      - "net.ipv4.conf.all.src_valid_mark=1"
    volumes:
      - "./config/warproxy/settings.json:/var/lib/cloudflare-warp/settings.json"
    tmpfs:
      - "/var/lib/cloudflare-warp:mode=770,size=16M,uid=0,gid=0"
    restart: "unless-stopped"

I even went back in warp-docker versions, but now the versions that used to work stopped to do so with the same error. Clearly, this was not an issue with any recent updates to this package.

I then went into trying to figure out what had changed on my system recently and the one thing I remembered was that apt had upgraded the containerd.io package on my Ubuntu Server 24.10 from 1.7.23-1 to 1.7.24-1.

I downgraded this package back to 1.7.23-1 and voila, problem solved.

I did not get to the root cause of this issue - I don't know if it's an incompatibility with warp-docker and this new version of containerd.io or with Cloudflare WARP itself.

I checked the changes that happened from containerd 1.7.23 to 1.7.24, but I see nothing that would immediately raise any flags. I did notice that runc and a few other dependencies got updated, so maybe more digging

--

I suspect no other issues like this have cropped up here given that this requires an update action for something that was released pretty recently. Perhaps we'll see more people mention this later as they upgrade their servers.

[cmj2002]

Thanks for your investigation.

This is an intentional change in runc:

opencontainers/runc#3468

I will give a solution ASAP.

[cmj2002]

Pass device /dev/net/tun can fix the problem. I will update the documentation.

[wanegrow]

First of all, thank you for putting this together. It's a great piece of software! ❤️

I've been using warp-docker for a while now in tunnel_only mode and it's all been great until a couple of days ago when connections relying on this proxy started failing.

I started looking into this and the first thing I saw was this error in the warp-docker logs, from warp-svc:

ERROR main_loop: warp::warp_service: Connection error error=IoError(AnnotatedIoError { err: Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" }, context: "open tun" })

I thought initially this was a permissions issue trying to interact with /dev/net/tun, so I went and had a look: crw------- 1 root root 10, 200 Dec 7 16:53 /dev/net/tun

Looked fine. The warp user is a sudoer, but for sanity's sake I chown'd /dev/net/tun to warp and even chmod'd it to 666. Same issue.

I tried a bunch of other combinations of settings and docker configurations (including using MASQUE and WG for the tunnel, and different modes too), but I wasn't able to resolve the issue or find a root cause.

For context, here's my warp-docker Docker Compose definition:

  warproxy:
    image: "caomingjun/warp"
    container_name: "warproxy"
    init: true
    cap_add:
      - "NET_ADMIN"
    environment:
      GOST_ARGS: "-L :42069"
    networks:
      - "warproxy_nw"
    sysctls:
      - "net.ipv6.conf.all.disable_ipv6=0"
      - "net.ipv4.conf.all.src_valid_mark=1"
    volumes:
      - "./config/warproxy/settings.json:/var/lib/cloudflare-warp/settings.json"
    tmpfs:
      - "/var/lib/cloudflare-warp:mode=770,size=16M,uid=0,gid=0"
    restart: "unless-stopped"

I even went back in warp-docker versions, but now the versions that used to work stopped to do so with the same error. Clearly, this was not an issue with any recent updates to this package.

I then went into trying to figure out what had changed on my system recently and the one thing I remembered was that apt had upgraded the containerd.io package on my Ubuntu Server 24.10 from 1.7.23-1 to 1.7.24-1.

I downgraded this package back to 1.7.23-1 and voila, problem solved.

I did not get to the root cause of this issue - I don't know if it's an incompatibility with warp-docker and this new version of containerd.io or with Cloudflare WARP itself.

I checked the changes that happened from containerd 1.7.23 to 1.7.24, but I see nothing that would immediately raise any flags. I did notice that runc and a few other dependencies got updated, so maybe more digging

--

I suspect no other issues like this have cropped up here given that this requires an update action for something that was released pretty recently. Perhaps we'll see more people mention this later as they upgrade their servers.

I am on containerd.io version 1.7.22, I updated to caomingjun/warp:latest and warp refused to start w/out the devices: /dev/net/tun in compose.

Additionally - after adding device to my compose, nothing would connect through WARP.

After rolling back to caomingjun/warp:2024.11.309.0-2.12.0-d9881567a57b23bf435a15ae3738ca2854e8d654, everything works fine.

[cmj2002]

after adding device to my compose, nothing would connect through WARP.

Are you using swarm?

What's the log of "after adding device to my compose, nothing would connect through WARP"?

[mafosa]

Hi there,
running Debian 12, with containerd.io v1.7.24-1 docker can access /dev/net/tun in standalone mode with these options in compose file: privileged: true and cap_add: NET_ADMIN. No need to set devices: /dev/net/tun.
But when running within a stack in docker swarm, I must downgrade to container.io v1.7.23-1
No way in v1.7.24-1 so I cant upgrade the whole system.

[wanegrow]

after adding device to my compose, nothing would connect through WARP.

Are you using swarm?

What's the log of "after adding device to my compose, nothing would connect through WARP"?

Ok - so first issue - trying to run WARP compose.yml w/out devices: -/dev/net/tun gives this error in WARP
CRITIC: /dev/net/tun not pass, check https://github.com/cmj2002/warp-docker/blob/main/docs/tun-not-permitted.md for more information

containerd --version = 1.7.12

add in devices variable and WARP runs - no problem. This is w/out trying to run any traffic through WARP. Just simply trying to get WARP:latest to run w/out devices variable on containerd version 1.7.12

[cmj2002]

Hi there, running Debian 12, with containerd.io v1.7.24-1 docker can access /dev/net/tun in standalone mode with these options in compose file: privileged: true and cap_add: NET_ADMIN. No need to set devices: /dev/net/tun. But when running within a stack in docker swarm, I must downgrade to container.io v1.7.23-1 No way in v1.7.24-1 so I cant upgrade the whole system.

That's an existing compatibility issue of swarm and containerd. I can't solve it in image or container level.

[cmj2002]

after adding device to my compose, nothing would connect through WARP.

Are you using swarm?
What's the log of "after adding device to my compose, nothing would connect through WARP"?

Ok - so first issue - trying to run WARP compose.yml w/out devices: -/dev/net/tun gives this error in WARP CRITIC: /dev/net/tun not pass, check https://github.com/cmj2002/warp-docker/blob/main/docs/tun-not-permitted.md for more information

containerd --version = 1.7.12

add in devices variable and WARP runs - no problem. This is w/out trying to run any traffic through WARP. Just simply trying to get WARP:latest to run w/out devices variable on containerd version 1.7.12

But why there is no traffic through WARP when device is passed? There must be something wrong and it must be displayed in the log. I need to see the log to solve your problem.

[cmj2002]

For everyone: it seems that containerd's removal of TUN device's rule from default ruleset has cause a hell lot of troubles. I will try to use device_cgroup_rules to overwrite the default ruleset next week.

[cmj2002]

New fix is built and pushed. TUN device is no longer need to be passed, but device_cgroup_rules is needed. Read https://github.com/cmj2002/warp-docker/blob/main/docs/tun-not-permitted.md for instructions.

The new solution should also be effective for Docker Swarm.