Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot connect to gost because Warp is grabbing incoming traffic #23

Closed
ostrolucky opened this issue Jul 12, 2024 · 24 comments
Closed

Cannot connect to gost because Warp is grabbing incoming traffic #23

ostrolucky opened this issue Jul 12, 2024 · 24 comments

Comments

@ostrolucky
Copy link

Inside container

warp@1db191c9c609:/$ curl --socks5-hostname 127.0.0.1:1080 https://cloudflare.com/cdn-cgi/trace
fl=636f241
h=cloudflare.com
ip=8.29.109.153
ts=1720827095.986
visit_scheme=https
uag=curl/7.81.0
colo=FRA
sliver=none
http=http/2
loc=DE
tls=TLSv1.3
sni=plaintext
warp=on
gateway=on
rbi=off
kex=X25519

Outside container

❯ curl --socks5-hostname 127.0.0.1:1080 https://cloudflare.com/cdn-cgi/trace -v
*   Trying 127.0.0.1:1080...
* Connected to 127.0.0.1 (127.0.0.1) port 1080
* Recv failure: Connection reset by peer
* SOCKS: Failed receiving initial SOCKS5 response: Failure when receiving data from the peer
* Closing connection
curl: (97) Recv failure: Connection reset by pee

Once I kill warp-svc process inside container, then previous command works (but then it doesn't go through cloudfare network of course).

Do you have any advice what to do in this case?

@cmj2002
Copy link
Owner

cmj2002 commented Jul 13, 2024

The command you are using in the container is going through gost (you specified --socks5-hostname). Therefore, gost is working, and traffic is being transmitted through warp. However, outside the container (assuming you are running the command on the host machine), it cannot connect to gost, so the issue lies in the connection between the container and the host machine. Have you made any modifications to docker-compose.yml (such as modifying or deleting mapped ports), or do you have special firewall or iptables settings on your machine that prevent Docker port mapping from functioning properly?

@ostrolucky
Copy link
Author

ostrolucky commented Jul 13, 2024

Like I said when I kill warp-svc process inside container, connection from outside container starts working, so it must be caused by warp catching (and dropping or something like that) this income traffic from whatever else that's not coming through lo interface. I've spun up separate service in same docker-compose network (so container-container connection) and I cannot connect from over there either unless I stop warp.

@cmj2002
Copy link
Owner

cmj2002 commented Jul 14, 2024

I can't think of any reason that caused it. I also run a similar setup myself, where another container accesses the warp inside the container, and it works well:

root@2c285c19ef85:/app# curl --socks5-hostname warp-socks:1080 https://cloudflare.com/cdn-cgi/trace
fl=650f66
h=cloudflare.com
ip=104.28.202.178
ts=1720924102.809
visit_scheme=https
uag=curl/7.88.1
colo=EWR
sliver=none
http=http/2
loc=US
tls=TLSv1.3
sni=plaintext
warp=plus
gateway=off
rbi=off
kex=X25519

Here is my configuration for your reference, hoping it will be helpful to you.

version: '3.9'

services:
    rsshub:
        image: diygod/rsshub:chromium-bundled
        restart: always
        container_name: rsshub
        environment:
            PROXY_URI: 'http://warp-socks:1080'
        depends_on:
            - warp-socks
        networks:
            - exposed
            - rsshub
    warp-socks:
        image: caomingjun/warp:latest
        container_name: warp-socks
        restart: always
        volumes:
            - /home/cmj/warp/var:/var/lib/cloudflare-warp
        cap_add:
            - NET_ADMIN
        sysctls:
            net.ipv6.conf.all.disable_ipv6: 0
            net.ipv4.conf.all.src_valid_mark: 1
        networks:
            - rsshub
        environment:
            - WARP_SLEEP=2
            - WARP_LICENSE_KEY=***

networks:
    rsshub:
        name: rsshub
    exposed:
        name: exposed
        external: true

where:

  • expose is the network where nginx is located
  • I use http because rsshub cannot use socks, but as shown in the above command line result, socks5 can be accessed within the container.

@vinothsubramanian
Copy link
Contributor

vinothsubramanian commented Aug 4, 2024

I too have same issue. The docker port 1080 is reachable only after disconnecting warp-cli. Any other specific settings we need to enable to make this work when warp=on

pi@pi5-2:~/infra $ curl --socks5-hostname 127.0.0.1:1080 https://cloudflare.com/cdn-cgi/trace -v
*   Trying 127.0.0.1:1080...
* Connected to 127.0.0.1 (127.0.0.1) port 1080 (#0)
* Recv failure: Connection reset by peer
* SOCKS4: Failed receiving initial SOCKS5 response: Failure when receiving data from the peer
* Closing connection 0
curl: (97) Recv failure: Connection reset by peer
pi@pi5-2:~/infra $ docker exec -it warp-socks bash
warp@b894cf47e024:/$ warp-cli disconnect
Success
warp@b894cf47e024:/$ exit
exit
pi@pi5-2:~/infra $ curl --socks5-hostname 127.0.0.1:1080 https://cloudflare.com/cdn-cgi/trace -v
*   Trying 127.0.0.1:1080...
* Connected to 127.0.0.1 (127.0.0.1) port 1080 (#0)
* SOCKS5 connect to cloudflare.com:443 (remotely resolved)
* SOCKS5 request granted.
* Connected to 127.0.0.1 (127.0.0.1) port 1080 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=cloudflare.com
*  start date: Jan 30 00:00:00 2024 GMT
*  expire date: Dec 31 23:59:59 2024 GMT
*  subjectAltName: host "cloudflare.com" matched cert's "cloudflare.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /cdn-cgi/trace]
* h2h3 [:scheme: https]
* h2h3 [:authority: cloudflare.com]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x5555c7da0a90)
> GET /cdn-cgi/trace HTTP/2
> Host: cloudflare.com
> user-agent: curl/7.88.1
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200
< date: Sun, 04 Aug 2024 01:57:51 GMT
< content-type: text/plain
< access-control-allow-origin: *
< server: cloudflare
< cf-ray: 8adaf062efc217ac-MAA
< x-frame-options: DENY
< x-content-type-options: nosniff
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< cache-control: no-cache
<
fl=59f167
h=cloudflare.com
ip=******
ts=1722736671.19
visit_scheme=https
uag=curl/7.88.1
colo=MAA
sliver=none
http=http/2
loc=IN
tls=TLSv1.3
sni=plaintext
warp=off
gateway=off
rbi=off
kex=X25519
* Connection #0 to host 127.0.0.1 left intact
pi@pi5-2:~/infra $

@cmj2002
Copy link
Owner

cmj2002 commented Aug 4, 2024

What's your docker engine version? Are you using arm device to run this? (For both @vinothsubramanian and @ostrolucky ) I try to reproduce the problem when it opened 3 weeks ago but failed, so I need more information.

@vinothsubramanian
Copy link
Contributor

I am running the docker on RaspberryPI 5 - arm64 architecture. Here are the version numbers.

pi@pi5-2:~ $ docker -v
Docker version 27.0.3, build 7d4bcd8
pi@pi5-2:~ $ uname -a
Linux pi5-2 6.6.20+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.20-1+rpt1 (2024-03-07) aarch64 GNU/Linux
pi@pi5-2:~ $ dpkg --print-architecture
arm64
pi@pi5-2:~ $

@ostrolucky
Copy link
Author

I'm also running it on arm64, it's macbook m1 with MacOS and I'm using OrbStack as docker engine

❯ docker version
Client:
 Version:           26.1.3
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        b72abbb
 Built:             Thu May 16 08:30:38 2024
 OS/Arch:           darwin/arm64
 Context:           orbstack

Server: Docker Engine - Community
 Engine:
  Version:          26.1.4
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       de5c9cf
  Built:            Wed Jun  5 11:29:18 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          v1.7.19
  GitCommit:        2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc:
  Version:          1.1.13
  GitCommit:        58aa9203c123022138b22cf96540c284876a7910
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

@cmj2002
Copy link
Owner

cmj2002 commented Aug 4, 2024

So there might be a problem in ARM image. I will look into it ASAP after my vacation.

@vinothsubramanian
Copy link
Contributor

@cmj2002 I am thinking that it could be related to configuration in cloudflare tunnel or policies.

The forwarding doesn't work only when we are connected to a team.

If we don't connect to a team, and connect to warp with default settings everything works fine.

@cmj2002
Copy link
Owner

cmj2002 commented Aug 4, 2024

I found this on stackoverflow: https://stackoverflow.com/questions/77053747/not-able-to-ssh-into-server-after-setting-up-cloudflare-warp

So maybe warp is grabbing the response packet of gost, and obviously it cannot reach the host by cloudflare exit node. Try add the host or the container ip which using warp to the exclude list may help?

I am not sure of this, grabbing a packet of an existing TCP connection which belongs to another network interface is uncommon.

@vinothsubramanian
Copy link
Contributor

@cmj2002 I am yet to try this solution. However I was able to overcome the problem by tweaking the entrypoint.sh to use mdm.xml as explained in the document https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/private-net/warp-connector/#4-install-a-warp-connector.

Now I have a working warp-to-warp connector by running warp-docker image.

I will raise a pull request shortly.

@vinothsubramanian
Copy link
Contributor

vinothsubramanian commented Aug 5, 2024

@ostrolucky Can you try the latest image with mdm.xml setup and see if you are able to make it to work

@ostrolucky
Copy link
Author

ostrolucky commented Aug 5, 2024

I can't really configure it according the guide, because I'm just an employee that is forced to use cloudfare zero trust. I don't have any admin rights in Cloudfare admin panel, which is what the linked guide for setting up warp connector seem to expect, so not sure how I can get values to put to mdm.xml file.

Anyways, since I've created this thread I've discovered https://github.com/rany2/warp.sh with which I can remove official cloudfare-warp application and instead use wireguard tunnel, where it's trivial to configure split tunnels even when official application doesn't allow me so.

@vinothsubramanian
Copy link
Contributor

@ostrolucky do you still want this defect to be opened or it can be closed

@ostrolucky
Copy link
Author

I think I would prefer to leave this opened until your fix is merged

@cmj2002
Copy link
Owner

cmj2002 commented Aug 12, 2024

@ostrolucky I successfully reproduced the issue. As a cross-check, please provide the results of running warp-cli tunnel dump and ip address inside the container when unable to connect.

@ostrolucky
Copy link
Author

warp-cli tunnel dump

Excluded: [
    224.0.0.251/32,
    192.168.33.0/24,
    52.112.0.0/14,
    2603:1027::/48,
    192.168.192.0/22,
    10.160.30.78/32,
    192.168.166.0/24,
    52.122.0.0/15,
    2603:1063::/38,
    2603:1057::/48,
    52.238.119.141/32,
    192.168.120.0/22,
    2603:1037::/48,
    10.25.0.0/22,
    2603:1047::/48,
    169.254.0.0/16,
    127.0.0.1/8,
    13.107.64.0/18,
    239.255.255.250/32,
    2620:1ec:6::/48,
    192.168.1.0/24,
    172.168.2.0/24,
    192.168.56.0/24,
    2620:1ec:40::/42,
    52.244.160.207/32,
    127.0.0.0/8,
    52.112.0.0/14,
    52.122.0.0/15,
    169.254.0.0/16,
    13.107.64.0/18,
    10.25.0.0/22,
    192.168.120.0/22,
    192.168.192.0/22,
    172.168.2.0/24,
    192.168.1.0/24,
    192.168.33.0/24,
    192.168.56.0/24,
    192.168.166.0/24,
    10.160.30.78/32,
    52.238.119.141/32,
    52.244.160.207/32,
    162.159.36.1/32,
    162.159.137.105/32,
    162.159.138.105/32,
    162.159.193.2/32,
    224.0.0.251/32,
    239.255.255.250/32,
    255.255.255.255/32,
    fe80::/10,
    2603:1063::/38,
    2620:1ec:40::/42,
    2603:1027::/48,
    2603:1037::/48,
    2603:1047::/48,
    2603:1057::/48,
    2620:1ec:6::/48,
    2606:4700:7::a29f:8969/128,
    2606:4700:7::a29f:8a69/128,
    2606:4700:4700::1111/128,
]

ip address

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: CloudflareWARP: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc mq state UNKNOWN group default qlen 500
    link/none 
    inet 172.16.0.2/32 scope global CloudflareWARP
       valid_lft forever preferred_lft forever
    inet6 2606:4700:110:85db:ce7d:c4bd:4b71:8878/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::5133:f231:2ef8:28fe/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
23: eth0@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:c0:a8:cf:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.207.2/24 brd 192.168.207.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:c0ff:fea8:cf02/64 scope link 
       valid_lft forever preferred_lft forever

@cmj2002
Copy link
Owner

cmj2002 commented Aug 12, 2024

From the information you provided, warp-docker use 192.168.207.2 (eth0@if24) to communicate with host, at which point the host's address is usually 192.168.207.1; both this subnet 192.168.207.0/24 and the host's address are not within the excluded IP addresses.

Since you do not have permission to modify the excluded address list, I suggest selecting one from the existing excluded CIDRs to use as the IP range for the docker network, and placing warp-docker under this docker network.

Edit: carefully choose the CIDR, it should be a private IP address and not used by your other devices.

@ostrolucky
Copy link
Author

Yep, that seemed to have fixed the issue! But I was thinking that container could adjust the routing table automatically after cloudfare-cli starts so that this cannot happen? Otherwise this can be quite brittle and manual.

@cmj2002
Copy link
Owner

cmj2002 commented Aug 12, 2024

Yep, that seemed to have fixed the issue! But I was thinking that container could adjust the routing table automatically after cloudfare-cli starts so that this cannot happen? Otherwise this can be quite brittle and manual.

Of course, I will try to automate this process in the next few days. However, the host IP may conflict with the IP addresses of services within the user's organization, making it unavailable; therefore, I will set it as an optional feature that is disabled by default.

@cmj2002
Copy link
Owner

cmj2002 commented Aug 14, 2024

Fixed in 5499865, you can follow the new guidelines in the document to resolve this issue.

@cmj2002
Copy link
Owner

cmj2002 commented Aug 15, 2024

I just discovered that the previously released version still attempts to add rule even when the rules already exist. It doesn't seem to cause any issues, but I still recommend updating to the newly built version.

@fireinice
Copy link

I still could not connect the gost exposed service from the host even with BETA_FIX_HOST_CONNECTIVITY=1
the container ip is 172.35.0.37
and the output of the dump is

Excluded: [
    10.0.0.0/8,
    100.64.0.0/10,
    169.254.0.0/16,
    172.16.0.0/12,
    192.0.0.0/24,
    192.168.0.0/16,
    224.0.0.0/24,
    240.0.0.0/4,
    239.255.255.250/32,
    255.255.255.255/32,
    fe80::/10,
    fd00::/8,
    ff01::/16,
    ff02::/16,
    ff03::/16,                                                                                                                                 
    ff04::/16,                                                                                                                                 
    ff05::/16,                                                                                                                                 
    fc00::/7,                                                                                                                                  
    17.249.0.0/16,                                                                                                                             
    17.252.0.0/16,                                                                                                                             
    17.57.144.0/22,                                                                                                                            
    17.188.128.0/18,                                                                                                                           
    17.188.20.0/23,                                                                                                                            
    2620:149:a44::/48,                                                                                                                         
    2403:300:a42::/48,                                                                                                                         
    2403:300:a51::/48,                                                                                                                         
    2a01:b740:a42::/48,                                                                                                                        
    240.0.0.0/4,                                                                                                                               
    10.0.0.0/8,                                                                                                                                
    100.64.0.0/10,                                                                                                                             
    172.16.0.0/12,                                                                                                                             
    17.249.0.0/16,                                                                                                                             
    17.252.0.0/16,                                                                                                                             
    169.254.0.0/16,                                                                                                                            
    192.168.0.0/16,                                                                                                                            
    17.188.128.0/18,                                                                                                                           
    17.57.144.0/22,                                                                                                                            
    17.188.20.0/23,                                                                                                                            
    192.0.0.0/24,                                                                                                                              
    224.0.0.0/24,                                                                                                                              
    104.16.80.230/32,                                                                                                                          
    104.17.143.163/32,                                                                                                                         
    162.159.192.7/32,                                                                                                                          
    239.255.255.250/32,                                                                                                                        
    fc00::/7,                                                                                                                                  
    fe80::/10,                                                                                                                                 
    ff02::/15,                                                                                                                                 
    ff04::/15,                                                                                                                                 
    ff01::/16,                                                                                                                                 
    2403:300:a42::/48,                                                                                                                         
    2403:300:a51::/48,                                                                                                                         
    2620:149:a44::/48,                                                                                                                         
    2a01:b740:a42::/48,                                                                                                                        
    2606:4700::6810:50e6/128,                                                                                                                  
    2606:4700::6811:8fa3/128,                                                                                                                  
]   

I think 172.35.0.37 is in the range of 172.16.0.0/12.

and I try to expose a shadowsocks service by gost:
ss+ohttp://aes-256-gcm:password@172.35.0.37:9876

any idea? thanks

@cmj2002
Copy link
Owner

cmj2002 commented Oct 1, 2024

I still could not connect the gost exposed service from the host even with BETA_FIX_HOST_CONNECTIVITY=1 the container ip is 172.35.0.37 and the output of the dump is

Excluded: [
    10.0.0.0/8,
    100.64.0.0/10,
    169.254.0.0/16,
    172.16.0.0/12,
    192.0.0.0/24,
    192.168.0.0/16,
    224.0.0.0/24,
    240.0.0.0/4,
    239.255.255.250/32,
    255.255.255.255/32,
    fe80::/10,
    fd00::/8,
    ff01::/16,
    ff02::/16,
    ff03::/16,                                                                                                                                 
    ff04::/16,                                                                                                                                 
    ff05::/16,                                                                                                                                 
    fc00::/7,                                                                                                                                  
    17.249.0.0/16,                                                                                                                             
    17.252.0.0/16,                                                                                                                             
    17.57.144.0/22,                                                                                                                            
    17.188.128.0/18,                                                                                                                           
    17.188.20.0/23,                                                                                                                            
    2620:149:a44::/48,                                                                                                                         
    2403:300:a42::/48,                                                                                                                         
    2403:300:a51::/48,                                                                                                                         
    2a01:b740:a42::/48,                                                                                                                        
    240.0.0.0/4,                                                                                                                               
    10.0.0.0/8,                                                                                                                                
    100.64.0.0/10,                                                                                                                             
    172.16.0.0/12,                                                                                                                             
    17.249.0.0/16,                                                                                                                             
    17.252.0.0/16,                                                                                                                             
    169.254.0.0/16,                                                                                                                            
    192.168.0.0/16,                                                                                                                            
    17.188.128.0/18,                                                                                                                           
    17.57.144.0/22,                                                                                                                            
    17.188.20.0/23,                                                                                                                            
    192.0.0.0/24,                                                                                                                              
    224.0.0.0/24,                                                                                                                              
    104.16.80.230/32,                                                                                                                          
    104.17.143.163/32,                                                                                                                         
    162.159.192.7/32,                                                                                                                          
    239.255.255.250/32,                                                                                                                        
    fc00::/7,                                                                                                                                  
    fe80::/10,                                                                                                                                 
    ff02::/15,                                                                                                                                 
    ff04::/15,                                                                                                                                 
    ff01::/16,                                                                                                                                 
    2403:300:a42::/48,                                                                                                                         
    2403:300:a51::/48,                                                                                                                         
    2620:149:a44::/48,                                                                                                                         
    2a01:b740:a42::/48,                                                                                                                        
    2606:4700::6810:50e6/128,                                                                                                                  
    2606:4700::6811:8fa3/128,                                                                                                                  
]   

I think 172.35.0.37 is in the range of 172.16.0.0/12.

and I try to expose a shadowsocks service by gost: ss+ohttp://aes-256-gcm:password@172.35.0.37:9876

any idea? thanks

Edit: I just noticed that you opened a new issue. I suggest we continue the discussion there.

You must have made a mistake. 172.35.0.37 does not belong to 172.16.0.0/12 (the subnet ends at 172.31.255.255). 172.35.0.37 is a public IP address, which appears to be a home user IP belonging to T-Mobile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants