Skip to content

馃悰 Cloudflared never fallbacks to HTTP2聽#749

Closed

Description

@romeroyonatan

Describe the bug
Cloudflared always tries to connect using QUIC when protocol is auto. In the previous version (2022.8.2) it displays a warning message and connect to HTTP2.

To Reproduce
Steps to reproduce the behavior:

  1. Block UDP outgoing connections
  2. Connect the tunnel
  3. See error RR Serve tunnel error error="DialContext error: dial tcp 198.41.x.x:7844: i/o timeout" connIndex=0 ip=198.41.x.x

If it's an issue with Cloudflare Tunnel:
4. Tunnel ID :
5. cloudflared config:

Expected behavior
It should connect using HTTP2 when --protocol is auto and QUIC fails.

Environment and versions

  • OS: Ubuntu 20.04
  • Architecture: x86_64
  • Version: 2022.8.4

Logs and errors

Sep 02 06:29:25 cloudflared[157543]: 2022-09-02T06:29:25Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.x.x:7844: i/o timeout" connIndex=0 ip=198.41.x.x
Sep 02 06:29:25 cloudflared[157543]: 2022-09-02T06:29:25Z INF Retrying connection in up to 1m4s seconds connIndex=0 ip=198.41.200.43
Sep 02 06:29:42 cloudflared[157543]: 2022-09-02T06:29:42Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.x.x:7844: i/o timeout" connIndex=0 ip=198.41.x.x

Additional context
I think the regression was introduced here e3390fc

Activity

  1. changed the title [-]馃悰 Cloudflared never fallback to HTTP2[/-] [+]馃悰 Cloudflared never fallbacks to HTTP2[/+] on Sep 5, 2022
  2. nmldiegues commented on Sep 5, 2022

    @nmldiegues
    Contributor

    Hello,

    Thanks for your report.
    I've tried to reproduce and was unable to:

    2022-09-05T14:50:41Z INF Version 2022.8.4
    2022-09-05T14:50:41Z INF Initial protocol quic
    2022-09-05T14:50:46Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 ip=127.0.0.1
    2022-09-05T14:50:46Z INF Retrying connection in up to 2s seconds connIndex=0 ip=127.0.0.1
    2022-09-05T14:50:46Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
    If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not work unless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0 ip=127.0.0.1
    2022-09-05T14:50:46Z INF Switching to fallback protocol http2 connIndex=0 ip=127.0.0.1
    2022-09-05T14:50:46Z INF Connection 91787203-177d-4973-aec6-e61b5939deb8 registered connIndex=0 ip=127.0.0.1 location=DEV1
    

    I think the difference is that I had a no recent network activity whereas you had i/o timeout.

    Maybe this was working before for you, but it was not "on purpose", and this was buggy all along. It just happened that fixing e3390fc may have caused the bug to now trigger for your case.

    Can you point me as to exactly how you are setting up your scenario, step by step, so that I can verify what's going on?

  3. romeroyonatan commented on Sep 6, 2022

    @romeroyonatan
    Author

    Yes,

    • Sep 1st.: We created a VM with cloudflared version 2022.8.2. The connection was established successfully.
    • Sep 2nd: the auto-updater update cloudflared to 2022.8.4 and it cannot establish a connection again
    • Sep 5th: We found the issue, we force the protocol using --protocol http2. The connection was established again, but there are a lot of warning messages in the logs

    cloudflared.log
    cloudflared-http2-forced.log

  4. nmldiegues commented on Sep 6, 2022

    @nmldiegues
    Contributor

    Thank you for providing the logs.

    Those logs show that:

    • in "cloudflared.log", when running with "auto", cloudflared gives up on "quic" and fallsback to "http2", which also fails to connect to Cloudflare edge
    • in "cloudflared-http2-forced.log", when running with "http2", cloudflared also fails to connect to Cloudflare edge

    The log that you showed in the original comment of this thread:

    Sep 02 06:29:25 cloudflared[157543]: 2022-09-02T06:29:25Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.x.x:7844: i/o timeout" connIndex=0 ip=198.41.x.x
    Sep 02 06:29:25 cloudflared[157543]: 2022-09-02T06:29:25Z INF Retrying connection in up to 1m4s seconds connIndex=0 ip=198.41.200.43
    Sep 02 06:29:42 cloudflared[157543]: 2022-09-02T06:29:42Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.x.x:7844: i/o timeout" connIndex=0 ip=198.41.x.x
    

    **is telling you that your cloudflared tunnel process is unable to dial to Cloudflare edge port 7844 on both UDP and TCP **, which is a requirement as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/do-more-with-tunnels/ports-and-ips/

    So the title of the issue is wrong: cloudflared is indeed falling back to HTTP2, which is coherent with my tests above, and your problem is actually that your machine has no connectivity to Cloudflare's IPs on port 7844 on neither TCP (http2) nor UDP (quic)

  5. romeroyonatan commented on Sep 6, 2022

    @romeroyonatan
    Author

    You are right, it fallbacks to HTTP2. I going to check the network rules and close the issue.

    Thanks @nmldiegues

  6. romeroyonatan commented on Sep 6, 2022

    @romeroyonatan
    Author

    You were right, we block the IPs defined in region1.v2.argotunnel.com and region2.v2.argotunnel.com

    I want to add more information if somebody else are facing the same issue:

    We allowed the TCP traffic to the IPs defined in region1.argotunnel.com and region2.argotunnel.com, however in 2022.7.0 (c7a6304) the edge discovery algorithm changed and now it looks for IPs in region1.v2.argotunnel.com and region2.v2.argotunnel.com (Note that there are a v2)

    These fqdn add more IPs to allow in the firewalls.

    That's all,
    Thank you again @nmldiegues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions