File transfer limit

[ItsSK]

I wanted to replace my Cloudflare Tunnel with Pangolin since Cloudflare has a 100MB limit. I noticed that Pangolin has a similar limit at around 370MB, is it possible to remove the limit?

[oschwartz10612]

Hi,

I am not sure what limit you are referring to? We should have no baked in data transfer limits.

What is happening when you get to 370? What service are you proxying that fails? Nextcloud?

[ItsSK]

Heyy, I did some testing with multiple docker applications, the test was preformed on the same application with Pangolin & NginxProxyManager being the only difference, Just sharing my findings

File: 1.16GB GoPro video
Pangolin:
Nextcloud: Uploaded successfully and playback works
Immich: fails to upload (Error in Immich log: [39m �[33mRequest error while uploading file, cleaning up�[39m)
Synology Photos: says uploaded but file is nowhere to be found.

Nginx Proxy Manager:
Nextcloud: Uploaded successfully and playback works
Immich: Uploaded successfully and playback works
Synology Photos: Uploaded successfully and playback works

[Sinofage]

Heyy, I did some testing with multiple docker applications, the test was preformed on the same application with Pangolin & NginxProxyManager being the only difference, Just sharing my findings

File: 1.16GB GoPro video Pangolin: Nextcloud: Uploaded successfully and playback works Immich: fails to upload (Error in Immich log: [39m �[33mRequest error while uploading file, cleaning up�[39m) Synology: Photos: says uploaded but file is nowhere to be found.

Nginx Proxy Manager: Nextcloud: Uploaded successfully and playback works Immich: Uploaded successfully and playback works Synology: Uploaded successfully and playback works

I had issue with synology photo as well, tried to upload new pic failed half way.

[oschwartz10612]

This is all very interesting thanks for the feedback. I just want to confirm: are you running the latest version of gerbil and newt? Should be beta.3 and beta.4 respectively. I fixed many issues around MTU, MSS, and the TCP proxy.

[ItsSK]

For gerbil I'm running beta.3 and for newt all I show is fosrl/newt:latest

[Sinofage]

I had issue with synology photo as well, tried to upload new pic failed half way.

did try again today, it seemed to work fine as long long the speed bandwidth is above 100kb as sec( was testing yesterday on 4g phone connection)

[oschwartz10612]

For gerbil I'm running beta.3 and for newt all I show is fosrl/newt:latest

Just to be 100% sure have you docker pulled the latest container if you are using the latest tag? Not sure when you set it up...

[dhanadhan]

I am also experiencing the same issue uploading bigger files/videos to Immich. Running all containers with the latest tag.

[oschwartz10612]

One thing you could try but I would not expect to instantly solve the issue is the MTU. I set it pretty low by default on the latest builds at 1280 to try to support all kinds of networks. You can use the --mtu or the MTU env var to set it higher to something like 1420 on BOTH gerbil and newt and see if it helps.

At the least it should speed it up a bit.

[dhanadhan]

One thing you could try but I would not expect to instantly solve the issue is the MTU. I set it pretty low by default on the latest builds at 1280 to try to support all kinds of networks. You can use the --mtu or the MTU env var to set it higher to something like 1420 on BOTH gerbil and newt and see if it helps.

At the least it should speed it up a bit.

Just tried with MTU 1420 and 1400 set on gerbil and newt, nothing changed. Upload still failed after about 370 MB.

[oschwartz10612]

I am currently investigating if we need some Traefik config to resolve this. I am looking at a comment in this thread: https://community.traefik.io/t/increase-maximum-body-size-without-buffering/15928/6

I will do some experimentation on my side, but in the meantime could anyone here try to reproduce and see if there is any output in the Traefik logs? You might want to turn on DEBUG for the log level in traefik_config.yml

[dhanadhan]

I am currently investigating if we need some Traefik config to resolve this. I am looking at a comment in this thread: https://community.traefik.io/t/increase-maximum-body-size-without-buffering/15928/6

I will do some experimentation on my side, but in the meantime could anyone here try to reproduce and see if there is any output in the Traefik logs? You might want to turn on DEBUG for the log level in traefik_config.yml

Not sure if I am reading the right logs. I changed traefik config to add DEBUG under logs. Then watched for traefik container logs while uploading. This is what it showed when it failed to upload.

DBG github.com/traefik/traefik/v3/pkg/server/service/proxy.go:100 > 502 Bad Gateway error="readfrom tcp 100.89.128.1:56390->100.89.128.4:47672: i/o timeout

[dhanadhan]

Made the change mentioned in the above link, and upload was Successful!

transport:
  respondingTimeouts:
    readTimeout: 30m

[oschwartz10612]

@dhanadhan Fantastic!

IDK if anyone else can confirm if this helps them. I will put this into the Traefik config for the next release.

[ItsSK]

Made the change mentioned in the above link, and upload was Successful!

websecure: address: ":443" transport: respondingTimeouts: readTimeout: 30m

What was the exact change?