slow PUT requests run into a 60s timeout

[butonic]

Describe the bug

A slow upload will run into a bad gateway error after 60s

Steps to reproduce

1. Try to upload a large (500mb) file slowly:

curl 'https://cloud.opencloud.test/remote.php/dav/spaces/ca679817-bf97-48cd-80ef-88940b807821$22b4f614-07e0-1040-941c-e96158f6afa5/tempfile4.bin' \
  -X PUT -v --limit-rate 100k \
  -H 'Content-Type: application/offset+octet-stream' \
  -H 'Authorization: Bearer eyJhbGciOiJS...' \
  -T tempfile.bin

2. see the server responding with a 502:

< HTTP/2 502 
< content-length: 11
< date: Wed, 13 Aug 2025 09:15:24 GMT
* HTTP error before end of send, stop sending
* abort upload after having sent 4520136 bytes
< 
* Connection #0 to host cloud.opencloud.test left intact
Bad Gateway%    

3. The log will contain sth like

{
    "level": "error",
    "service": "proxy",
    "remoteAddr": "10.42.0.1",
    "request-id": "",
    "proto": "HTTP/1.1",
    "method": "PUT",
    "path": "/remote.php/dav/spaces/ca679817-bf97-48cd-80ef-88940b807821$22b4f614-07e0-1040-941c-e96158f6afa5/tempfile4.bin",
    "query": "",
    "fragment": "",
    "error": "context canceled",
    "time": "2025-08-13T09:20:18Z",
    "line": "github.com/opencloud-eu/opencloud/services/proxy/pkg/proxy/proxy.go:52",
    "message": "error happened in MultiHostReverseProxy"
}
{
    "level": "error",
    "service": "ocdav",
    "name": "eu.opencloud.web.ocdav",
    "traceid": "70216c16179187472deb3495c17736bf",
    "request-id": "opencloud-api-6f4cb57679-vvl5q/nsvVXwrVn2-003486",
    "spaceid": "ca679817-bf97-48cd-80ef-88940b807821$22b4f614-07e0-1040-941c-e96158f6afa5",
    "path": "/tempfile4.bin",
    "error": "Put \"https://cloud.opencloud.test/data\": context canceled",
    "time": "2025-08-13T09:20:18Z",
    "message": "error doing PUT request to data service"
}
{
    "level": "info",
    "service": "proxy",
    "proto": "HTTP/1.1",
    "request-id": "opencloud-api-6f4cb57679-vvl5q/nsvVXwrVn2-003485",
    "traceid": "70216c16179187472deb3495c17736bf",
    "remote-addr": "10.42.0.1",
    "method": "PUT",
    "status": 502,
    "path": "/remote.php/dav/spaces/ca679817-bf97-48cd-80ef-88940b807821$22b4f614-07e0-1040-941c-e96158f6afa5/tempfile4.bin",
    "duration": 59999.685773,
    "bytes": 0,
    "time": "2025-08-13T09:20:18Z",
    "line": "github.com/opencloud-eu/opencloud/services/proxy/pkg/middleware/accesslog.go:34",
    "message": "access-log"
}
{
    "level": "error",
    "service": "proxy",
    "remoteAddr": "10.42.0.190",
    "request-id": "",
    "proto": "HTTP/1.1",
    "method": "PUT",
    "path": "/data",
    "query": "",
    "fragment": "",
    "error": "context canceled",
    "time": "2025-08-13T09:20:18Z",
    "line": "github.com/opencloud-eu/opencloud/services/proxy/pkg/proxy/proxy.go:52",
    "message": "error happened in MultiHostReverseProxy"
}
{
    "level": "info",
    "service": "proxy",
    "proto": "HTTP/1.1",
    "request-id": "opencloud-api-6f4cb57679-ps45c/A9BAoZUBtz-003461",
    "traceid": "70216c16179187472deb3495c17736bf",
    "remote-addr": "10.42.0.190",
    "method": "PUT",
    "status": 502,
    "path": "/data",
    "duration": 59927.421974,
    "bytes": 0,
    "time": "2025-08-13T09:20:18Z",
    "line": "github.com/opencloud-eu/opencloud/services/proxy/pkg/middleware/accesslog.go:34",
    "message": "access-log"
}
{
    "level": "error",
    "service": "frontend",
    "pkg": "rhttp",
    "traceid": "baf38577c88d9841690f6b033606d40f",
    "request-id": "opencloud-api-6f4cb57679-ps45c/A9BAoZUBtz-003462",
    "error": "Put \"http://localhost:9158/data/simple/5e6e91de-ae91-4e40-922c-36a6523b274d\": readfrom tcp 127.0.0.1:59772->127.0.0.1:9158: unexpected EOF",
    "time": "2025-08-13T09:20:18Z",
    "message": "error doing PUT request to data service"
}

Expected behavior

PUT should return a 201 Created

Actual behavior

The ocdav service ran into a context timeout while sending data to the storage users dataprovider

First reported by @TheOneRing when trying to upload with the iOS client. That one seems to not detect that it can do TUS.

[butonic]

ocdav.go initializes a client like this:

	s := &svc{
		c:             conf,
		webDavHandler: new(WebDavHandler),
		davHandler:    new(DavHandler),
		client: rhttp.GetHTTPClient(
			rhttp.Timeout(time.Duration(conf.Timeout*int64(time.Second))),
			rhttp.Insecure(conf.Insecure),
		),
		gatewaySelector:     selector,
		favoritesManager:    fm,
		LockSystem:          ls,
		userIdentifierCache: ttlcache.NewCache(),
		nameValidators:      ValidatorsFromConfig(conf),
		urlSigner:           signer,
	}

but OCDAV_GATEWAY_REQUEST_TIMEOUT defaults to 84300 seconds, a little less than 24h:

// DefaultConfig returns a basic default configuration
func DefaultConfig() *config.Config {
	return &config.Config{
                // [...]
		Insecure:          false,
		Timeout:           84300,
                // [...]
	}
}

The other service involved is the proxy ...

[butonic]

ok it is not our proxy ... but the messy way we send requests... It causes the request to go through the ingress again. In my case that is traefik in k3s, which has a default 60s timeout.

We can increase it there by updating the helm chart config by applying this:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
      - "--entryPoints.web.transport.respondingTimeouts.readTimeout=90s"
      - "--entryPoints.web.transport.respondingTimeouts.writeTimeout=90s"
      - "--entryPoints.web.transport.respondingTimeouts.idleTimeout=90s"
      - "--entryPoints.websecure.transport.respondingTimeouts.readTimeout=90s"
      - "--entryPoints.websecure.transport.respondingTimeouts.writeTimeout=90s"
      - "--entryPoints.websecure.transport.respondingTimeouts.idleTimeout=90s"

kudos to @FleetAdmiralButter https://www.puddl.net/increasing-traefik-ingress-controller-timeouts-k3s

Traefik will restart and then the request will timeout after 90s:

{
    "level": "error",
    "service": "proxy",
    "remoteAddr": "10.42.0.1",
    "request-id": "",
    "proto": "HTTP/1.1",
    "method": "PUT",
    "path": "/remote.php/dav/spaces/ca679817-bf97-48cd-80ef-88940b807821$22b4f614-07e0-1040-941c-e96158f6afa5/tempfile3.bin",
    "query": "",
    "fragment": "",
    "error": "context canceled",
    "time": "2025-08-13T10:43:23Z",
    "line": "github.com/opencloud-eu/opencloud/services/proxy/pkg/proxy/proxy.go:52",
    "message": "error happened in MultiHostReverseProxy"
}
{
    "level": "info",
    "service": "proxy",
    "proto": "HTTP/1.1",
    "request-id": "opencloud-api-6f4cb57679-vrgmc/yIM2WGXok9-000477",
    "traceid": "03fb9d79d16032cabfa8982880f7367b",
    "remote-addr": "10.42.0.1",
    "method": "PUT",
    "status": 502,
    "path": "/remote.php/dav/spaces/ca679817-bf97-48cd-80ef-88940b807821$22b4f614-07e0-1040-941c-e96158f6afa5/tempfile3.bin",
    "duration": 90000.182496,
    "bytes": 0,
    "time": "2025-08-13T10:43:23Z",
    "line": "github.com/opencloud-eu/opencloud/services/proxy/pkg/middleware/accesslog.go:34",
    "message": "access-log"
}
{
    "level": "error",
    "service": "ocdav",
    "name": "eu.opencloud.web.ocdav",
    "traceid": "03fb9d79d16032cabfa8982880f7367b",
    "request-id": "opencloud-api-6f4cb57679-vrgmc/yIM2WGXok9-000478",
    "spaceid": "ca679817-bf97-48cd-80ef-88940b807821$22b4f614-07e0-1040-941c-e96158f6afa5",
    "path": "/tempfile3.bin",
    "error": "Put \"https://cloud.opencloud.test/data\": context canceled",
    "time": "2025-08-13T10:43:23Z",
    "message": "error doing PUT request to data service"
}

This is related to owncloud/ocis#6296 and I tried to directly upload to the dataprovider in cs3org/reva#4065 and owncloud/ocis#6811 but that never got merged...

[butonic]

For our k3s deployment we are actually sending the request to the internal frontend service, not through the ingress again, but the timeout still applies and traefik kills the request. the reverse proxy detects that and logs the context timeout.

[butonic]

Well this only happens if you have your own traefik configuration. the docker compose full example uses:

      # change default timeouts for long-running requests
      # this is needed for webdav clients that do not support the TUS protocol
      - "--entryPoints.https.transport.respondingTimeouts.readTimeout=12h"
      - "--entryPoints.https.transport.respondingTimeouts.writeTimeout=12h"
      - "--entryPoints.https.transport.respondingTimeouts.idleTimeout=3m"

• document somewhere that this not only affects TUS connections ... but all connections. AFAICT even SSE