Upload extremely slow and no debug information for upload

[agowa]

I now gave utahfs another shot and I noticed the following two drawbacks:

1. For some reason it is very slow when uploading a 129MB file (it took over 5 hours for the first 70MB) to the s3 bucket at Backblaze (I'm using the generic s3 api not the specific backblaze one).
2. Even when utahfs-client is started with -v it doesn't show any information about the upload to the backend server.

The VPS is over at Hetzner and the S3 is at Backblaze.
I've started it with utahfs-client -cfg ./utahfs.yaml -mount ./utahfs -v and this configuration file (credentials changed)

storage-provider:
  s3-app-id: 2ce35e922f464cee961f17636c
  s3-app-key: f21ecb9b119a4437ac236478a7c80b02
  s3-bucket: c026f6e2-a8e6-40b7-aa62-8e58a2b112aa
  s3-url: https://s3.us-west-002.backblazeb2.com
  s3-region: us-west-002
keep-metadata: true
password: ioN8Dgr7WtDuF8g5efpWWBpcOHBWOjGsVzKTDf4B0abognzBp2xTEM2aBbbEYqFqMnhqxk7zrQK8E7rBky1BhKz1BymvIXgzMY1AE4X3YpUihc6SUeF6OEv3

Edit: Now additional two hours later it finally finished uploading the 129MB file.

[Bren2010]

There's a prometheus metrics server at localhost:3001 (configured with the --metrics-addr flag on utahfs-client) and that will tell you how quickly the upload is going and the WAL size, among other things. Setting up Grafana to track the metrics over time is pretty Fun and Easy™.

But you're also right that Backblaze can be slow. You can speed it up with the wal-parallelism setting from the Advanced Configuration guide. I think I've used 4 in the past

[agowa]

I installed prometheus and grafana now, but I don't know what I should be looking for.
I found these two that look related, but I'm not sure.
I also now added the wal-parallelism: 4 configuration as you've recommended.
It still is extremely slow that doesn't look normal at all.

[Bren2010]

The graphs definitely look how I'd expect in general. The first one shows active writes to S3, and the second one shows the WAL decreasing.

I'll reopen this and find time to look into the performance. There could be an issue with the S3 driver, or maybe Backblaze S3 is just slow