Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Backblaze B2 as remote provider #127

Merged
merged 2 commits into from
Sep 5, 2024
Merged

Conversation

dpunkturban
Copy link

@dpunkturban dpunkturban commented Aug 5, 2024

Hi,

We use pg_back to back up and restore our databases to Backblaze. Currently we are using the S3 remote provider, which does not work very well with Backblaze, because sometimes backups randomly fail due to multipart upload failures. This is not a problem with pg_back, it is a problem with Backblaze (we find other tools on the internet that have exactly the same problem, such as wal-g/wal-g#1741 (comment)).

To avoid these problems and to speed up backups to Backblaze, I implemented the Backblaze native protocol B2 for pg_back, which is insanely faster than the S3 implementation and works out of the box.

Why Backblaze?

Backblaze is one of the largest blob storage backup providers, so I think it's very useful to support them. Other backup tools such as Restic also support the B2 protocol.

Todos:

  • Add Readme
  • Add config to pg_back.conf
  • Check naming consistency

@dpunkturban
Copy link
Author

Hi @orgrim,

Is Backblaze B2 remote provider something you can imagine merging into the pg_back main? If yes, I would work on the open to-do points and then remove the PR draft.

Thanks :)

@orgrim
Copy link
Owner

orgrim commented Aug 6, 2024

Hi, I'm currently afk for some days, I'll review the PR as soon as I can.

Thanks for your patience

@dpunkturban
Copy link
Author

Hi, I'm currently afk for some days, I'll review the PR as soon as I can.

Thanks for your patience

Hi @orgrim, no prob, I will refactor the code in the next few days and work on the open todos. I will remove the draft when I'm finished :)

@dpunkturban dpunkturban marked this pull request as ready for review August 11, 2024 15:28
@dpunkturban dpunkturban force-pushed the implement_b2 branch 4 times, most recently from d4655bc to 7ba4522 Compare August 11, 2024 15:52
@orgrim
Copy link
Owner

orgrim commented Aug 16, 2024

Hi, so the tests run fine, I could run the docker tests locally. The docker tests are failing because github made a change on
actions in actions/runner-images#9692

Copy link
Owner

@orgrim orgrim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR, the code works as expected, I could test it with a B2 bucket.

If the region is not used by B2, the --b2-region should be dropped. The --b2-endpoint is not used in the code by it seems good to allow to configure the endpoint

upload.go Outdated Show resolved Hide resolved
upload.go Outdated Show resolved Hide resolved
upload.go Outdated Show resolved Hide resolved
@dpunkturban dpunkturban force-pushed the implement_b2 branch 3 times, most recently from c2d0695 to 4da6474 Compare August 18, 2024 09:49
Signed-off-by: Dennis Urban (github@dennisurban.de)
@dpunkturban
Copy link
Author

Hey @orgrim, is there anything else needed to get this merged? thx

@orgrim orgrim merged commit ebbcdfb into orgrim:master Sep 5, 2024
4 of 9 checks passed
@orgrim
Copy link
Owner

orgrim commented Sep 5, 2024

Hey @orgrim, is there anything else needed to get this merged? thx

Sorry for the delay, the PR is merged. Thanks a lot for this contribution!

@dpunkturban
Copy link
Author

Hi @orgrim ,thanks for merging. Is it possible to create a new Release with the changes? :)

@orgrim
Copy link
Owner

orgrim commented Sep 6, 2024

Hi, yes but I need to have a look at the CI action that fails first, I hope to do that next week

@orgrim
Copy link
Owner

orgrim commented Sep 7, 2024

Hi @dpunkturban , release 2.5.0 is out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants