fix: serve V1 ListObjects via the backend's V2 API (Hetzner rejects V1)#92
Merged
Merged
Conversation
Hetzner Object Storage (and other modern S3 backends) only implement
ListObjectsV2 and reject the legacy V1 ListObjects with HTTP 400. That
broke every V1 client routed through the proxy:
- scylla-manager agent (bundled rclone 1.51.0 lists with V1) -> all
Scylla backups failed at the location-check / list step
- barman-cloud-backup-delete (V1 ListObjects) -> CNPG retention
enforcement failed (BadRequest on ListObjects), so Postgres backups
completed but old ones were never pruned
handle_list_objects_v1 now calls the backend's list_objects_v2 instead of
list_objects (V1). V1 marker pagination is stateless (marker == last key
returned), which maps exactly onto V2 StartAfter, so the translation is
lossless for the recursive listings these clients use. NextMarker is
synthesized from the largest raw backend key when truncated; keys are
still URL-encoded under encoding-type=url (V1 markers are keys, unlike the
opaque V2 continuation token).
This was referenced Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Hetzner Object Storage only implements ListObjectsV2 — it rejects the legacy V1
ListObjectswith HTTP 400 BadRequest. The proxy forwarded V1 client requests to the backend as V1, so every V1 client broke:provider: Minio) → all Scylla backups failed at the location-check / list step:ListObjects) → CNPG retention enforcement failed even after the V2 continuation-token fix landed:HEAD/PUTand all V2 (list-type=2) clients (botocore, aws-sdk-go, aws-sdk-java) were unaffected.Fix
handle_list_objects_v1now serves the client's V1 request by calling the backend'slist_objects_v2instead oflist_objects(V1). V1 marker pagination is stateless (marker == last key returned), which maps exactly onto V2StartAfter, so the translation is lossless for the recursive listings these clients use:marker→ backend V2StartAfterNextMarkersynthesized from the largest raw backend key/prefix when truncated (raw, not the internal-key-filtered list, so an all-internal page still advances)encoding-type=url(V1 markers are keys — unlike the opaque V2 continuation token in fix: don't URL-encode V2 continuation tokens under encoding-type=url #91)ponytail note in-code: a single delimiter-level prefix with >max_keys distinct sub-prefixes can re-emit one boundary prefix across pages (harmless/idempotent for read-only catalog walks); upgrade path is a redis-backed marker→continuation-token map.
Test
tests/unit/test_list_objects_v1_via_v2.py: proves the handler calls the backend V2 API (the fake's V1 method asserts if hit), mapsmarker→StartAfter, filters internal keys, and synthesizesNextMarkerfrom the largest raw key when truncated. Full unit suite green (542 passed); ruff check + format clean.