Repository navigation
Lane C S3 follow-ups after #77: backfill rejections, best-effort backfill, legal holds, 12-month monthly locks (D15) - #93
Merged
Conversation
prune_s3 took the string max of "legal-hold" and ISO dates for locked_until, so a single legal hold hid the latest retain-until date. locked_until now reports the latest date only, and legal_holds counts the refused versions held by a legal hold. The _FakeS3 test double gains legal holds, per-object retention with put_object_retention, ETags, get_object read counts and per-operation error injection for the follow-up tests. Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>
…fill fail a copy The verified-marker backfill silently skipped a candidate whose checksum failed, and with no store checksum read that copy back again every night. Only ClientError was caught, so a BotoCoreError (ReadTimeoutError, ResponseStreamingError) during the read-back escaped after the new set was already copied, and the CLI never moved the off-site gauge. - Each candidate is checked by HEAD first (presence, size, the store's checksum), so a missing or truncated copy is rejected without a download. Only then are objects without a usable checksum read back. - A read-back rejection is remembered beside the root set (sets/<id>.offsite-rejected, keyed by destination and the objects' ETags), so the same objects are not downloaded again; a copy uploaded again is checked afresh. retention.prune removes the record with the set. - Rejections are reported in backfill_rejected (id and reason). - The backfill is best effort: network and permission errors are reported in backfill_errors and retried next run; the copy, its marker and the gauge are unaffected. Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>
…(D15) D15 is S3 Object Lock in GOVERNANCE mode with retention matching 30 daily + 12 monthly sets, but the bucket default (30 days) locked the monthly sets only for the daily window. With Object Lock on the bucket, prune --destination s3:// now extends the GOVERNANCE retention of every object of each set retention keeps as a monthly set (files, manifest, verified marker) with put_object_retention. The date (monthly_lock_until) is the earliest day the policy could expire that set: the first day of the month `monthly` months later plus the daily window, so the lock never outlives the retention and prune can always delete the set once the policy lets go. Retention is never shortened, a COMPLIANCE lock is left alone, and the bypass permission is never used. A denied extension is reported in errors and fails the run. retention.retained_monthly exposes which kept sets only the monthly rule keeps. Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>
Runbook: the bucket default covers the daily window and prune extends the monthly sets' GOVERNANCE retention; s3:PutObjectRetention joins the prune permission table (s3:BypassGovernanceRetention stays "never"); locked_until and legal_holds are reported apart; the S3 backfill reports backfill_rejected / backfill_errors and never fails the copy. DR objectives and ADR 0035: drop the "monthly sets locked only for the default period" limit. Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>
A root set whose manifest cannot be read (BackupInvalid, OSError) is now reported in backfill_errors like a network error, so the copy and the off-site gauge are unaffected. Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>
monthly_lock_until assumes a set every day. When daily backups were missed, a monthly set stays kept past that date, and a set never extended before (first run after deploy, a late change of the month's representative, an earlier denied extension) got put_object_retention with a past date. Real S3 rejects that with InvalidArgument, so prune exited 1 every night until the set expired. Prune now leaves such a set's lock alone and counts it in retention_past_due: a lock then would outlive the retention. The _FakeS3 rejects a past RetainUntilDate with InvalidArgument as S3 does. Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>
Owner
Author
|
Integrator merge evidence (Lane C S3 follow-ups after #77; D15 monthly locks), head 976dc9b:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-ups the integrator listed when reviewing #77 (HOM-212/213/214), plus the AppVision D15 limit on monthly locks. Base:
mainafter #77 (935c41c), now cb7c9e2.Changes (
Homayar/backend/backup/replicate.py,retention.py)replicatenow lists them asbackfill_rejected: [{backup_id, reason}]. Each candidate is checked by HEAD first (presence, size, the store's checksum), so a missing or truncated copy is rejected without a download. Objects with no usable store checksum are read back once. A read-back rejection is remembered insets/<id>.offsite-rejected, keyed by destination and the objects' ETags, so later nights report it without downloading again. A copy uploaded again (new ETags) is checked afresh.retention.prunedeletes the record together with its set.backfill_errorsand is retried on the next run. The copy, its marker and the CLI'soffsitegauge are unaffected (cmd_replicateexits 0)._error_codealso handles errors whoseresponseis None.locked_untilis the latest retain-until date (compared as datetimes).legal_holdscounts the refused versions held by a legal hold. Before this, the string max of"legal-hold"and ISO dates could hide the date.prune --destination s3://…extends the GOVERNANCE retention of every object of each set thatretentionkeeps as a monthly set: its files, manifest and verified marker. It usesput_object_retention;retention.retained_monthlyis new.monthly_lock_until): the first day of month M +monthly, plusdailydays. This is the earliest day the policy could expire the set, so the lock never outlives the retention and ends at most a day before it. A property test checks this againstretainedfor 4 daily/monthly settings.retention_kept).s3:BypassGovernanceRetentionis never used.errorsand fails the run.monthly_locked_until,retention_extendedandretention_kept.Docs
docs/operations/backup-restore.md:s3:PutObjectRetentionadded to the prune row of the permission table; Bypass stays "never".legal_holds,backfill_rejectedandbackfill_errorsare documented.docs/operations/disaster-recovery-objectives.mdand ADR 0035: the same limit is removed.Tests (
Homayar/backend/tests/test_backup_format.py, in-memory_FakeS3)_FakeS3gains:held)put_object_retention, which refuses shortening as S3 does without the bypassget_objectread countsraising)New tests, each failing on the pre-change code:
test_s3_prune_reports_legal_holds_apart_from_retention_dates(KeyErrorlegal_holds)test_a_rejected_backfill_candidate_is_reported_and_not_downloaded_every_night(KeyErrorbackfill_rejected)test_a_network_error_in_the_backfill_never_fails_the_copy_or_the_gauge: ReadTimeoutError escapes on the old code; the unreadable-root-set part makes the CLI exit 1 before the last committest_s3_prune_locks_monthly_sets_for_their_whole_retention_never_shorter(ImportError)test_a_monthly_lock_never_outlives_the_retention_that_keeps_the_set×4 (ImportError)Run:
ruff check backend/backup backend/tests/test_backup_format.py: cleanruff format --check: cleanpytest backend/tests/test_backup_format.py: 65 passedtest_backup_restore.py: 13 skipped (needs PostgreSQL);retained()output is unchanged by constructionNot touched: ci.yml, .env.example, migrations, saas_models.py, rls.py, config.py. Not exercised against a real S3/MinIO with Object Lock.
🤖 Generated with Claude Code