Skip to content

Lane C S3 follow-ups after #77: backfill rejections, best-effort backfill, legal holds, 12-month monthly locks (D15) - #93

Merged
Kohandezh merged 7 commits into
mainfrom
claude/lane-c-s3-followups
Oct 19, 2026
Merged

Kohandezh merged 7 commits into
mainfrom
claude/lane-c-s3-followups

Conversation

@Kohandezh

Copy link
Copy Markdown
Owner

Follow-ups the integrator listed when reviewing #77 (HOM-212/213/214), plus the AppVision D15 limit on monthly locks. Base: main after #77 (935c41c), now cb7c9e2.

Changes (Homayar/backend/backup/replicate.py, retention.py)

  1. Rejected backfill candidates are reported (L1). replicate now lists them as backfill_rejected: [{backup_id, reason}]. Each candidate is checked by HEAD first (presence, size, the store's checksum), so a missing or truncated copy is rejected without a download. Objects with no usable store checksum are read back once. A read-back rejection is remembered in sets/<id>.offsite-rejected, keyed by destination and the objects' ETags, so later nights report it without downloading again. A copy uploaded again (new ETags) is checked afresh. retention.prune deletes the record together with its set.
  2. The backfill is best effort (L2). A BotoCoreError (ReadTimeoutError, ResponseStreamingError), a ClientError, or an unreadable local root set goes into backfill_errors and is retried on the next run. The copy, its marker and the CLI's offsite gauge are unaffected (cmd_replicate exits 0). _error_code also handles errors whose response is None.
  3. Legal holds and lock dates are reported apart (nit). locked_until is the latest retain-until date (compared as datetimes). legal_holds counts the refused versions held by a legal hold. Before this, the string max of "legal-hold" and ISO dates could hide the date.
  4. Monthly sets are locked for 12 months (D15). With Object Lock on the bucket, prune --destination s3://… extends the GOVERNANCE retention of every object of each set that retention keeps as a monthly set: its files, manifest and verified marker. It uses put_object_retention; retention.retained_monthly is new.
    • Date (monthly_lock_until): the first day of month M + monthly, plus daily days. This is the earliest day the policy could expire the set, so the lock never outlives the retention and ends at most a day before it. A property test checks this against retained for 4 daily/monthly settings.
    • Retention is never shortened and a COMPLIANCE lock is left alone (retention_kept). s3:BypassGovernanceRetention is never used.
    • A denied extension goes into errors and fails the run.
    • Reported as monthly_locked_until, retention_extended and retention_kept.

Docs

  • docs/operations/backup-restore.md:
    • s3:PutObjectRetention added to the prune row of the permission table; Bypass stays "never".
    • The monthly lock is now described, and the "monthly locked only for the default period" limit is removed.
    • legal_holds, backfill_rejected and backfill_errors are documented.
  • docs/operations/disaster-recovery-objectives.md and ADR 0035: the same limit is removed.

Tests (Homayar/backend/tests/test_backup_format.py, in-memory _FakeS3)

_FakeS3 gains:

  • legal holds (held)
  • per-object retention with put_object_retention, which refuses shortening as S3 does without the bypass
  • ETags
  • 404s for missing keys
  • get_object read counts
  • per-operation error injection (raising)

New tests, each failing on the pre-change code:

  • test_s3_prune_reports_legal_holds_apart_from_retention_dates (KeyError legal_holds)
  • test_a_rejected_backfill_candidate_is_reported_and_not_downloaded_every_night (KeyError backfill_rejected)
  • test_a_network_error_in_the_backfill_never_fails_the_copy_or_the_gauge: ReadTimeoutError escapes on the old code; the unreadable-root-set part makes the CLI exit 1 before the last commit
  • test_s3_prune_locks_monthly_sets_for_their_whole_retention_never_shorter (ImportError)
  • test_a_monthly_lock_never_outlives_the_retention_that_keeps_the_set ×4 (ImportError)

Run:

  • ruff check backend/backup backend/tests/test_backup_format.py: clean
  • ruff format --check: clean
  • pytest backend/tests/test_backup_format.py: 65 passed
  • test_backup_restore.py: 13 skipped (needs PostgreSQL); retained() output is unchanged by construction

Not touched: ci.yml, .env.example, migrations, saas_models.py, rls.py, config.py. Not exercised against a real S3/MinIO with Object Lock.

🤖 Generated with Claude Code

prune_s3 took the string max of "legal-hold" and ISO dates for
locked_until, so a single legal hold hid the latest retain-until date.
locked_until now reports the latest date only, and legal_holds counts
the refused versions held by a legal hold.

The _FakeS3 test double gains legal holds, per-object retention with
put_object_retention, ETags, get_object read counts and per-operation
error injection for the follow-up tests.

Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>
…fill fail a copy

The verified-marker backfill silently skipped a candidate whose
checksum failed, and with no store checksum read that copy back again
every night. Only ClientError was caught, so a BotoCoreError
(ReadTimeoutError, ResponseStreamingError) during the read-back
escaped after the new set was already copied, and the CLI never moved
the off-site gauge.

- Each candidate is checked by HEAD first (presence, size, the store's
  checksum), so a missing or truncated copy is rejected without a
  download. Only then are objects without a usable checksum read back.
- A read-back rejection is remembered beside the root set
  (sets/<id>.offsite-rejected, keyed by destination and the objects'
  ETags), so the same objects are not downloaded again; a copy uploaded
  again is checked afresh. retention.prune removes the record with the set.
- Rejections are reported in backfill_rejected (id and reason).
- The backfill is best effort: network and permission errors are
  reported in backfill_errors and retried next run; the copy, its
  marker and the gauge are unaffected.

Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>
…(D15)

D15 is S3 Object Lock in GOVERNANCE mode with retention matching 30
daily + 12 monthly sets, but the bucket default (30 days) locked the
monthly sets only for the daily window.

With Object Lock on the bucket, prune --destination s3:// now extends
the GOVERNANCE retention of every object of each set retention keeps as
a monthly set (files, manifest, verified marker) with
put_object_retention. The date (monthly_lock_until) is the earliest day
the policy could expire that set: the first day of the month `monthly`
months later plus the daily window, so the lock never outlives the
retention and prune can always delete the set once the policy lets go.
Retention is never shortened, a COMPLIANCE lock is left alone, and the
bypass permission is never used. A denied extension is reported in
errors and fails the run. retention.retained_monthly exposes which kept
sets only the monthly rule keeps.

Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>
Runbook: the bucket default covers the daily window and prune extends
the monthly sets' GOVERNANCE retention; s3:PutObjectRetention joins the
prune permission table (s3:BypassGovernanceRetention stays "never");
locked_until and legal_holds are reported apart; the S3 backfill reports
backfill_rejected / backfill_errors and never fails the copy.
DR objectives and ADR 0035: drop the "monthly sets locked only for the
default period" limit.

Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>
A root set whose manifest cannot be read (BackupInvalid, OSError) is
now reported in backfill_errors like a network error, so the copy and
the off-site gauge are unaffected.

Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>
monthly_lock_until assumes a set every day. When daily backups were
missed, a monthly set stays kept past that date, and a set never
extended before (first run after deploy, a late change of the month's
representative, an earlier denied extension) got put_object_retention
with a past date. Real S3 rejects that with InvalidArgument, so prune
exited 1 every night until the set expired.

Prune now leaves such a set's lock alone and counts it in
retention_past_due: a lock then would outlive the retention. The
_FakeS3 rejects a past RetainUntilDate with InvalidArgument as S3 does.

Co-Authored-By: Mohammad Kohandezh <kohandezh@hotmail.com>
@Kohandezh

Copy link
Copy Markdown
Owner Author

Integrator merge evidence (Lane C S3 follow-ups after #77; D15 monthly locks), head 976dc9b:

  • Independent review at 9d60244: MERGEABLE apart from L1, a past retain-until date that made prune exit 1 every night after missed daily backups. The fix in 976dc9b skips past-due sets and counts them in retention_past_due. The test fake now rejects a past RetainUntilDate, as real S3 does. A missed-days test fails on 9d60244.
  • Verified: retention is never shortened; compliance locks are left alone; there is no governance bypass; monthly dates are computed in UTC; a denied PutObjectRetention leaves deletion unaffected; all Lane C privacy follow-ups: journal identity evidence, verified off-site copies, held checkpoints (HOM-212/213/214) #77 guarantees hold; the permissions table adds only s3:PutObjectRetention. test_backup_format: 66 passed. ruff is clean.
  • CI on 976dc9b: every check passes.
    Merging.

@Kohandezh
Kohandezh merged commit 6ae7785 into main Oct 19, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant