Skip to content

docs: recovery objectives (RPO/RTO, rollback, capacity) as they are today - #182

Merged
sinashamsizadeh merged 2 commits into
mainfrom
docs/recovery-objectives
Oct 10, 2026
Merged

sinashamsizadeh merged 2 commits into
mainfrom
docs/recovery-objectives

Conversation

@sinashamsizadeh

Copy link
Copy Markdown
Collaborator

Root cause

Evaluator reasons 4 and 6: no evidence of recovery objectives, operational stability or formal documentation. The
spike (docs/features/db-maturity/RESEARCH.md, ADR-028) set RPO/RTO targets and the owner accepted most of them, and
several units are now on main. But no single document said, per failure scenario: the target, what on main meets it,
how it is measured, and what is NOT met yet. An evaluator or an operator had to piece it together.

Design doc

docs/features/db-maturity/RESEARCH.md (sections 5.5, 5.6, 5.9, 6 item 5, 9) and ADR-028. This PR is the document
itself: docs/engineering/RECOVERY_OBJECTIVES.md (Persian), plus one link line in the runbook.

What it says, honestly:

  • Four scenarios: one install's data damaged; a bad deploy or migration; the whole cluster damaged; the server lost.
  • For each: the target (owner decision with its date, or "derived, not confirmed" for the 5-minute server-loss RPO;
    scenario (b) has no accepted target), what on main meets it with file:line, how it is measured, and today's state.
  • Today: point-in-time recovery (pgBackRest) is NOT built, so the RPO is the time since the newest verified backup,
    not 5 minutes; no SFTP destination exists yet, so losing the server loses everything; nothing has been measured on
    the production server; the pre-deploy dump is not verified, drilled or copied off-site automatically.
  • Rollback (code and data paths, expand/contract as a rule, not a guarantee), capacity (every number labelled as an
    owner decision, a dev-Mac measurement or an estimate), and the open work, server steps and owner questions.

Tests

Docs only, so no new test. tests/test_no_legacy_engine_references.py passes locally. Every file:line citation was
checked by script against the base commit (94 in range), and the critic re-checked a sample of 90 plus the 12 new ones
by hand. No sentence claims a server measurement or a human review.

Security

No security impact: documentation only.

AI assistance

Written by Claude agents in a Herdr team: writer drill-t1-impl (Opus 5.5); critic drill-t1-rev (Opus 5.5) wrote a
30-question checklist from the task before reading the document, then reviewed two rounds (round 1: 1 medium, the
pre-deploy dump's limits were missing, plus 5 lows; round 2: accepted); lead foreman7 (Opus 5.5) checked the medium
against the code. Log entry in docs/engineering/AI_ASSISTANCE_LOG.md.

Human review

sinashamsizadeh and others added 2 commits October 10, 2026 13:36
One document for the evaluator and the operator: for each failure
(one install's data, a bad deploy, the whole cluster, the server lost)
it gives the RPO and RTO target with its source, what on main meets it
(file:line at 90b72c2), how it is measured, and what is not met yet:
PITR is not built, so today's RPO is the backup interval, not 5 minutes;
no SFTP destination exists, so losing the server loses everything;
nothing has been measured on the server. It also summarises rollback,
capacity (spike numbers labelled as Mac measurements or estimates, the
300 s dump and 1800 s restore limits), and the open work and owner
questions. The runbook links to it from its last line, so its line
numbers stay as cited.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- The pre-deploy dump is not verified, drilled or copied off-site
  automatically (padyar-deploy.sh only calls create()); say so where it
  is offered as the way back, and tell the operator to press the
  health-check button first.
- Today's RPO is the time since the newest verified backup, not just
  the interval; point to SLO.md SLI 4.
- The 300 s dump cap also refuses a panel restore, through its safety
  dump.
- The drill's free-space check reads the backups folder's disk; no
  total disk budget is set yet (only the parts are listed).
- expand/contract is a reviewed rule, not a guarantee.
- storage.py:122 -> :226 (free disk, not database size).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@sinashamsizadeh
sinashamsizadeh marked this pull request as ready for review October 10, 2026 10:25
@sinashamsizadeh
sinashamsizadeh merged commit f83e7f0 into main Oct 10, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant