Context
PR #1532 moves task deletion cleanup into async DBOS workflows. The current near-term fix can improve retries and logging around object deletion, but a full durable cleanup/reconciliation path is larger than that PR.
When task rows are deleted, task artifact and runtime session cleanup refs can disappear through FK cascade before blob/object deletion has permanently succeeded. If object cleanup later exhausts retries, operators need a durable in-app way to identify and retry orphan cleanup.
Proposal
Add a durable cleanup manifest/tombstone mechanism for task deletion and retention cleanup workflows.
Possible shape:
- Persist a cleanup manifest before destructive row deletion, including task ids, artifact object keys, runtime session object keys, requester/context, force flag, and workflow id.
- Mark manifest phases explicitly, for example: planned, rows_deleted, objects_deleted, failed, reconciled.
- Make object deletion retryable from the manifest after the original DBOS workflow has failed or aged out.
- Add a maintenance reconciler that periodically retries failed/incomplete cleanup manifests in bounded batches.
- Expose enough logs/metrics to diagnose partial cleanup without scraping DBOS internals.
- Ensure retention cleanup and explicit task deletion share the same manifest/retry model.
Acceptance criteria
- Failed object deletion after task row deletion leaves durable cleanup state with object keys.
- A maintenance workflow can retry incomplete object cleanup later.
- Cleanup work is bounded by batch size/page size, not by all artifacts/sessions attached to the deleted tasks.
- Operators can correlate cleanup attempts by manifest id/workflow id/task ids.
- Tests cover partial object deletion failure and later reconciliation.
Related
Context
PR #1532 moves task deletion cleanup into async DBOS workflows. The current near-term fix can improve retries and logging around object deletion, but a full durable cleanup/reconciliation path is larger than that PR.
When task rows are deleted, task artifact and runtime session cleanup refs can disappear through FK cascade before blob/object deletion has permanently succeeded. If object cleanup later exhausts retries, operators need a durable in-app way to identify and retry orphan cleanup.
Proposal
Add a durable cleanup manifest/tombstone mechanism for task deletion and retention cleanup workflows.
Possible shape:
Acceptance criteria
Related