Skip to content

Add durable cleanup manifests for async task deletion #1545

Description

@legreffier

Context

PR #1532 moves task deletion cleanup into async DBOS workflows. The current near-term fix can improve retries and logging around object deletion, but a full durable cleanup/reconciliation path is larger than that PR.

When task rows are deleted, task artifact and runtime session cleanup refs can disappear through FK cascade before blob/object deletion has permanently succeeded. If object cleanup later exhausts retries, operators need a durable in-app way to identify and retry orphan cleanup.

Proposal

Add a durable cleanup manifest/tombstone mechanism for task deletion and retention cleanup workflows.

Possible shape:

  • Persist a cleanup manifest before destructive row deletion, including task ids, artifact object keys, runtime session object keys, requester/context, force flag, and workflow id.
  • Mark manifest phases explicitly, for example: planned, rows_deleted, objects_deleted, failed, reconciled.
  • Make object deletion retryable from the manifest after the original DBOS workflow has failed or aged out.
  • Add a maintenance reconciler that periodically retries failed/incomplete cleanup manifests in bounded batches.
  • Expose enough logs/metrics to diagnose partial cleanup without scraping DBOS internals.
  • Ensure retention cleanup and explicit task deletion share the same manifest/retry model.

Acceptance criteria

  • Failed object deletion after task row deletion leaves durable cleanup state with object keys.
  • A maintenance workflow can retry incomplete object cleanup later.
  • Cleanup work is bounded by batch size/page size, not by all artifacts/sessions attached to the deleted tasks.
  • Operators can correlate cleanup attempts by manifest id/workflow id/task ids.
  • Tests cover partial object deletion failure and later reconciliation.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions