Skip to content

feat(session): add retry endpoint for failed archive Phase 2 extraction#3340

Open
njuboy11 wants to merge 1 commit into
volcengine:mainfrom
njuboy11:feat/retry-archive-extraction
Open

feat(session): add retry endpoint for failed archive Phase 2 extraction#3340
njuboy11 wants to merge 1 commit into
volcengine:mainfrom
njuboy11:feat/retry-archive-extraction

Conversation

@njuboy11

@njuboy11 njuboy11 commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Description

When a session archive's Phase 2 (memory extraction) step fails — typically due to transient VLM/LLM API errors, timeouts, or upstream rate-limits — OpenViking currently writes a .failed.json marker and gives up. Operators have no programmatic way to re-trigger extraction; the only options are to restart the service (which loses all in-flight extraction state) or to manually delete the marker and hope the next commit re-runs.

This change adds a first-class retry endpoint:

POST /api/v1/sessions/{session_id}/archives/{archive_id}/retry

The handler re-reads the archive's messages.jsonl, deletes the .failed.json marker if present, and re-runs _run_memory_extraction as a background task. The response carries a task_id so callers can poll progress through the existing GET /tasks/{task_id} endpoint.

Human Involvement

  • A human participated in the implementation or review loop

Type of Change

  • New feature (non-breaking change that adds functionality)

Changes Made

openviking/session/session.py

  • New keyword argument skip_previous_archive_check: bool = False on _run_memory_extraction. When True, the call to _wait_for_previous_archive_done is skipped — necessary for retries since the previous archive may itself lack .done.
  • New Session.retry_archive_extraction(archive_index) method (≈70 lines) that performs the retry: re-reads messages, deletes .failed.json, creates a tracker task, schedules _run_memory_extraction with skip_previous_archive_check=True.

openviking/server/routers/sessions.py

  • New POST /{session_id}/archives/{archive_id}/retry endpoint that delegates to the service-layer wrapper. Uses the same run_operation telemetry path as extract_session.

openviking/service/session_service.py

  • New SessionService.retry_archive(session_id, archive_id, ctx) wrapper that resolves the session and forwards to retry_archive_extraction.

Diff stat

 openviking/server/routers/sessions.py |  30 ++++
 openviking/service/session_service.py |  17 ++
 openviking/session/session.py         |  74 +++++++++-
 3 files changed, 120 insertions(+), 1 deletion(-)

Testing

Production validation (OpenViking 0.4.10, 2026-07-18)

A batch of 19 failed archives was retried via the new endpoint:

Outcome Count Notes
HTTP 200 accepted 19/19 100% acceptance
Transitioned to .done with new memory_diff.json 16 47 add + 13 update ops across 1,280 committed messages
Persistent failure (LLM empty_response, plugin API incompat, orphan Phase 2) 3 Out of scope for this PR but easier to triage with the new endpoint
Mid-poll self-heal 1 memory_diff.json and .done both written 24 ms apart by retry task

All three modified files compile cleanly (python3 -m py_compile) and the service starts normally (/health returns {"status":"ok"}).

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this on the following platforms:
    • Linux

(Unit-test coverage for the new endpoint is not included in this PR; the production validation above was performed against a live 0.4.10 deployment. Happy to add pytest coverage as a follow-up if reviewers want.)

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Compatibility

  • The skip_previous_archive_check kwarg defaults to False; existing callers see no behavioural change.
  • The new endpoint is purely additive.
  • No protocol-level changes; existing clients continue to work.

Additional Notes

Why endpoint and not a CLI subcommand?

Two options were considered:

  1. ov retry-archive <sid> <archive_id> CLI subcommand — would only work when the operator has shell access to the server.
  2. HTTP endpoint — works for remote operation, fits naturally with the existing extract_session endpoint, and is trivially invokable from the OpenClaw plugin (which already calls OV over HTTP).

The endpoint was chosen for parity with POST /sessions/{id}/extract.

Design notes

  • retry_archive_extraction reuses _run_memory_extraction so all the existing telemetry, tracker integration, and error-handling paths apply unchanged. There is no duplicated Phase 2 logic.
  • skip_previous_archive_check=True is mandatory on the retry path because the prior archive may itself have failed Phase 2 and left no .done marker; without this flag, retry would deadlock waiting for that marker to appear.
  • The endpoint does not currently retry recursively; if the retried Phase 2 fails again, a fresh .failed.json is written and the operator can call the endpoint once more.

When a session archive's Phase 2 (memory extraction) step fails —
typically due to transient VLM/LLM API errors, timeouts, or upstream
rate-limits — OpenViking currently writes a ``.failed.json`` marker
and gives up. Operators have no programmatic way to re-trigger the
extraction; the only options are to restart the service (which loses
all in-flight extraction state) or to manually delete the marker and
hope the next commit re-runs.

This change adds a first-class retry endpoint:

    POST /api/v1/sessions/{session_id}/archives/{archive_id}/retry

The handler re-reads the archive's messages.jsonl, deletes the
``.failed.json`` marker if present, and re-runs ``_run_memory_extraction``
as a background task. The response carries a ``task_id`` so callers can
poll progress through the existing ``GET /tasks/{task_id}`` endpoint.

Three files touched:

  * ``openviking/session/session.py``
    - New ``skip_previous_archive_check`` keyword argument on
      ``_run_memory_extraction`` (default ``False`` — backwards
      compatible). When ``True``, the call to
      ``_wait_for_previous_archive_done`` is skipped, which is necessary
      for retries since the previous archive may itself lack ``.done``.
    - New ``Session.retry_archive_extraction(archive_index)`` method
      that performs the retry.

  * ``openviking/server/routers/sessions.py``
    - New ``POST /{session_id}/archives/{archive_id}/retry`` endpoint
      that delegates to the service-layer wrapper.

  * ``openviking/service/session_service.py``
    - New ``SessionService.retry_archive`` wrapper that resolves the
      session and forwards to ``retry_archive_extraction``.

Production validation
---------------------

Live-tested on OpenViking 0.4.10 with 19 failed archives:

  - 19/19 POST requests returned HTTP 200
  - 16 archives transitioned to ``.done`` with new ``memory_diff.json``
    containing 47 ``add`` ops and 13 ``update`` ops across 1,280
    committed messages
  - 3 archives remained stuck due to persistent failures (LLM
    ``empty_response``, plugin API incompatibility, missing Phase 2
    invocation entirely) — these are not addressed by this change but
    are now easier to triage with the retry endpoint
  - 1 archive self-healed mid-poll when its ``memory_diff.json`` and
    ``.done`` marker were both written 24 ms apart by the retry task,
    confirming the polling path is consistent

Related issues
--------------

  * volcengine#2936 — auto-commit threshold wiring
  * volcengine#2476 — force flag to skip blocking failed archives
  * volcengine#3297 — event-memory extraction fails when a
    new page_id is encountered

Compatibility
-------------

  * The ``skip_previous_archive_check`` kwarg defaults to ``False``;
    existing callers see no behavioural change.
  * The new endpoint is purely additive.
  * No protocol-level changes; existing clients continue to work.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

1 participant