Skip to content

feat(memory-engine): add off-peak consolidation windows - #3233

Open
xiaoxiaoHe-E wants to merge 2 commits into
vectorize-io:mainfrom
xiaoxiaoHe-E:feat/consolidation-window
Open

feat(memory-engine): add off-peak consolidation windows#3233
xiaoxiaoHe-E wants to merge 2 commits into
vectorize-io:mainfrom
xiaoxiaoHe-E:feat/consolidation-window

Conversation

@xiaoxiaoHe-E

Copy link
Copy Markdown

Summary

Implements #2762 — restrict the consolidation background job (Hindsight's most LLM-heavy operation) to a configurable daily off-peak window (e.g. 01:00–06:00), instead of firing whenever facts become eligible.

Tasks that land outside the window are deferred, not failed: the worker holds them as pending with next_retry_at set to the next window open, and the backlog drains automatically when the window reopens (claim SQL next_retry_at <= NOW()).

Changes

  • New static config trio (config.py):
    • HINDSIGHT_API_CONSOLIDATION_WINDOW_START / HINDSIGHT_API_CONSOLIDATION_WINDOW_ENDHH:MM (24h) daily window boundaries; must be set together. START > END = cross-midnight window (e.g. 22:00–06:00); START == END = always-open. Unset = always-on (previous behaviour).
    • HINDSIGHT_API_CONSOLIDATION_WINDOW_TZ — IANA timezone the boundaries are evaluated in; unknown names fall back to UTC with a warning.
  • engine/consolidation_window.py (new) — pure, deterministic next_window_open() schedule arithmetic (normal / cross-midnight / DST-aware), unit-tested.
  • _handle_consolidation gate — defers via the existing DeferOperation mechanism (no retry_count bump, no error_message) while the window is closed.
  • Wires the previously-dead validate_consolidate extension hook into the same deferral path — it was defined on the validator interface but never called for consolidation. A rejection is now a deferral (retry-backoff), not an HTTP 403.
  • Sync task backend safeguard — on SyncTaskBackend (tests / CLI / embedded single-process) there is no worker queue to re-claim a deferred task, so raising DeferOperation would 500 the HTTP caller and strand a pending op that blocks banks_needing_consolidation() from re-submitting the bank. There it logs and runs now (window closed) / skips (hook rejection) instead.

Notes

  • The manual sync path run_consolidation() (explicit user trigger) is intentionally not gated by the window — the window applies to the async background job.
  • Window boundaries use wall-clock replace(); on DST spring-forward/fall-back transition days the boundary can shift by one hour (window times are typically in the early morning).

Testing

  • 21 unit tests: schedule arithmetic (next_window_open) + env parsing (_parse_consolidation_window*).
  • 7 integration tests: worker deferral semantics (no retry bump / no error, next_retry_at = next window open), reconcile non-resubmission, extension rejection deferral, sync-backend run-now/skip behaviour.
  • ruff, ty, and the repo pre-commit hooks (lint.sh, generate-docs-skill.sh, check-unused.sh) all pass.

Closes #2762

…#2762)

Restrict the consolidation background job (Hindsight's most LLM-heavy
operation) to a configurable daily off-peak window, e.g. 01:00-06:00.
Tasks that land outside the window are deferred — held as pending with
next_retry_at set to the next window open — instead of running during
peak hours. The backlog drains automatically when the window reopens
(claim SQL "next_retry_at <= NOW()").

- New static config trio: HINDSIGHT_API_CONSOLIDATION_WINDOW_START/END/TZ.
  START > END means a cross-midnight window; START == END is treated as
  always-open. Unset = always-on (previous behaviour).
- Window arithmetic lives in engine/consolidation_window.py as a pure,
  fully deterministic function (normal, cross-midnight, DST-aware).
- _handle_consolidation defers via DeferOperation while the window is
  closed; on the sync task backend (tests/CLI/embedded, no worker queue
  to re-claim the task) it logs and runs anyway instead of 500-ing the
  HTTP caller or stranding a pending op.
- Wire the previously-defined-but-never-called validate_consolidate
  extension hook into the same deferral path.
- Docs, .env.example and the embed env template updated; 21 unit +
  integration tests added.
…e-io#2762)

Add deterministic unit tests for _parse_consolidation_window and
_parse_consolidation_window_boundary: HH:MM validation (valid, missing,
malformed), the START/END-must-be-paired rule, the START==END degenerate
case, and the timezone fallback (invalid/empty -> UTC). These are the
operator-facing entry point of the off-peak window config and previously
had no direct coverage — the gate tests bypass parsing by mutating
_get_raw_config() fields directly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Off-peak / scheduled consolidation windows (run async consolidation only during a configurable time window)

1 participant