Skip to content

Chronos-2: document resource-safety contract of after_batch callback or wrap predict loop in try/finally #490

@shaun0927

Description

@shaun0927

Problem

PR #436 added an after_batch callback to Chronos2Pipeline.predict (see src/chronos/chronos2/pipeline.py:627-652). The callback is invoked inside the per-batch loop, but the loop is not wrapped in a try / finally, and the callback's resource-safety contract is undocumented.

If a user callback raises (for example the AutoGluon time-limit signalling referenced in #436), any batch-scoped resources acquired in the loop (the DataLoader created at line 627 with pin_memory=True on CUDA, in-flight CUDA streams, open worker processes) are released only when Python's GC happens to run, and any predictions already appended to all_predictions are discarded without being returned to the caller.

Proposal

Pick one of the following and document it in the after_batch parameter docstring:

  1. Contract: state that after_batch must not raise; any exception will abort predict and discard in-flight results. Callers are responsible for catching inside the callback.
  2. Hardening: wrap the batch loop in try / finally so the DataLoader and pinned host buffers are always released deterministically, and optionally return the partial predictions collected so far when the callback signals early stop via return value.

Option 1 is zero-code; option 2 covers the AutoGluon timeout path without requiring users to audit their callbacks for resource safety.

References

  • Introduced in PR Chronos-2: Add after_batch callback #436 (Chronos-2: Add after_batch callback).
  • Affected lines: src/chronos/chronos2/pipeline.py:627-652.
  • Reproduction: ast inspection of main confirms zero try/finally wrapping the after_batch_callback() call at line 652.

Environment

  • chronos-forecasting main as of commit 6d68ed7.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions