Skip to content

fix(sessions): persist deferred approval history across resumes - #4828

Merged
jbeckwith-oai merged 49 commits into
openai:mainfrom
dixso:fix-deferred-interrupted-session-write
Oct 6, 2026
Merged

jbeckwith-oai merged 49 commits into
openai:mainfrom
dixso:fix-deferred-interrupted-session-write

Conversation

@dixso

@dixso dixso commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This pull request fixes missing tool calls in Session history when an approval resumes with output guardrails and a non-default tool_use_behavior.

Withheld response items survive serialized and detached approval resumes until a permitted save. Both runners share the resumed fold/defer/settle operation, preserving accepted history while honoring handoff filtering and output redaction. Local tool outputs and hosted MCP approval responses survive callback failures followed by serialized retry. Settlement reuses append recovery and post-write compaction.

Compaction uses the parked response's recorded storage policy, including when output guardrails block completion. Same-response partial approvals preserve that policy; a new detached park advances response identity and storage policy together. Both runners synchronize the owning turn before fresh interruption registration, including detached resumes where output guardrails have been disabled. Released schema 1.17 remains readable; held-write fields use unreleased schema 1.18. No public API or dependency is added.

The branch includes current main. A streamed retry that finishes from a previously committed tool output now settles held history even when the retry produces no new items, and its completed checkpoint remains loadable.

Test plan

  • Public Runner regressions cover both runner modes, approval/rejection, JSON retry, partial approvals, detached re-parking, handoff filtering, output guardrails, append recovery, compaction response identity/storage mode, and max-turn fallback.
  • Two independent reviewers cleared the final integrated diff after the streamed terminal-retry fix.
  • Full formatting, lint, Mypy, Pyright, and local test verification passed: 11,866 main tests and 89 serial tests, with 65 and 4 skips respectively under the prescribed local sandbox configuration.
  • Hosted CI results are attached to the PR; native macOS sandbox tests run on the dedicated CI runner.

Issue number

Closes #4827.

Checks

  • Added representative regression tests
  • Ran .agents/skills/code-change-verification/scripts/run.sh
  • Confirmed all local verification steps pass
  • Completed independent code review before submission

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e52216fa5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3730311a98

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run_internal/blocked_output.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9ba9fefa94

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run_internal/run_loop.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d93e3bf76a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run_internal/session_persistence.py Outdated
Comment thread src/agents/run_internal/run_loop.py Outdated
Comment thread src/agents/run_internal/session_persistence.py Outdated
Comment thread src/agents/run.py Outdated
Comment thread src/agents/run_internal/session_persistence.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1c6e6470a2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run_internal/session_persistence.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b09d8d14a2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run_internal/session_persistence.py Outdated
Comment thread src/agents/run_internal/run_loop.py Outdated

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t think (type, call_id) is collision-free across the Session. Custom model providers can reuse a call ID on a later turn; if an older matching call is still in this tail window, present suppresses the current deferred call and its output can again be persisted without its call. Could this key include a response/turn identity, or otherwise scope the match to the current response instead of treating call_id as globally unique?

@dixso

dixso commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

@sylvesterkaczmarek Reproduced before answering, and it fails exactly as you describe.

With call_1 already present in the Session tail from an earlier turn, and the current deferred response emitting call_1 again, deferred_interrupted_session_prefix returns []. The new call is suppressed while its output can still be persisted, recreating the exact orphan this PR is meant to prevent, reached through a different path.

tests/test_tool_approval_call_id_reuse.py also suggests the repo already treats call ID reuse as a real case, so I don't think we can rely on (type, call_id) being unique enough for reconciliation.

Scoping the match to the current response is the right direction, but I couldn't find a reliable way to do that from Session history alone. A Session persists a flat sequence of function_call / function_call_output items, with no marker identifying which model response or turn produced them. So adding response or turn identity to the key doesn't help unless that identity is persisted too, which feels like a broader Session contract change rather than something this fix should introduce implicitly.

I also tested the narrower alternative of matching the entire converted prefix as an ordered block instead of matching individual items. That fixes the collision case, but breaks partial writes: if an earlier attempt persisted the calls but failed before persisting the output, the full prefix no longer matches and the calls are appended again.

That seems to be the recurring signal from the edge cases on this PR: we're trying to answer "was this batch already written?" from Session history, but Session history doesn't contain enough provenance to answer that reliably.

So I think the cleaner direction is to stop inferring it.

RunState._pending_session_write already represents almost exactly what we need: a canonical session append that is pending acknowledgement. It's serialized by to_json, restored by from_json, and reconciled in order by resume_pending_session_write.

I prototyped changing the deferred park so that it records the withheld batch as the pending session write rather than dropping it and reconstructing it later. On resume, we then reconcile a declared batch instead of guessing from history. That removes the id-reuse, partial-write, detached-resume and cancellation cases I was able to construct, and actually deletes a fair amount of the reconciliation logic added by this PR.

There are two semantics I don't want to choose on behalf of the maintainers, though:

  1. RunResult.to_state() creates a fresh RunState through _populate_state_from_result, and _pending_session_write only survives when the result already carries a _state. So the first park currently loses it. We'd need to propagate it through the result, similar to _current_turn_persisted_item_count.

  2. Settling that deferred batch makes _current_turn_persisted_item_count > 0, which then triggers Cannot resume an approval checkpoint with output guardrails after current-turn items were persisted. Whether settling the deferred batch should count toward that guard is really a question about the intended invariant.

Full write-up and reproducer are in #4827.

I can push the prototype to this PR, open it separately, or hand the approach over if you'd rather own that shape. If you'd prefer to keep #4828 narrow and land the deeper persistence change separately, I can also just adjust the key here.

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the non-streaming loop still has the deferred-prefix loss case. On NextStepRunAgain, it clears deferred_session_prefix even when turn_session_items is empty and therefore nothing was persisted; the streamed loop now guards that case. Could the non-streaming path keep the deferred prefix until at least one resolved turn item is actually saved?

@dixso

dixso commented Sep 4, 2026

Copy link
Copy Markdown
Contributor Author

@sylvesterkaczmarek I reproduced this before answering, and the result is clearer than I expected.

I built the empty-resolved-turn case end to end: park a gated call, approve it, then resolve into a turn whose session items are emptied by a handoff input_filter. I ran it on both runners. They already behave identically, and identically badly: neither writes a dangling call, but both lose the deferred batch completely. The approved tool executes, yet neither its call nor its output reaches the Session.

streamed  items=[('user',''), ('message','')]  call_PARKED present: False
nostream  items=[('user',''), ('message','')]  call_PARKED present: False

I then traced the non-streaming loop for that exact run. The NextStepRunAgain check at run.py:1264 is evaluated, but the clear at :1267 never executes because the emptied turn resolves into NextStepHandoff, not RunAgain.

I also couldn't construct a RunAgain case with empty turn_session_items: approving or rejecting an interruption always produces an output item, while the only path I found that empties the turn is a handoff filter, which takes the other branch.

Finally, as a mutation test, I deleted the clear entirely and re-ran the eight reproducers from this PR plus the full suite. The outcomes were byte-identical.

The variable only lives for a single resume pass: that branch is entered while _current_step is an interruption, and update_run_state_after_resume replaces it before the loop continues. Nothing reads the variable after that pass, and a later resume recomputes it from state.

So my earlier comment on that line ("later turns of this run must not re-send it") was incorrect.

The loss your review points out is real, but it isn't reachable through that clear, and retaining the variable can't bridge it. Within the pass there is no later consumer; across runs, the only carrier is reconciliation against Session history, which your collision finding already showed can't be made reliable.

The one place the batch survives all of this is the serialized checkpoint. That's the _pending_session_write direction in #4827, pending the two maintainer decisions listed there.

What I did push is a regression test (23124284) pinning what we can verify: an emptied resolved turn writes no dangling call and no orphaned output in either runner, and both runners produce identical Session contents in that shape. It's proven red against 9ba9fefa, where the streamed path wrote both calls dangling.

I left the clear itself alone. Deleting it changes nothing measurable, and changing dead code as if it fixed this would be misleading.

If a maintainer weighs in on #4827, I'll finish the declarative version, which makes this whole family of cases unreachable.

seratch
seratch previously requested changes Sep 5, 2026

@seratch seratch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ordinary orphan-output case is real, but the current prefix-inference approach still has supported resume failures: the lookup happens after approved tool execution, the streamed lookup drops the context wrapper, legacy no-argument Sessions fail, and finalization/reconnect can duplicate or lose the batch. I recommend redesigning around the deferred response's durable ownership instead of adding another inference branch. Reusing pending-write recovery must also preserve the output-guardrail persistence gate; eagerly writing the held prefix before that gate is not a safe replacement.

@dixso
dixso force-pushed the fix-deferred-interrupted-session-write branch from 2312428 to 47bfebe Compare September 5, 2026 10:03
@dixso dixso changed the title fix(sessions): persist deferred interrupted-turn items when the approval resume continues the run fix(sessions): declare the withheld interrupted write as a held pending Session write Sep 5, 2026
@dixso

dixso commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

@seratch

Thanks for the direction, it was the right call. I have replaced the reconciliation
approach entirely with durable ownership on the checkpoint, as you suggested.

The park now registers the withheld batch on the existing
RunState._pending_session_write slot with a held marker. Registering touches only
the checkpoint, so the output-guardrail persistence gate is fully preserved; the batch
settles at a gate-legal exit of a later resume, landing ahead of the resolved turn's
items in one ordered append and inheriting the existing digest-based crash recovery.
The lookup, the identity heuristics, and the lookback constant are deleted.

Each of your four points now fails by construction rather than by inference: there is
no post-execution lookup, every settle flows through wrapper-carrying machinery, the
legacy-session read shape is gone (plus a signature probe for the settle's inherited
limit= reads), and the declaration survives serialization, the run-again flip, and
detached reconnects, with explicit settle, extend, or discard at every exit.

The PR description is rewritten with the full design, the serialized-state note
(held extends unreleased 1.17, following the same pattern as the commit that
introduced pending_session_write), and the decisions I took with their rationale.
The new tests drive every scenario through both runners and a serialized round trip,
and I ran a mutation pass proving each new guard turns at least one test red.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 47bfebebf9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run_internal/session_persistence.py
Comment thread src/agents/run.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

resumed_write_state=(
run_state
if run_state is not None
and isinstance(run_state._current_step, NextStepRunAgain | NextStepInterruption)
else None

P1 Badge Arm recovery before settling a final held batch

When a streamed approval resume resolves directly to NextStepFinalOutput, _save_resumed_stream_items has already removed the held batch via take_held_session_write, but this condition excludes the final step from resumed_write_state. A Session append failure or lost acknowledgement therefore leaves no pending write in the checkpoint; retrying skips the completed tool while its call/output exchange remains absent from the Session. Route final settlement through the pending-write recovery path before clearing the held record.

AGENTS.md reference: AGENTS.md:L104-L104

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run_internal/session_persistence.py
Comment thread src/agents/run_internal/session_persistence.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f007612498

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run.py Outdated
Comment thread src/agents/run_internal/agent_runner_helpers.py
Comment thread src/agents/run_internal/session_persistence.py

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b470baec5c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run_internal/session_persistence.py Outdated
Comment thread src/agents/run_internal/session_persistence.py
Comment thread src/agents/run_internal/session_persistence.py
Comment thread src/agents/run_internal/session_persistence.py
Comment thread src/agents/run.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

resumed_write_state._pending_session_write = {
"session_id": session.session_id,
"items": copy.deepcopy(items_to_save),
"before": None,
"persisted_count": (
resumed_write_state._current_turn_persisted_item_count + saved_run_items_count
),
}

P1 Badge Include held items in the pending write recovery count

When a held multi-approval checkpoint is resumed with the persistence gate off but without a new approval decision, the approval placeholders trigger settlement but convert to zero new_items, while the held calls are passed through original_input; consequently saved_run_items_count is zero and this pending record also stores a zero persisted count. If the append fails or loses its acknowledgement, resume_pending_session_write() reconciles the held calls but restores that zero count, so a later gate-enabled resume can pass the output-guardrail safety check and append those calls again. Fresh evidence beyond the prior successful-settle counting thread is that save_resumed_turn_items() adds len(held_input) only after success, leaving this failure-recovery metadata uncorrected.

AGENTS.md reference: AGENTS.md:L104-L104

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run_internal/agent_runner_helpers.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 821afdc3f7

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run_state.py Outdated
Comment thread src/agents/run_internal/session_persistence.py Outdated
Comment thread src/agents/run_internal/run_loop.py Outdated

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the current head is ready to approve yet. There are still unresolved correctness/compatibility issues on the current diff: the new pending_session_write.held serialization needs schema-version treatment consistent with the released reader contract; held-write reattachment bypasses the Responses compaction bookkeeping used by canonical persistence; and the streaming max-turn-handler terminal path can retain a held write after reporting completion. The post-tool callback failure window documented in the open thread is also a real durability gap at the commit boundary. Please resolve the remaining current-head threads, or narrow/document the compatibility contract sufficiently, before re-requesting review.

@dixso

dixso commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, that was a precise list. All four are addressed on the current head, each with a regression test proven red against the previous commit.

Schema. You and Codex were right and my reasoning was wrong: I leaned on 1.17 being unreleased, but the 1.17 reader validates the pending write by exact key set, so a checkpoint written under that label with the extra keys is not loadable by a 1.17 reader, and my own regression called the four-key form released behaviour. CURRENT_SCHEMA_VERSION is now 1.18, 1.17 keeps its four-key form and its original summary, the held keys are gated to 1.18, and the corpus fixtures, sources, README and version-boundary test are updated. Both directions are pinned: a 1.17 payload still restores and settles eagerly, and the held variant under a 1.17 label is refused.

Compaction bookkeeping. Fixed at the root rather than patched: the entry settle no longer appends behind the canonical path, it goes through save_result_to_session like every other settle, so it inherits the Conversations sanitization, the ordered dedup, the pending-write registration and the compaction bookkeeping. That needed the response the batch belongs to, which the park now records (the second key the version bump covers). Pinned by a test that asserts a compaction-aware backend receives the bookkeeping for the parked response.

Max-turn terminal path. Both runners now discard a still-standing held batch when a max-turn handler ends the run, so the finished run's checkpoint stays loadable and the streaming result no longer reports terminal output while carrying a resumable pending write the non-streaming result had already dropped.

Post-tool callback window. Fixed rather than documented: the resumed turn's output committer folds the committed output into the held batch as it commits it, so a callback that raises afterwards cannot leave a retry that skips the completed invocation and drops the executed call and its result.

One more round, self-inflicted. Reviewing my own diff afterwards turned up four defects it had introduced, all fixed on this head with a test proven red against the previous commit:

  • The resumed turn's committer folds the executed output into the held batch (the post-tool fix above), which made a wholesale discard on an emptied resolved turn wrong: it threw away a fully paired call and output, so the Session lost its only record that the approved tool ran and the next run would re-issue the side effect. The emptied turn now settles the batch's paired part and drops only the unpaired requests, which is the predicate every other settle already uses.
  • Widening the compaction check to see the settling batch made it read the whole original_input slot, which carries the caller's own input on an ordinary save. It now reads that slot only on the calls that actually settle a batch.
  • The settled count added the batch's raw length, overcounting whatever the dedup dropped. The append reports what it wrote, because that count slices a later save of the same turn positionally.
  • The max-turn discard ran before validate_handler_final_output, so a wrongly typed handler output lost a batch the streamed runner keeps. It now sits below the validation.

Verification on this head: full suite green (9471 passed) except one pre-existing sandbox failure that also fails here with this branch stashed, mypy and mypy --platform win32 clean, pyright clean, and a mutation pass in which every new guard was proven to turn a test red.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 14c8315506

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run_internal/run_loop.py Outdated
Comment thread src/agents/run_internal/run_loop.py Outdated
Comment thread src/agents/run_internal/session_persistence.py Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review


P1 Badge Include settled items in the compaction return count

When a partial-approval turn settles a held call/output batch into an OpenAIResponsesCompactionSession, this local-output branch returns only saved_run_items_count, omitting the settled_batch_items that were appended. The current persisted count can therefore remain zero; if the caller later re-enables the output-guardrail gate and approves the remaining call, the resumed-safety check permits the final sweep to append the already-stored calls again, corrupting Session history. Fresh evidence beyond the earlier count thread is that this compaction-only return bypasses the corrected common return on line 790. Return the combined count here as well.

AGENTS.md reference: AGENTS.md:L102-L102

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/agents/run.py
Comment thread src/agents/run_internal/session_persistence.py Outdated
@jbeckwith-oai
jbeckwith-oai enabled auto-merge (squash) September 28, 2026 16:37
markstuart-oai
markstuart-oai previously approved these changes Sep 28, 2026

@markstuart-oai markstuart-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 1c16e5b2, including the complete PR against its unchanged base and the follow-up to the previous detached-resume finding. The fresh interruption branch now synchronizes the turn before registering held history, so response B retains its own storage policy when guardrails are disabled and the run later reattaches. The extended public regression covers both runners and serialized resumes; same-response approvals still retain the original policy.

No remaining actionable findings. The shared persistence operation and decomposed test suites address the earlier structural concerns, while settlement continues through the existing recoverable append and compaction path.

All 21 hosted checks passed on this commit. Source-only review here, with an independent pass over the turn/state fix; I did not run repository tests or builds locally.

@dixso

dixso commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the quick turnaround on this. The current-response boundary is a cleaner ownership rule than the fold marker, and moving the storage selection ahead of tripwire redaction was a good catch.

I verified 1c16e5b2 against our own application, where the issue reported in #4827 originally surfaced. I built a wheel from that exact commit and ran it through our real application stack from the browser, without a test harness.

I couldn't reproduce any duplicates or 400 No tool call found errors. During the pending window, the parked call lives only in the checkpoint, with held: true, before: null, and the recorded store. Approval then settles it in a single ordered append. Most importantly, the follow-up turn that previously killed the conversation now answers correctly from history.

I ran the original incident three times, then exercised chained approvals, rejection, a page refresh while approval was pending, and approving one sibling while rejecting the other.

I did find a few orphaned outputs, but traced those back to our application rather than the SDK. Our code writes a synthetic rejection output without resuming the run, which closes a call_id that the gate is still holding. I've tracked that separately on our side. Those conversations kept working normally because drop_orphan_function_calls pruned the orphan before the request, and I verified against the actual persisted history that it removed that item and nothing else.

I also wrote independent probes rather than rerunning the tests from this PR, and mutated the implementation to make sure those probes fail when the relevant guarantees are broken. Two results may be worth calling out:

  • I couldn't reproduce the max-turn report anywhere in a sweep of max_turns from 1 through 8 across both runners, not only at 3.
  • I exercised two responses carrying an identical preamble across a detached re-park. Content equality cannot distinguish those responses, while the response boundary can, and each preamble remained at exactly one copy.

A couple of limits so this isn't read as broader validation than it is: our application only uses the streamed runner, so non-streamed parity is covered by your tests rather than my application-level verification. This is also one application and one tool shape, and I didn't exercise the sandbox or voice paths.

Happy to run anything else against the real stack if it would be useful, including cases that are awkward to reproduce in tests.

@jbeckwith-oai
jbeckwith-oai dismissed stale reviews from markstuart-oai and dpiet-oai via de0e92d October 1, 2026 22:25
auto-merge was automatically disabled October 1, 2026 22:25

Head branch was pushed to by a user without write access

dpiet-oai
dpiet-oai previously approved these changes Oct 1, 2026

@dpiet-oai dpiet-oai left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed de0e92d. No actionable correctness or compatibility findings in this bounded static review.

The streamed terminal-retry path now settles held history even when no new items are produced, after the persistence and guardrail gates. The regression covers serialized callback-failure recovery, retained call/output history, and a loadable completed checkpoint. The storage-policy and detached-turn fixes preserve response ownership, and held-write fields remain gated to schema 1.18 while 1.17 remains readable.

Scope: changed runtime paths for held-history registration, both runners' settlement, callback-failure checkpointing, handoff filtering, storage policy, append recovery and schema validation, with representative regression coverage. Tests and builds were not run locally; no live backend checks were performed.

Hosted evidence includes successful exact-head Tests, CodeQL and Release Readiness runs, but newer runs for the same head are action_required. This approval is a source-review conclusion and does not assert that all current CI gates are green.

@jbeckwith-oai jbeckwith-oai mentioned this pull request Oct 1, 2026
@seratch

seratch commented Oct 2, 2026

Copy link
Copy Markdown
Member

Thanks for the update. The held-write approach addresses the missing call/output history, but v0.23.0 has now shipped with RunState schema 1.18.

Please move the new held-write format to schema 1.19 and update the schema summary, fixtures, generator, and version assertions accordingly. Split validation so the compaction metadata already released in 1.18 remains accepted under 1.18, while held, reasoning_item_id_policy, and current_response require 1.19.

Please add compatibility coverage using a checkpoint emitted by the v0.23.0 writer, alongside coverage showing that held-write checkpoints emit 1.19 and cannot be relabeled as 1.18. The released 1.18 reader rejects these new fields, so extending that schema under the same label would misidentify the durable format.

If other maintainers persuades this PR w/ co-authored-by credit of the original contributor, that should be fine too.

dixso added 4 commits October 2, 2026 09:34
v0.23.0 released schema 1.18 with the compaction metadata on the pending write
and a reader that validates it by exact key set. A held record under that label
is therefore rejected by the released reader, so these tests pin the contract
seratch asked for: compaction metadata keeps loading under 1.18, a held
checkpoint emits the version that introduced the held keys, and relabeling it as
1.18 or 1.17 is refused. The emit and relabel cases are red on the current head.
…lit validator

v0.23.0 shipped RunState schema 1.18 with the compaction metadata on the
pending write, and its reader validates that object by exact key set. The held
keys arrived after that release, so writing them under the 1.18 label emits
checkpoints the released reader rejects and misidentifies the durable format.

The held pending write now introduces 1.19. The validator gates each key to the
version that introduced it: the four base keys from 1.17, the compaction
metadata from 1.18, and held, reasoning_item_id_policy and current_response
from 1.19, so a released 1.18 payload keeps loading while a held record is
refused under any older label. The 1.18 summary is restored to the text v0.23.0
published and 1.19 carries the held write. Version assertions compare against
CURRENT_SCHEMA_VERSION or the version tuple instead of a 1.18 literal, and the
corpus generator records v0.23.0 as the historical 1.18 writer and this commit
as the 1.19 writer; the regenerated fixtures follow in the next commit.
The corpus now carries 1.18 fixtures emitted by the v0.23.0 tag itself, both
the minimal payload and a pending write holding the compaction metadata that
release published, so the reader branch for the released label is covered by
historical-writer output instead of a relabeled canonical payload. The held
pending write moves to 1.19 fixtures emitted by this branch's own bump commit.

Only these entries were regenerated. tests/fixtures/run_state/generate_corpus.py
cannot rebuild the whole corpus in this checkout: the pending-approval scenario
pins writer 92aa1b9 (2026-08-05) while its code imports agents.testing, which
landed in openai#4362 on 2026-08-13, so that scenario fails before it can emit.
The relabel helper rewrote agent-scoped approvals and MCP recipient bindings for
every target label, but schema 1.18 supports both. Relabeling a current payload
as 1.18 therefore handed the reader a 1.17-shaped payload wearing a newer label,
and the test asserting that the released 1.18 format still loads would have
stayed green through a regression in that reader. The helper now downgrades only
below 1.18, and the held-gate test relabels the batch untouched so the held keys
are the only possible reason for the refusal. Both are proven red against a
reader whose approval gate moves off 1.18.

The generator and corpus README now record that the 1.19 writer SHA lives on
this branch and must be re-pinned to the squashed merge commit, the same way
821afdc stopped resolving for the entry this one replaces.
@dixso

dixso commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Done in fa2c43bc.

The held-write change now introduces schema version 1.19, and the validator gates each key on the version that introduced it: base keys from 1.17, compaction metadata from 1.18, and held, reasoning_item_id_policy, and current_response from 1.19. The 1.18 summary is restored to the text published with v0.23.0.

The corpus now records the v0.23.0 tag as the historical writer for 1.18, so both its minimal and feature fixtures come from actual writer output. Two regressions cover the remaining boundary: a held checkpoint emits the current version, and the same payload relabeled as 1.18 is rejected.

I also verified this against the released reader, not just the test suite. With v0.23.0 installed, its own 1.18 checkpoint loads, a 1.19 held checkpoint is rejected as unsupported, and the same checkpoint relabeled as 1.18 is rejected as invalid. The current branch loads the v0.23.0 checkpoint and preserves its compaction metadata.

@jbeckwith-oai jbeckwith-oai mentioned this pull request Oct 2, 2026

Copy link
Copy Markdown

Additional offline validation of fa2c43bc9e252c75d76d60c06959a725f1dc0205: a useful coverage case is reopening a disk-backed SQLiteSession, restoring the JSON checkpoint into a fresh agent, and switching between Runner.run and Runner.run_streamed.

The sequence tested was:

  1. Pause a scripted run on a tool requiring approval and serialize its checkpoint.
  2. Close and reopen the session database; restore into a fresh agent with the same tool definition.
  3. Approve or reject, then resume using the other runner mode.
  4. Reopen the database again and run a subsequent conversation against the saved history.

All four combinations of initial runner and approval/rejection passed on this head: resolved call/output pairs were saved in order exactly once in these runs, rejected tools did not execute, and the subsequent model input retained the pair. This exercises persistence across reconstructed objects, not a process crash.

In the larger public-Runner matrix, head passed 45 checks; base 430da63b3ba9895e3ddc77585012d5efa03615ba and sampled main a575a6e637feb9aea1b591237b007dd4991ddfba each passed 18 and failed 27. Those failures cover related history invariants, not 27 separate defects. The head's existing deferred-session-write resume/recovery/filter modules also passed all 98 tests.

No additional runtime fix emerged from this scope. Validation used Python 3.14.6, scripted models, SQLite, the head's locked dependency environment, and blocked network access. It does not establish real-provider behavior, crash safety, concurrent exactly-once semantics, or a full-diff approval.

AI disclosure: Codex generated and ran these checks; these are automated test observations.

@jnohclee-rgb

Copy link
Copy Markdown

Independent offline confirmation for #4827 and this existing fix, using ScriptedModel with fictional tool results and SQLite sessions reopened from disk. I tested streamed/non-streamed × JSON-round-tripped/in-memory RunState × guardrail/no-guardrail (eight cases). On main d3304cff56f1be2325d00e0f172534d613697f88, all four guarded cases persisted an orphaned function_call_output; four unguarded controls preserved complete pairs. On this PR's exact head fa2c43bc9e252c75d76d60c06959a725f1dc0205, all eight cases preserved complete pairs.

Python 3.14.7, SDK source metadata 0.23.1, tracing disabled, network denied, no API keys or calls. This is narrow local persistence evidence; it does not replace the full test suite, CI or broader guardrail/redaction/retry coverage. The exact diagnostic is below.

"""Offline public Runner/SQLiteSession approval persistence diagnostic."""
import asyncio
import json
import tempfile
from pathlib import Path
from agents import Agent, GuardrailFunctionOutput, Runner, RunState, SQLiteSession, StopAtTools, function_tool, output_guardrail, set_tracing_disabled
from agents.testing import ModelStep, ScriptedModel, assistant_message, function_call

set_tracing_disabled(True)

@function_tool(needs_approval=True)
def write_thing(query: str) -> str:
    return f"fictional:{query}"

@function_tool
def look_up(query: str) -> str:
    return f"fictional lookup:{query}"

@output_guardrail
async def always_fine(ctx, agent, output):
    return GuardrailFunctionOutput(output_info=None, tripwire_triggered=False)

async def run(agent, inp, session, streamed):
    if not streamed:
        return await Runner.run(agent, inp, session=session)
    result = Runner.run_streamed(agent, inp, session=session)
    async for _ in result.stream_events():
        pass
    return result

async def case(streamed, serialized, guarded):
    model = ScriptedModel([
        ModelStep(output=[function_call('look_up', {'query':'x'}, call_id='call_LOOKUP')]),
        ModelStep(output=[function_call('write_thing', {'query':'x'}, call_id='call_APPROVAL')]),
        ModelStep(output=[assistant_message('done')]),
    ])
    agent = Agent(name='fictional', model=model, tools=[look_up, write_thing], output_guardrails=[always_fine] if guarded else [], tool_use_behavior=StopAtTools(stop_at_tool_names=['finish']))
    with tempfile.TemporaryDirectory() as directory:
        database = str(Path(directory)/'session.sqlite')
        session = SQLiteSession('fictional', database)
        try:
            first = await run(agent, 'fictional request', session, streamed)
            assert len(first.interruptions)==1
            state = first.to_state()
            if serialized:
                state = await RunState.from_json(agent, json.loads(json.dumps(state.to_json())))
            state.approve(state.get_interruptions()[0])
            final = await run(agent, state, session, streamed)
            assert final.final_output == 'done'
        finally:
            session.close()
        reader = SQLiteSession('fictional', database)
        try:
            items = await reader.get_items()
        finally:
            reader.close()
    calls = {i.get('call_id') for i in items if i.get('type')=='function_call'}
    orphans = [i.get('call_id') for i in items if i.get('type')=='function_call_output' and i.get('call_id') not in calls]
    return dict(streamed=streamed, serialized=serialized, output_guardrail=guarded, orphan_call_ids=orphans, persisted_types=[i.get('type') or i.get('role') for i in items])

async def main():
    rows = []
    for guarded in (False, True):
        for streamed in (False, True):
            for serialized in (False, True):
                rows.append(await case(streamed, serialized, guarded))
    report = {'cases':rows, 'orphan_cases':sum(bool(r['orphan_call_ids']) for r in rows), 'live_api_calls':0, 'scope':'ScriptedModel with fictional tool results and temporary SQLite databases reopened for persistence checks.'}
    print(json.dumps(report))

asyncio.run(main())

Diagnostic assistance: prepared with Codex and independently executed against the pinned source; all inputs are fictional.

@dpiet-oai dpiet-oai left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed e63e1af against merge base d7e52c3. No actionable correctness or compatibility findings in this bounded static review of the complete diff and relevant runtime context.

Both runners preserve deferred approval history through serialized and detached resumes while retaining handoff filtering, current-response redaction, recoverable settlement and response-owned storage policy. Callback recovery records committed outputs in model order; the regression deliberately reverses completion order and verifies ordered Session history after serialized retry in both runners. Schema 1.19 correctly separates the held-write fields from released 1.18 compaction metadata.

Source-only review. No code, tests, builds or live backend probes were executed locally. Exact-head hosted Tests and CodeQL workflows passed, including Windows and native macOS coverage. All 21 returned jobs were successful; Release Readiness's substantive validation steps were skipped, so its green result is not evidence of executed release validation.

@jbeckwith-oai
jbeckwith-oai dismissed seratch’s stale review October 6, 2026 17:04

I am going to land this, and we can deal with aftermath later

@jbeckwith-oai
jbeckwith-oai merged commit c757a43 into openai:main Oct 6, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

8 participants