Flaky E2E test in rpc_session_state.e2e.test.ts (model switchto) #1629

Activity

1. github-actions

   added

   bug

   on Jun 11, 2026
2. github-actions commented on Jun 11, 2026

   

   github-actionsbot

   on Jun 11, 2026 – with GitHub Actions

   Contributor

   More actions

   Investigation findings

   Verdict: confirmed bug — the test should call session rpc model switchto in nodejs/test/e2e/rpc_session_state.e2e.test.ts has two distinct problems that together explain the flaky behaviour on Windows CI.

   1. Wrong assertion (expect(after.modelId).toBe(before.modelId))

   After calling session.rpc.model.switchTo({ modelId: "gpt-4.1", ... }), the test calls session.rpc.model.getCurrent() and expects the result to still equal the original model:

   expect(after.modelId).toBe(before.modelId);   // expects OLD model — wrong

   This directly contradicts:

   • The skipped sibling test in rpc.e2e.test.ts (line 104), which was written to test the same feature and correctly expects the new model:

     expect(after.modelId).toBe("gpt-4.1");   // expects NEW model — correct

   • The behaviour confirmed by session.e2e.test.ts tests (should set model on existing session, should set model with reasoningEffort), which verify via the session.model_change event that the model is updated to the switched value.

   The wrong assertion is what makes the test flaky: because the CLI may update its internal state and emit the session.model_change event asynchronously after returning the JSON-RPC response to switchTo, getCurrent() sometimes races with that state update. When getCurrent() wins the race (state not yet updated) the wrong assertion accidentally passes; when the state update wins (more likely under different OS scheduling on Windows), getCurrent() returns "gpt-4.1" and the wrong assertion fails.

   2. Missing event-subscription pattern (race condition)

   session.e2e.test.ts subscribes to session.model_change before triggering the switch:

   // nodejs/test/e2e/session.e2e.test.ts:960
   const modelChangePromise = getNextEventOfType(session, "session.model_change");
   await session.setModel("gpt-4.1");
   const event = await modelChangePromise;

   The rpc_session_state test never subscribes to this event at all. The correct pattern for verifying the post-switch state would be to wait for the model_change event (to confirm the CLI has committed the switch) before calling getCurrent().

   Files investigated

   File	Relevance
   nodejs/test/e2e/rpc_session_state.e2e.test.ts (lines 52–71)	Bug location — active test with wrong assertion
   nodejs/test/e2e/rpc.e2e.test.ts (lines 84–105)	Skipped test with the correct expect(after.modelId).toBe("gpt-4.1") assertion
   nodejs/test/e2e/session.e2e.test.ts (lines 956–981)	Reference for correct subscribe-before-switch pattern
   test/snapshots/rpc_session_state/should_call_session_rpc_model_switchto.yaml	Snapshot only has claude-sonnet-4.5; gpt-4.1 works because the CLI does not validate model IDs against the /models endpoint

   Suggested fix

   1. Subscribe to session.model_change before calling switchTo.
   2. After awaiting the event, call getCurrent() and assert after.modelId === "gpt-4.1".
   3. Remove or move the it.skip in rpc.e2e.test.ts once the active test is fixed (it already has the correct assertions).

   Generated by Bug Handler for issue #1629 · sonnet46 2.4M · ◷
3. patniko commented on Sep 8, 2026

   

   patniko

   on Sep 8, 2026

   Contributor

   More actions

   Closing as fixed after verification against the current public SDK sources and bundled behavior.

   Public evidence

   • #1622 — [E2E] session.todos_changed event + readSqlTodosWithDependencies (6 languages) (merged 2026-06-15)
   • commit bd7b50a17628 — [E2E] session.todos_changed event + readSqlTodosWithDependencies (6 languages) ([E2E] session.todos_changed event + readSqlTodosWithDependencies (6 languages) #1622)
   • nodejs/test/e2e/rpc_session_state.e2e.test.ts — createSdkTestContext; switchTo; getCurrent
   • nodejs/test/e2e/rpc_session_state.e2e.test.ts — should call session rpc model switchto

   The reported behavior/request is implemented at the current SDK baseline, so this issue is being closed as completed.

   Public SDK baseline: github/copilot-sdk d5c9d06d8c41.
4. patniko

   closed this as completedon Sep 8, 2026