test: test_with_llm failing on master - missing on_llm_start/on_llm_end events

[zhangzhefang-github]

Description

The test test_with_llm in libs/core/tests/unit_tests/runnables/test_runnable_events_v1.py is failing on master branch. The test expects on_llm_start and on_llm_end events to be emitted when using FakeStreamingListLLM with astream_events, but these events are not being generated.

Steps to Reproduce

1. Checkout the latest master branch
2. Run the test:

   cd libs/core
   pytest tests/unit_tests/runnables/test_runnable_events_v1.py::test_with_llm -xvs

Expected Behavior

The test expects 9 events to be emitted:

1. on_chain_start (RunnableSequence)
2. on_prompt_start (my_template)
3. on_prompt_end (my_template)
4. on_llm_start (FakeStreamingListLLM) ← Missing
5. on_llm_end (FakeStreamingListLLM) ← Missing
6. on_chain_stream (chunk "a")
7. on_chain_stream (chunk "b")
8. on_chain_stream (chunk "c")
9. on_chain_end (RunnableSequence)

Actual Behavior

Only 7 events are emitted (missing events 4 and 5 above).

Error Message

AssertionError: assert 7 == 9
  +  where 7 = len([...actual events...])
  +  and   9 = len([...expected events...])

Additional warning in logs:

WARNING  langchain_core.callbacks.manager:manager.py:411 Error in LogStreamCallbackHandler.on_llm_end callback: TracerException('No indexed run ID ...')

Environment

• Affected versions: All Python versions (3.10-3.14) and all Pydantic versions (2.7.0-2.12.0)
• Test file: libs/core/tests/unit_tests/runnables/test_runnable_events_v1.py
• Line: Around line 1732-1800

Impact

This test failure is currently blocking CI for multiple PRs, as it runs across:

• 5 Python versions for libs/core
• 6 Pydantic versions for libs/core
• 2 Python versions for libs/langchain
• 6 Pydantic versions for libs/langchain
• 1 for libs/langchain_v1

Total: 21 CI job failures from this single test.

Additional Context

This appears to be a pre-existing issue on master, not introduced by recent PRs. The test may have been passing before but a recent change to the event streaming logic may have broken it.

Discovered while working on #34060 - all tests related to that PR pass except for this pre-existing failure.

Possible Solutions

1. Fix the event emission logic in FakeStreamingListLLM or the event streaming handler to ensure on_llm_start/on_llm_end are properly emitted
2. If this behavior is intentional, update the test expectations to match the actual behavior
3. Mark this test as xfail until it can be properly fixed

Would appreciate guidance from maintainers on the preferred approach.