feat: export LLM traces for all call sites, not just deriver#529
feat: export LLM traces for all call sites, not just deriver#529
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
WalkthroughThree files are modified to add Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
[Summary]
[Motivation]
Previously, only the minimal deriver and dialectic chat paths emitted traces. This meant there was no visibility into LLM inputs/outputs for summarization or dream cycle calls which makes it harder to debug memory quality, audit model behavior, or benchmark across the full pipeline. With this change, setting a single env var captures every generative LLM call with its module-level tag, input/output pairs, token counts, and tool call history.
This PR will ensure that we have a standardized way to get the traces data we need for training models, benchmarking (also affects excadrill), and doing downstream analysis for different modules. This can enable easy benchmarking for things like the summary, dreamer, etc as per the wishlist.
[Traced call sites]
minimal_deriverdialectic_chatdreamer_deductiondreamer_inductionshort_summarylong_summary[Test plan]
Summary by CodeRabbit