Skip to content

perf(eval): decouple eval from response flow โ€” async fire-and-forget#549

Merged
mangowhoiscloud merged 2 commits intodevelopfrom
feat/chat-eval-pipeline
Feb 10, 2026
Merged

perf(eval): decouple eval from response flow โ€” async fire-and-forget#549
mangowhoiscloud merged 2 commits intodevelopfrom
feat/chat-eval-pipeline

Conversation

@mangowhoiscloud
Copy link
Contributor

Summary

  • Eval Pipeline์„ ๋ฉ”์ธ ๊ทธ๋ž˜ํ”„์—์„œ ๋ถ„๋ฆฌํ•˜์—ฌ done ์ด๋ฒคํŠธ ๋ฐœํ–‰ ํ›„ asyncio.create_task๋กœ ๋น„๋™๊ธฐ ์‹คํ–‰
  • ์‚ฌ์šฉ์ž ์ฒด๊ฐ ์ง€์—ฐ ์ œ๊ฑฐ
๋ณ€๊ฒฝ ์ „: answer โ†’ eval(๋ธ”๋กœํ‚น) โ†’ done   โ† ์ŠคํŠธ๋ฆฌ๋ฐ ํ›„ ๋”œ๋ ˆ์ด
๋ณ€๊ฒฝ ํ›„: answer โ†’ done(์ฆ‰์‹œ) โ†’ eval(fire-and-forget)
  • factory.py: eval subgraph ์ œ๊ฑฐ, answer โ†’ END ๋ณต์›
  • dependencies.py: get_eval_subgraph() ๋…๋ฆฝ ์บ์‹œ, ProcessChatCommand์— ์ฃผ์ž…
  • process_chat.py: _run_eval_async() ์ถ”๊ฐ€, done ์ดํ›„ ๋น„๋™๊ธฐ ์‹คํ–‰

Test plan

  • pytest -m eval_unit โ€” 165 tests ALL PASS
  • black --check && ruff check โ€” lint clean
  • ์ŠคํŠธ๋ฆฌ๋ฐ ์™„๋ฃŒ ํ›„ ๋”œ๋ ˆ์ด ์—†์ด done ์ด๋ฒคํŠธ ์ฆ‰์‹œ ์ˆ˜์‹  ํ™•์ธ
  • ์›Œ์ปค ๋กœ๊ทธ์—์„œ Eval completed (async) ํ™•์ธ

๐Ÿค– Generated with Claude Code

mangowhoiscloud and others added 2 commits February 10, 2026 10:23
Eval Pipeline์„ ๋ฉ”์ธ ๊ทธ๋ž˜ํ”„์—์„œ ๋ถ„๋ฆฌํ•˜์—ฌ done ์ด๋ฒคํŠธ ๋ฐœํ–‰ ํ›„
asyncio.create_task๋กœ ๋น„๋™๊ธฐ ์‹คํ–‰. ์‚ฌ์šฉ์ž ์ฒด๊ฐ ์ง€์—ฐ ์ œ๊ฑฐ.

๋ณ€๊ฒฝ ์ „: answer โ†’ eval(๋ธ”๋กœํ‚น) โ†’ done
๋ณ€๊ฒฝ ํ›„: answer โ†’ done(์ฆ‰์‹œ) โ†’ eval(fire-and-forget)

- factory.py: eval subgraph ์ œ๊ฑฐ, answer โ†’ END ๋ณต์›
- dependencies.py: get_eval_subgraph() ๋ณ„๋„ ์บ์‹œ, ProcessChatCommand์— ์ฃผ์ž…
- process_chat.py: _run_eval_async() ์ถ”๊ฐ€, done ์ดํ›„ ๋น„๋™๊ธฐ ์‹คํ–‰

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- eval_config.py: eval_model default gpt-4o-mini โ†’ gpt-5.2
- e2e report: async fire-and-forget ์•„ํ‚คํ…์ฒ˜ ๋ฐ˜์˜, ์—…๊ณ„ ์กฐ์‚ฌ ์„น์…˜ ์ถ”๊ฐ€

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@mangowhoiscloud mangowhoiscloud self-assigned this Feb 10, 2026
@github-project-automation github-project-automation bot moved this to Backlog in @Ecoยฒ Feb 10, 2026
@mangowhoiscloud mangowhoiscloud merged commit db07266 into develop Feb 10, 2026
16 checks passed
@github-project-automation github-project-automation bot moved this from Backlog to Done in @Ecoยฒ Feb 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant