agentic: add answer-mode retrieval core (contract, evaluator, baselines)#4869
Open
AdnanElAssadi56 wants to merge 2 commits into
Open
agentic: add answer-mode retrieval core (contract, evaluator, baselines)#4869AdnanElAssadi56 wants to merge 2 commits into
AdnanElAssadi56 wants to merge 2 commits into
Conversation
AdnanElAssadi56
force-pushed
the
feat/agentic-answer-core
branch
from
c2c2cd4 to
cef55cf
Compare
Samoed
reviewed
Jul 12, 2026
Samoed
left a comment
Samoed
left a comment
Member
There was a problem hiding this comment.
Currently a bit hard to judge without full pipeline
| self.questions = questions | ||
| self.references = references | ||
| self.corpus = corpus | ||
| self.judge = judge |
Member
There was a problem hiding this comment.
I don't think we need an llm judge, because we can just compare ids. With llm answer and cheking it I think this is too much LLM checking
| per_question: list[dict[str, Any]] | ||
|
|
||
|
|
||
| class AnswerEvaluator: |
Member
There was a problem hiding this comment.
I think we could use some tools, but I don't have, but I don't think we will have much
| from __future__ import annotations | ||
|
|
||
|
|
||
| class InMemoryCorpus: |
Member
There was a problem hiding this comment.
I think we can reuse our existing indexes for that
| from collections.abc import Sequence | ||
|
|
||
| # Provider-agnostic chat message. | ||
| Message = dict[str, str] |
Member
There was a problem hiding this comment.
I think we would need to add multimodal support in future
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First PR of the
mteb.agenticsubsystem (see #4868 ).AnswerSystem,CorpusHandle,ChatModel,AnswerResult,UsageAnswerEvaluator: runs a system per query, scores accuracy / cost / latencyExactMatchJudge,LLMJudgeClosedBookSystem(floor),OracleContextSystem(ceiling)from_mteb_retrievaldata adaptertests/test_agentic.py)