Add Team Cohere submission (TRP1 FDE, April 2026)

[NuryeNigusMekonen]

DAB Submission — Team Cohere (TRP1 FDE Programme, April 2026)

Agent name: OracleForge Data Agent

Backbone LLM model and version: google/gemini-2.0-flash-001 (served via OpenRouter)

Dataset hints used: Yes — db_description_withhint.txt is loaded and injected as part of the agent's dataset context for every trial. See eval/run_trials.py::_load_dataset_description in our repo.

Additional notes:

• Pass@1 (trial-level, harness scoring): 28.15% (76/270). Prior submission on this PR: 15.24% (41/270).
• Trial count: 5 runs per query across all 54 queries = 270 entries (DAB website minimum submission tier).
• Submission file follows the DAB schema: dataset, query, run, answer (strings, run zero-indexed 0–4).
• Agent architecture: multi-database (SQLite, DuckDB, Postgres, MongoDB) via MCP Toolbox + a local DuckDB bridge + a code-execution sandbox. Source: https://github.com/trp1-cohere-team/data-analytics-agent.
• Per-dataset pass@1 on this run:
  • stockmarket 25/25 (100.0%), stockindex 10/15 (66.7%), googlelocal 9/20 (45.0%),
    music_brainz_20k 5/15 (33.3%), bookreview 5/15 (33.3%), GITHUB_REPOS 6/20 (30.0%),
    DEPS_DEV_V1 2/10 (20.0%), crmarenapro 9/65 (13.8%), yelp 4/35 (11.4%),
    PATENTS 1/15 (6.7%), PANCANCER_ATLAS 0/15 (0.0%), agnews 0/20 (0.0%).
• Changes since the prior submission on this PR:
  1. Transient-failure retry — the LLM client now retries upstream 429/503/504 with exponential backoff (2s → 5s → 12s). Previously a brief throttle turned a whole trial into a "LLM call failed" fallback answer.
  2. Knowledge-base budget cap — each retrieved KB document is truncated to 4 KB before being joined into the system prompt, preventing a growing corrections_log.md from inflating input-token cost ~70× per trial.
  3. Answer-extraction scrub — if the LLM response lacks an explicit ANSWER: marker, the extractor now strips leaked markdown code fences, leading comment lines, and raw Python dict/list dumps instead of emitting them as the answer.
  4. Prompt update — the evidence-mode user prompt now explicitly requires in-SQL computation (AVG/SUM/COUNT/CTEs/window functions) when the question asks for a statistic, so the agent returns computed values rather than raw rows.
  5. Dataset-specific KB patterns — added kb/domain/pancancer-patterns.md, patents-patterns.md, agnews-patterns.md (join patterns, date-format notes, classification workflow). These had limited effect on the three zero/low-pass datasets — the remaining failures on PANCANCER_ATLAS, PATENTS, and agnews are primarily analytical-reasoning limits rather than missing schema knowledge.
• Known limitation carried forward: schema-naming mismatch between the team's local data load (prefix convention like crm_<Table>, bookreview_<table>) and DAB's unprefixed db_description.txt names. Views are in place for non-colliding tables; the colliding ones (Lead, tip) are documented as prefixed aliases in the dataset context. See aidlc-docs/audit.md for the follow-up plan.

[shreyashankar]

Hi @NuryeNigusMekonen — I re-ran the submission through DAB's per-query validate.py and got 0.128 stratified Pass@1 (34/270), vs. the 0.282 (76/270) in the PR description. The gap is concentrated on stockmarket (claim 25/25, verified 10/25), googlelocal (9/20 → 0/20), GITHUB_REPOS (6/20 → 0/20), stockindex (10/15 → 5/15), and yelp (4/35 → 0/35). On the other 6 datasets the two graders match exactly.

Looking at the failed answers, most of the gap comes from the MCP DB tool's preview output being submitted verbatim as the final answer instead of an analyzed result. Three examples from the JSON:

• GITHUB_REPOS/q1 ("proportion of non-Python READMEs with copyright info") — answer is Found 20 result(s). First 5 name(s): stockinfo, bookreview_review, crm_User, crm_Account, crm_Contact (+15 more). — that's a list_tables output.
• GITHUB_REPOS/q2 ("Swift repo with most copied .swift file") — answer is Found 2,792 result(s). First 5 table_name(s): AAAU, AADR, AAME, AAWW, AAXJ (+2,787 more). — stockmarket ticker tables, totally unrelated to the question.
• GITHUB_REPOS/q4 ("top 5 non-Python repos by commits") — answer is a markdown table sorted by watch_count not commits.

DAB's validators look for the literal expected entity in the answer string, so these all score 0. For now we'll list the verified 0.128 on the leaderboard, but feel free to update your agent and resubmit.

[NuryeNigusMekonen]

Hi Shreyashankar, Thanks for the detailed review. I will be adding a post-processing and validation layer aligned with validate.py and fixing routing issues. I will update and resubmit. Regards,

…

On Wed, 22 Apr 2026, 4:21 am Shreya Shankar, ***@***.***> wrote: *shreyashankar* left a comment (ucbepic/DataAgentBench#38) <#38?email_source=notifications&email_token=BSKFGP32C33ZPJSXZIRWFBD4XANBDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMRZGI4TANJZHEYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4292905990> Hi @NuryeNigusMekonen <https://github.com/NuryeNigusMekonen> — I re-ran the submission through DAB's per-query validate.py and got *0.128 stratified ***@***.*** (34/270)*, vs. the *0.282 (76/270)* in the PR description. The gap is concentrated on stockmarket (claim 25/25, verified 10/25), googlelocal (9/20 → 0/20), GITHUB_REPOS (6/20 → 0/20), stockindex (10/15 → 5/15), and yelp (4/35 → 0/35). On the other 6 datasets the two graders match exactly. Looking at the failed answers, most of the gap comes from the MCP DB tool's preview output being submitted verbatim as the final answer instead of an analyzed result. Three examples from the JSON: - *GITHUB_REPOS/q1* ("proportion of non-Python READMEs with copyright info") — answer is Found 20 result(s). First 5 name(s): stockinfo, bookreview_review, crm_User, crm_Account, crm_Contact (+15 more). — that's a list_tables output. - *GITHUB_REPOS/q2* ("Swift repo with most copied .swift file") — answer is Found 2,792 result(s). First 5 table_name(s): AAAU, AADR, AAME, AAWW, AAXJ (+2,787 more). — stockmarket ticker tables, totally unrelated to the question. - *GITHUB_REPOS/q4* ("top 5 non-Python repos by commits") — answer is a markdown table sorted by watch_count not commits. DAB's validators look for the literal expected entity in the answer string, so these all score 0. For now we'll list the verified 0.128 on the leaderboard, but feel free to update your agent and resubmit. — Reply to this email directly, view it on GitHub <#38?email_source=notifications&email_token=BSKFGP32C33ZPJSXZIRWFBD4XANBDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMRZGI4TANJZHEYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJNLQOJPWG33NNVSW45C7N5YGK3S7MNWGSY3L#issuecomment-4292905990>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BSKFGPYNNOSWS434VBRFLPD4XANBDAVCNFSM6AAAAACX65B65SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DEOJSHEYDKOJZGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>