🔥 A list of tools, frameworks, and resources for building AI web agents
-
Updated
Jul 25, 2025
#
llms-benchmarking
Here are 81 public repositories matching this topic...
An extensible benchmark for evaluating large language models on planning
-
Updated
Sep 17, 2025 - PDDL
A comprehensive set of LLM benchmark scores and provider prices.
-
Updated
Sep 15, 2025 - JavaScript
[NeurIPS 2025] BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models
-
Updated
Sep 20, 2025 - Python
What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks
-
Updated
Jul 26, 2024 - Jupyter Notebook
Benchmark that evaluates LLMs using 759 NYT Connections puzzles extended with extra trick words
-
Updated
Sep 20, 2025 - Python
How good are LLMs at chemistry?
-
Updated
Sep 11, 2025 - Python
Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
metrics good-first-issue llm prompt-engineering generative-ai llmops llm-eval llm-tools llm-evaluation llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework
-
Updated
Feb 13, 2025 - Python
Thematic Generalization Benchmark: measures how effectively various LLMs can infer a narrow or specific "theme" (category/rule) from a small set of examples and anti-examples, then detect which item truly fits that theme among a collection of misleading candidates.
-
Updated
Sep 22, 2025
Language Model for Mainframe Modernization
-
Updated
Aug 23, 2024 - Python
Develop reliable AI apps
-
Updated
Sep 2, 2025 - Python
[NeurIPS'25] MLLM-CompBench evaluates the comparative reasoning of MLLMs with 40K image pairs and questions across 8 dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. CompBench covers diverse visual domains, including animals, fashion, sports, and scenes
benchmark reasoning vision-and-language multimodal-deep-learning human-annotation foundation-models large-language-models llms vision-language-model multimodal-large-language-models evaluation-llms llms-benchmarking neurips-2024
-
Updated
Apr 21, 2025 - Jupyter Notebook
Awesome Mixture of Experts (MoE): A Curated List of Mixture of Experts (MoE) and Mixture of Multimodal Experts (MoME)
artificial-intelligence sparse moe load-balancing multimodal-learning mixture-of-experts mome gating-network foundation-models large-language-models llms large-language-model large-vision-language-models expert-network llms-reasoning llms-benchmarking mixtrure-of-multimodal-experts sparse-moe sparse-mixture-of-experts sparse-mixture-of-multimodal-experts
-
Updated
Jul 20, 2025
-
Updated
Mar 14, 2025 - Python
Training and Benchmarking LLMs for Code Preference.
-
Updated
Nov 15, 2024 - Python
The data and implementation for the experiments in the paper "Flows: Building Blocks of Reasoning and Collaborating AI".
ai competitive-programming agents competitive-programming-contests competitive-coding llms llms-reasoning llms-benchmarking aiflows
-
Updated
Feb 12, 2024 - Python
Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks
-
Updated
Aug 6, 2025 - Jupyter Notebook
Restore safety in fine-tuned language models through task arithmetic
-
Updated
Mar 28, 2024 - Python
A minimalist benchmarking tool designed to test the routine-generation capabilities of LLMs.
-
Updated
Nov 28, 2024 - Python
Improve this page
Add a description, image, and links to the llms-benchmarking topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the llms-benchmarking topic, visit your repo's landing page and select "manage topics."