🔥 A list of tools, frameworks, and resources for building AI web agents
-
Updated
Jul 10, 2026 - Python
#
llms-benchmarking
Here are 52 public repositories matching this topic...
[NeurIPS 2025] BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models
-
Updated
Mar 13, 2026 - Python
Benchmark that evaluates LLMs using 759 NYT Connections puzzles extended with extra trick words
-
Updated
Jul 17, 2026 - Python
How good are LLMs at chemistry?
-
Updated
Jan 26, 2026 - Python
Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
metrics good-first-issue llm prompt-engineering generative-ai llmops llm-eval llm-tools llm-evaluation llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework
-
Updated
Feb 13, 2025 - Python
Language Model for Mainframe Modernization
-
Updated
Aug 23, 2024 - Python
-
Updated
Mar 14, 2025 - Python
Develop reliable AI apps
-
Updated
Sep 2, 2025 - Python
Training and Benchmarking LLMs for Code Preference.
-
Updated
Nov 15, 2024 - Python
🐝 SwarmBench: Benchmarking LLMs' Swarm Intelligence
-
Updated
May 21, 2025 - Python
Restore safety in fine-tuned language models through task arithmetic
-
Updated
Mar 28, 2024 - Python
The data and implementation for the experiments in the paper "Flows: Building Blocks of Reasoning and Collaborating AI".
ai competitive-programming agents competitive-programming-contests competitive-coding llms llms-reasoning llms-benchmarking aiflows
-
Updated
Feb 12, 2024 - Python
Plancraft is a minecraft environment and agent suite to test planning capabilities in LLMs
minecraft planning interactive-environments llm multimodal-large-language-models llms-benchmarking agentic-ai
-
Updated
Nov 7, 2025 - Python
A minimalist benchmarking tool designed to test the routine-generation capabilities of LLMs.
-
Updated
Nov 28, 2024 - Python
[IEEE T-BIOM] FaceXBench: Evaluating Multimodal LLMs on Face Understanding
attributes face face-recognition gender-classification facial-expression-recognition race-detection age-estimation crowd-counting face-antispoofing multimodal face-segmentation face-perception deepfake-detection headpose-estimation llms llms-benchmarking multimodal-llms
-
Updated
Jan 15, 2026 - Python
Join 15k builders to the Real-World ML Newsletter ⬇️⬇️⬇️
-
Updated
Apr 19, 2024 - Python
Belief-Bias evaluation of local LLMs
-
Updated
Jul 3, 2025 - Python
A platform that enables users to perform private benchmarking of machine learning models. The platform facilitates the evaluation of models based on different trust levels between the model owners and the dataset owners.
platform benchmarking inference secure private mpc contamination trusted-execution-environment confidential-computing large-language-models llms-benchmarking private-benchmarking ezpc
-
Updated
Apr 22, 2026 - Python
The MERIT Dataset is a fully synthetic, labeled dataset created for training and benchmarking LLMs on Visually Rich Document Understanding tasks. It is also designed to help detect biases and improve interpretability in LLMs, where we are actively working. This repository is actively maintained, and new features are continuously being added.
donut biases synthetic-dataset-generation layoutlm synthetic-dataset layoutxlm token-classification layoutlmv3 layoutlmv2 llms-benchmarking idefics2
-
Updated
Jul 16, 2025 - Python
Improve this page
Add a description, image, and links to the llms-benchmarking topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the llms-benchmarking topic, visit your repo's landing page and select "manage topics."