Evaluate language models with lm-eval-harness and build a custom benchmark 💻📓
-
Updated
Oct 14, 2026 - Python
#
lm-eval-harness
Here are 10 public repositories matching this topic...
Lightweight NLP/LLM evaluation toolkit — metrics, LLM-as-Judge, significance testing, prompt robustness
-
Updated
Jul 1, 2026 - Python
Author verifiable eval records through a draft → review → revise → submit loop with server-enforced graders; compile to JSONL/CSV/Inspect/lm-eval via MCP. STDIO or Streamable HTTP.
typescript mcp evaluation grader evals model-context-protocol rlvr verifiable-rewards inspect-ai lm-eval-harness cyanheads
-
Updated
Oct 17, 2026 - TypeScript
Task-dependent benchmark gap between /v1/completions and /v1/chat/completions on instruction-tuned LLMs -- two case studies on Qwen3.x GGUFs, reproduction recipe, and probes.
benchmarking methodology instruction-following llm llama-cpp qwen gguf chat-completions lm-eval-harness ifeval
-
Updated
May 16, 2026 - Python
Fine-tuning pipeline for Qwen2.5-Coder-1.5B using QLoRA, DPO alignment, and AWQ quantization on free-tier Colab GPU.
python code python3 lora fine-tuning finetuning sft dpo huggingface huggingface-transformers awq llm lm-evaluation-harness bitsandbytes finetuning-llms awq-quantization qwen2-5-coder lm-eval lm-eval-harness
-
Updated
Aug 19, 2026 - Jupyter Notebook
Offline Self-Evolve pipeline using synthetic data generation — evaluate before/after with Rule+LLM Judge and lm-eval harness in measured workloads.
evaluation pytorch developer-tools data-generation harness synthetic-data llm llm-evaluation llm-judge self-evolution offline-evaluation lm-eval-harness
-
Updated
Oct 2, 2026 - Python
A custom lm-eval adapter built from scratch — implementing the full LM interface against Ollama's native API rather than relying on any off-the-shelf integration.
-
Updated
Mar 13, 2026 - Python
Custom evaluation harness with domain-specific benchmarks for enterprise LLMs
-
Updated
May 9, 2026 - Python
Evaluation framework for self-hosted LLMs. Systematic prompt ablation (baseline, CoT, few-shot, self-consistency voting) on Llama 3.1 8B via lm-evaluation-harness, with Wilson CI statistical analysis, determinism validation, and load testing under concurrency. Found chain-of-thought degrades accuracy 25pp at small scale.
benchmarking benchmark machine-learning natural-language-processing load-testing self-hosted statistical-analysis llama language-models reproducibility self-consistency determinism ablation-study llm prompt-engineering chain-of-thought evals ollama llm-evaluation lm-eval-harness
-
Updated
Aug 16, 2026 - Python
Apple Silicon LLM benchmark harness for local runtimes and OpenAI-compatible endpoints: speed, memory, eval coverage, and public reports.
python macos benchmark gemma mlx huggingface streamlit apple-silicon llm llama-cpp gguf lm-eval-harness
-
Updated
May 22, 2026 - Python
Add this topic to your repo
To associate your repository with the lm-eval-harness topic, visit your repo's landing page and select "manage topics."