NoBS benchmark

📚 A collection of AI hardware benchmarks

No magic scores. No-BullShit.

Compare M1 Air, M4 Max, RTX 3060, RTX 4060, A100 and other hardware on different AI tasks

---

Table of Contents

• Overview
• Philosophy
• Benchmark Results
  • Overall Ranking
  • By GPU Vendor
  • Detailed Performance
• Quick Start
• Contributing

---

Overview

NoBS is an open-source benchmark suite for evaluating real AI hardware performance — not synthetic FLOPS or polished demos.

It's a collection of reproducible tests and community-submitted results for:

• 🧩 Embeddings — ✅ Ready (sentence-transformers, IMDB dataset)
• 💬 LLM inference — 🚧 In Progress (LM Studio awailable, Awesome Prompts dataset)
• 👁️ VLM inference — 📋 Planned
• 🎨 Diffusion image generation — 📋 Planned
• 🔬 Classic ML — 📋 Planned (scikit-learn, XGBoost, LightGBM, Catboost)

---

Philosophy

"We don’t measure synthetic FLOPS. We measure how your GPU cries in real life."

NoBS was built to understand how different devices — from everyday laptops and PCs to large inference giants — actually perform on real AI tasks.

---

Benchmark Results

Last Updated: 2025-10-20

🏆 Overall Ranking

Rank	Device	Platform	CPU	RAM	GPU	VRAM	Embeddings	LLM	Total Score
🥇 1	Mac16,6	🍏 macOS	Apple M4 Max (14)	36 GB	Apple M4 Max (32 cores)	shared with system RAM	637.17	157.84	795.01
🥈 2	ASUSTeK COMPUTER INC. ASUS Vivobook Pro 15 N6506MV_N6506MV 1.0	🐧 Linux	Intel(R) Core(TM) Ultra 9 185H (16)	23 GB	NVIDIA GeForce RTX 4060 Laptop GPU	8 GB	539.73	26.42	566.15

📊 By GPU Vendor

⚫ Apple (1 device)

Rank	Device	Platform	CPU	RAM	GPU	VRAM	Embeddings	LLM	Total Score
🥇 1	Mac16,6	🍏 macOS	Apple M4 Max (14)	36 GB	Apple M4 Max (32 cores)	shared with system RAM	637.17	157.84	795.01

🟢 NVIDIA (1 device)

Rank	Device	Platform	CPU	RAM	GPU	VRAM	Embeddings	LLM	Total Score
🥇 1	ASUSTeK COMPUTER INC. ASUS Vivobook Pro 15 N6506MV_N6506MV 1.0	🐧 Linux	Intel(R) Core(TM) Ultra 9 185H (16)	23 GB	NVIDIA GeForce RTX 4060 Laptop GPU	8 GB	539.73	26.42	566.15

📈 Detailed Performance

Text Embeddings (100 IMDB samples)

Device	Model	Rows/sec	Time (s)	Embedding Dim	Batch Size
ASUSTeK COMPUTER INC. ASUS Vivobook Pro 15 N6506MV_N6506MV 1.0	nomic-ai/modernbert-embed-base	36.24	2.76	768	16
ASUSTeK COMPUTER INC. ASUS Vivobook Pro 15 N6506MV_N6506MV 1.0	thenlper/gte-large	25.57	3.91	1024	16
Mac16,6	nomic-ai/modernbert-embed-base	36.30	2.76	768	16
Mac16,6	thenlper/gte-large	34.55	2.89	1024	16

LLM Inference (3 prompts from awesome-chatgpt-prompts)

Device	Model	Tokens/sec	TTFT (s)	Latency (s)	Input Tokens	Output Tokens
ASUSTeK COMPUTER INC. ASUS Vivobook Pro 15 N6506MV_N6506MV 1.0	gpt-oss-20b	16.70	27.87	136.28	561	3443
Mac16,6	gpt-oss-20b	168.14	6.50	22.81	561	4280

---

All metrics are median values across 3 runs. Scores calculated as: num_tasks * 3600 / total_time_seconds.

⚡ Quick Start

Prerequisites

• Python 3.12+
• uv package manager

Installation

# Clone the repository
git clone https://github.com/bogdanminko/nobs.git
cd nobs

# Install dependencies
uv sync

Running Benchmarks

Run all benchmarks

uv run python main.py

This will:

1. Auto-detect your hardware (CUDA/MPS/CPU)
2. Run all available benchmarks (currently: embeddings)
3. Save results to results/report_{your_device}.json

Run specific benchmarks

# Embeddings only
uv run python -m src.tasks.text_embeddings.runner

# LLM inference (requires LM Studio running on localhost:1234)
uv run python -m src.tasks.llms.runner

LLM Benchmarks Setup

Note: LLM benchmarks currently require LM Studio running locally.

1. Download and install LM Studio
2. Load a model in LM Studio
3. Start the local server (default: http://localhost:1234)
4. Run the LLM benchmark:

   uv run python -m src.tasks.llms.runner

---

🤝 Contributing

We welcome contributions! Whether it's adding new benchmarks, supporting new models, or submitting your hardware results.

Development Setup

1. Fork and clone the repository

   git clone https://github.com/YOUR_USERNAME/nobs.git
   cd nobs

2. Install dependencies including dev tools

   uv sync --group quality --group dev

3. Install pre-commit hooks

   pre-commit install

   This sets up automatic code quality checks that run before each commit:

   • ruff — Fast Python linter and formatter
   • mypy — Static type checking
   • bandit — Security vulnerability scanner
   • Standard checks (trailing whitespace, YAML syntax, etc.)

Making Changes

1. Create a new branch

   git checkout -b feature/your-feature-name

2. Make your changes

   • Write code following the existing patterns
   • Add type hints where applicable
   • Update documentation if needed

3. Test your changes

   # Run benchmarks to ensure they work
   uv run python main.py
   
   # Update benchmark results tables (if you modified results)
   make
   
   # Run code quality checks manually (optional - pre-commit will run them automatically)
   make format

   Available Makefile commands:

   • make — Generate benchmark results tables (default)
   • make generate — Generate benchmark results tables
   • make format — Run pre-commit hooks on all files
   • make lint — Run ruff linter only
   • make clean — Clean Python cache files
   • make help — Show all available commands

4. Commit your changes

   git add .
   git commit -m "feat: your descriptive commit message"

   Pre-commit hooks will automatically:

   • Format your code
   • Check for type errors
   • Scan for security issues
   • Fix common issues (trailing whitespace, etc.)

   If any check fails, fix the issues and commit again.

5. Push and create a Pull Request

   git push origin feature/your-feature-name

Code Quality Standards

All contributions must pass:

• ✅ Ruff linting and formatting
• ✅ Mypy type checking
• ✅ Bandit security checks

These are enforced automatically via pre-commit hooks.

Adding New Benchmarks

See CLAUDE.md for detailed instructions on:

• Adding new models to existing benchmarks
• Creating new benchmark categories
• Data loading patterns
• Memory management best practices

---