Benchmarking LLMs through utilization of chess

Some of the results

Related research

Bridging the Gap between Expert and Language Models: Concept-guided Chess Commentary Generation and Evaluation

Large Language Models as General Pattern Machines

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Validity Challenges in Machine Learning Benchmarks

GAIA:A BENCHMARK FOR GENERAL AI ASSISTANTS

A critical review of large language models: Sensitivity, bias, and the path toward specialized AI

Measuring General Intelligence with Generated Games

Maia-2: A Unified Model for Human-AI Alignment in Chess

Judge AI: Assessing Large Language Models in Judicial Decision- Judge AI: Assessing Large Language Models in Judicial Decision-Making

Aligning Superhuman AI with Human Behavior: Chess as a Model System

OK, I can partly explain the LLM chess weirdness now

ChessGPT: Bridging Policy Learning and Language Modeling