human-annotation

YEDDA: A Lightweight Collaborative Text Span Annotation Tool. Code for ACL 2018 Best Demo Paper Nomination.

Data release for the ImageInWords (IIW) paper.

Video Generation Benchmark

[NeurIPS'25] MLLM-CompBench evaluates the comparative reasoning of MLLMs with 40K image pairs and questions across 8 dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. CompBench covers diverse visual domains, including animals, fashion, sports, and scenes

Code for the AAAI 2023 Paper "Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text"

Color-matching makeup shades using k-means clustering and Claude multimodal LLM, evaluated with Delta E / CIELAB. Includes human annotation pipeline and Streamlit app.

Repository of the paper "Supporting Online Toxicity Detection with Knowledge Graphs" (ICWSM 22).

A collection of MTurk templates designed to make complex tasks easier for human annotators.

Code and data for inherent biases in grammatical error correction and text simplification, including 2,500 human-written correction and simplification tasks.

Human-annotated NLP analysis of 262 Phasmophobia Steam reviews with blinded labeling, duplicate-aware validation, TF-IDF models, and an interactive dashboard.

Repository for the journal article 'SHAMSUL: Systematic Holistic Analysis to investigate Medical Significance Utilizing Local interpretability methods in deep learning for chest radiography pathology prediction'

Design-aware temporal reliability auditing for human annotation studies

Code and data for "Adaptation, Not Algorithm" (Findings of AACL-IJCNLP 2026): LoRA vs. full fine-tuning and jailbreak safety in five open-weight LLMs

Code repository for the paper "Estimating Ground Truth in a Low-labelled Data Regime: A Study of Racism Detection in Spanish" (NEATClasS 2022)

A dataset-auditing pipeline that flags and reviews label-quality issues across three sentiment classification datasets: the human-annotated SST-2 benchmark, and two LLM-generated synthetic datasets

Which leaderboard orderings survive a human reader? Verifier audit of 35,748 MATH-Hard responses, 400-item human audit, and certified pairwise orderings for 27 Open LLM Leaderboard models.

A public blackmail scorer fired on 0 of 669 responses (keyword gate); without the gate its grader is right 20 of 47.

Human-annotated benchmark for the completeness and relevance of long answers (Language Resources and Evaluation, 2026)