Gemma curation benchmark data

Human-curated gold-standard curation for a set of GEO expression experiments loaded in Gemma, released as a benchmark for evaluating automated curation of experimental design (factors + factor values) and whole-experiment annotations (tags).

This repository is a living, versioned dataset — content is refined over time and released under git tags. Pin a tag (e.g. v0.1) for reproducible evaluation.

Contents

The data comes as two disjoint curated splits: a 400-experiment set and a 100-experiment set (test-100). Both are gold; report on whichever fits your evaluation.

Path	What it is
gold/polished_gold_400.jsonl	The 400 set — 400 curated experiments, one JSON record per line
gold/polished_gold_400.meta.json	Provenance, record schema, sha256, counts, status
gse_list_400.txt	The 400 accessions (one per line)
gold/polished_gold_test100.jsonl	The test-100 set — 100 curated experiments
gold/polished_gold_test100.meta.json	Provenance, record schema, sha256, counts, status
gse_list_test100.txt	The 100 test-100 accessions (one per line)
metadata/difficulty_flags_400.json	Per-experiment difficulty signals + tally (400 set)
metadata/hard_members.json	The hard / moderate difficulty subsets, for stratified eval (400 set)
metadata/corpus_summary.json	Corpus statistics: species, topics, tag/factor distributions (400 set)

The metadata/ files (difficulty, corpus statistics) cover the 400 set only — they draw on GEO series metadata beyond the curated labels shipped here.

The two splits are disjoint, so results can be reported per-split (400 and test-100 separately).

Record format

Both gold files (polished_gold_400.jsonl and polished_gold_test100.jsonl) use the same record format. Each line is one experiment:

{
  "accession": "GSE10061",
  "experiment_id": 1181,
  "gold": {
    "factors": [
      {"name": "...", "category": "...", "description": "...",
       "type": "...", "factorValues": [ ... ]}
    ],
    "tags": [
      {"category": "...", "value": "...", "statements": [ ... ], ...}
    ]
  }
}

• accession — the GEO series accession. In the dev set a few series are curated as split sub-series (GSExxxxx.1 / .2), so the 400 records span 393 distinct base accessions; the test-100 has no split sub-series (100 records, 100 distinct accessions).
• experiment_id — Gemma's internal numeric id, for cross-referencing the dataset in Gemma.
• gold.factors — the curated experimental design: the factors that vary across samples, their factor values, and value-level statements.
• gold.tags — curated whole-experiment annotations (EEtags): properties constant across all profiled samples (disease, study design, disease model, …).

The experiment metadata itself (samples, GEO description, raw characteristics) is not shipped here — fetch it from Gemma or GEO by accession. This repository carries only the human-curated labels.

Status & provenance

Both sets are produced from human curator consensus in Gemma (curators Cy and Am). Internal build provenance and per-row export filenames have been removed for public release; see each file's *.meta.json for the retained provenance, sha256, and a status note. The dev-400 snapshot is flagged INTERIM (actively being polished); the test-100 is post-tiebreak (its two-curator design disagreements are resolved and folded in). The repository is versioned: pin a tag for a stable reference.

License

CC BY-NC 4.0 — Creative Commons Attribution-NonCommercial 4.0 International. You may share and adapt this data for non-commercial purposes with attribution. See LICENSE.

Citation

Please cite the Gemma curation-agents paper (in preparation) and this dataset by its tagged version. From the Pavlidis Lab, UBC.