v1.0.2

[AronDaron]

Hardening — Strict example schema validation prevents extra-key garbage in JSONL exports

Problem

A merged dataset (3378 rows) failed to load in Unsloth Studio with:

Failed to check dataset format: An error occurred while generating the dataset
All the data files must have the same columns, but at some point there are 1
new columns ({'gpt'})

One row out of 3378 had an extra top-level gpt key alongside conversations — the model leaked an additional turn pair as a separate top-level field instead of appending it to conversations. HuggingFace Datasets / Arrow treats top-level keys as columns, so 3377 rows with one column + 1 row with two columns broke schema unification at load time.

A second case was found during audit: another row had an extra metadata key inside individual turn objects (conversations[i].metadata.rationale). Arrow tolerates this by widening the struct schema (the extra field becomes nullable), and chat-template tokenisers ignore unknown turn keys — so this one slipped silently into fine-tuning. Still: schema noise that should not be in a clean dataset.

The existing _validate_example_structure in backend/app/services/job_runner.py accepted both. It checked that required keys had valid content (e.g. conversations is a non-empty alternating list of human/gpt turns) but never inspected the set of top-level keys, nor the set of keys inside each turn. The export and merge paths copied content_json 1:1 to JSONL with no filter.

Fix

New module backend/app/services/example_schema.py is the single source of truth for format whitelists:

Format	Top-level keys	Turn keys	Roles (alternating)
alpaca	instruction, input, output	n/a	n/a
sharegpt	conversations	from, value	human / gpt
chatml	messages	role, content	user / assistant

Two operations on this schema:

• validate_example(parsed, fmt) → ValidationResult — strict. Returns ok=False for any extra top-level key, extra turn key, role mismatch, missing required, wrong type, empty value, too few turns (<2), unknown format, or non-dict payload. Wired into the generation pipeline (_validate_example_structure is now a thin wrapper). Rejected examples follow the existing 2-attempt retry; on second failure they are skipped. The structured reason and detail are attached to the generation_invalid_structure activity event for UI surfacing.

• strip_to_schema(parsed, fmt) → (cleaned, dropped_paths) — defensive cleanup that removes extra keys without validating types. Used by serialize_for_jsonl, called from export_service.export_job and merge_service._jsonl_line. Both paths emit a single warning summary log when any rows were stripped; merge additionally fires a merge_strip_extra_keys activity event with rows_affected.

The DB is never mutated. Strip only sanitises the JSONL output, so source examples.content_json rows remain intact and any re-export is idempotent for legacy data. list_examples, the SSE stream, and dedup endpoints continue to return raw DB content (debug-friendly).

Impact

• New examples cannot enter the DB with extra top-level or per-turn keys.
• Re-exporting any legacy job produces a clean JSONL even when the DB content has stray fields, with no schema migration needed.
• Merged datasets are sanitised on the fly — no need to manually filter source jobs before POST /api/datasets/merge.
• The generation_invalid_structure activity event now carries reason and detail, so the UI can show why a generation was rejected (e.g. "extra top-level keys: ['gpt']").

Why this didn't surface earlier

The old validator's sharegpt branch iterated over conversations and verified each turn's from / value fields, but it never inspected the set of top-level keys. A model returning {"conversations": [...], "gpt": [...]} looked valid because conversations was internally well-formed; the second top-level key was simply ignored.

The bug only surfaces at HuggingFace load time, not during generation, validation, export, or upload — so it sat dormant in datasets until someone tried to fine-tune.

Tests

402 tests pass; no regressions in the existing suite.

Files:

• backend/app/services/example_schema.py (new)
• backend/app/services/job_runner.py
• backend/app/services/export_service.py
• backend/app/services/merge_service.py
• backend/app/services/event_log.py

---

Hotfix — JobConfig.categories max_length blocked merges with >10 sub-categories

Problem

GET /api/jobs returned 500 Internal Server Error after merging 8 source jobs. The History page wouldn't load ("Failed to fetch"). The backend itself was healthy (/api/health returned 200) — only the jobs list endpoint crashed.

Root cause in backend/app/models/jobs.py:23:

categories: List[CategoryConfig] = Field(..., min_length=1, max_length=10)

The max_length=10 was intended as a sanity cap for user-created jobs. But the same Pydantic model is used in the read path to parse existing jobs from the database. Merging 8 source jobs × 2-3 sub-categories each yields 22 sub-categories, which triggered:

pydantic_core._pydantic_core.ValidationError: 1 validation error for JobConfig
categories
  List should have at most 10 items after validation, not 22 [type=too_long]

→ _row_to_list_item raised → the entire endpoint failed.

Fix

One line in backend/app/models/jobs.py:23:

categories: List[CategoryConfig] = Field(..., min_length=1, max_length=64)

64 leaves headroom for nested merges (e.g. 8 sources × 3 sub × 2 levels of nesting).

Impact

• History page loads all jobs again
• Merged jobs with any number of sub-categories (up to 64) parse correctly
• No data migration — nothing in the database changes, only input/parse validation

Why this didn't surface earlier

Earlier merge tests (including the 400k-example stress test referenced in internal notes) used jobs with ≤10 sub-categories each. The first 8-job merge with ~3 sub-categories each landed in production on 2026-04-27 and hit this previously dormant constraint.

File: backend/app/models/jobs.py

---

This discussion was created from the release v1.0.2.