Skip to content

Fix silent data loss in the Prometheus pipeline: raise on missing tables, sync TRUTH.PROMETHEUS#911

Open
sevmag wants to merge 4 commits into
graphnet-team:mainfrom
sevmag:bug/prometheus-silent-data-loss
Open

Fix silent data loss in the Prometheus pipeline: raise on missing tables, sync TRUTH.PROMETHEUS#911
sevmag wants to merge 4 commits into
graphnet-team:mainfrom
sevmag:bug/prometheus-silent-data-loss

Conversation

@sevmag

@sevmag sevmag commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Closes #909.

Both failure modes in the Prometheus conversion/training path were silent — truth-less or photon-less datasets with no error. This PR makes the first one loud and fixes the second at its root.

Changes

PrometheusReader: raise on missing tables. The reader dispatched extractors by exact table-name match and silently skipped any extractor whose table wasn't in the file (e.g. a non-default photon_field_name in the Prometheus config, or a typo'd table_name). It now validates all configured tables once per file and raises a ValueError listing the missing and the available tables.

TRUTH.PROMETHEUS synced with current Prometheus output. The constant still described a truth schema (injection_*, primary_lepton_1_*, ...) that Prometheus stopped writing long ago; against a freshly converted file, Dataset._remove_missing_columns silently stripped every truth column. Now:

  • TRUTH.PROMETHEUS = the event-level truth written by every current Prometheus injection type (interaction + initial_state_*), matching Injection.to_dict().
  • The old list is preserved as TRUTH.PROMETHEUS_LEGACY for older files such as the bundled example database (prometheus-events.db).
  • PrometheusTruthExtractor derives its default columns from TRUTH.PROMETHEUS, so the constant and the extractor can no longer drift apart.
  • PrometheusTruthExtractor gained an optional columns override for truth fields that only some injection types write (e.g. LeptonInjector's bjorken_x, bjorken_y, column_depth — deliberately not in the default, since GENIE-based files don't have them).
  • All in-repo consumers point at the schema matching their data: the six examples/04_training scripts and the datamodule tests (which run on the bundled legacy database) use PROMETHEUS_LEGACY; PublicPrometheusDataset._event_truth now derives from TRUTH.PROMETHEUS instead of duplicating it.

Breaking changes

  • TRUTH.PROMETHEUS changes meaning: code using it against old databases must switch to TRUTH.PROMETHEUS_LEGACY. This is intentional — the default-looking name should describe what current Prometheus writes.
  • Prometheus files containing no photon table now raise in PrometheusReader instead of quietly converting to truth-only databases. Note this includes files from runs where zero events produced hits (Prometheus writes mc_truth-only files in that case) — previously these converted silently; failing loudly surfaces e.g. vertices injected outside the detector volume.

Testing

  • New tests/data/test_prometheus_reader.py: extraction of both tables from the shipped Prometheus test parquet, verification that every TRUTH.PROMETHEUS field exists in that (current-schema) file, the columns override, and the missing-table raise.
  • tests/data/test_datamodule.py + the new tests: 14 passed locally; pre-commit (black, flake8, docformatter, pydocstyle, mypy) green on all touched files.

🤖 Generated with Claude Code

sevmag and others added 4 commits July 14, 2026 10:22
An extractor whose table is absent from the input file (e.g. a
non-default Prometheus photon_field_name, or a typo'd table_name) was
silently skipped, producing truth-only databases with no diagnostic.
Validate all configured tables once per file and raise a ValueError
naming the missing and available tables.

Part of graphnet-team#909.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TRUTH.PROMETHEUS still described a truth schema (injection_*,
primary_lepton_1_*, ...) that Prometheus stopped writing long ago;
using it against a freshly converted file silently strips every truth
column. Rename that list to TRUTH.PROMETHEUS_LEGACY (it matches older
files such as the bundled example database, which the 04_training
examples run on) and set TRUTH.PROMETHEUS to the schema Prometheus
writes today. PrometheusTruthExtractor now takes its default columns
from TRUTH.PROMETHEUS so the constant and the extractor cannot drift
apart again.

Part of graphnet-team#909.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The default (TRUTH.PROMETHEUS) covers the event-level scalars every
Prometheus injection type writes; a columns override lets users add
injection-specific fields such as LeptonInjector's bjorken_x/bjorken_y/
column_depth without subclassing.

Part of graphnet-team#909.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The datamodule tests run on the bundled legacy example database, so
they need TRUTH.PROMETHEUS_LEGACY; PublicPrometheusDataset duplicated
the current schema as a literal list and now derives it from
TRUTH.PROMETHEUS instead.

Part of graphnet-team#909.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Silent data loss in the Prometheus pipeline: stale TRUTH.PROMETHEUS constant and exact-name table dispatch in PrometheusReader

1 participant