Skip to content

Export buffered dashboard data to NeXus files - #1355

Draft
SimonHeybrock wants to merge 3 commits into
mainfrom
1351-data-export
Draft

SimonHeybrock wants to merge 3 commits into
mainfrom
1351-data-export

Conversation

@SimonHeybrock

@SimonHeybrock SimonHeybrock commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Closes #1351.

Scientists need data that is never written as a run, e.g. alignment scans, to reference later. This adds an Export data button to the dashboard header. It opens a wizard modeled on the plot wizard:

  1. Select workflow and output (the plot wizard's first step).
  2. Select the export type. Time series is the buffered history of a 0-D time-series output, optionally correlated with up to two other time series. It is offered only for 0-D outputs. Current value uses the plots' window modes: since run start, latest update, or a sum/mean over a window.
  3. Select sources (the generic configuration form).
  4. Preview what is buffered (groups, shapes, time ranges) and download a NeXus file.

Only what DataService holds can be exported. Previously a key kept history only while a plot subscribed to it with a history extractor, so a scan done without the right plot open left nothing to export. A permanent TimeseriesRetention subscriber now keeps history for every 0-D time-series output, within the existing 20 MB per-key TemporalBuffer cap. A window aggregation is served only if the buffered updates reach back over the whole window, and is otherwise refused with what is buffered. 0-D time series are always covered. Other outputs are covered only while a plot in window mode buffers them, and then only for that plot's window.

Correlation. The file holds the unbinned per-update table, not the histogram: each data point carries the axis value in effect at its time as a coord, and each axis is also exported as its own time series. The lookup is shared with CorrelationHistogramPlotter.

NeXus. scippnexus cannot write DataArrays as NXdata, so nexus_writer.py writes one NXdata group per source with h5py. Tests check that scippnexus loads every group back to the exported DataArray; files also open in nexusformat (NeXpy). Source names with / (common for f144 PVs) are written with _, and the original names are kept as group attributes. The format is a field in the export params, so more formats can be added.

Transfer. pn.widgets.FileDownload sends the file as base64 over the websocket, which runs on the IOLoop all sessions share. Exports are therefore capped at 50 MB.

Placement of the modal. The modal's holder is passed into PlotGridTabs as an overlay next to its own modal container. In the header, panes the wizard builds after load stayed invisible (#1154). Wrapping the main content to hold it broke plot geometry (caught by render_geometry_test).

Open question: retention memory

Keeping history for all time series is not free. Instruments have 87 (DREAM) to 167 (ODIN) time-series keys. At 1 Hz a key fills its 20 MB cap in about a week, so a long-running dashboard could grow to 1.7–3.3 GB. Only keys with running workflows get data, so the real figure is lower. A smaller cap for retention-only keys, or a global budget, belongs with #1274. Should that land before this, or is it a follow-up?

Test plan

  • New unit tests cover the writer round-trip, export collection (retention, UTC times, correlation, name collisions, / names) and the wizard steps.
  • A Playwright run against the fake backend downloaded a correlated monitor/motion export and a detector image. Both load with scippnexus.
  • Export a correlation of a detector or monitor total against a motor on a real instrument, and load the file in the analysis tool of choice.
  • Check the header button and the wizard with basic auth enabled (next to the logout link).

🤖 Generated with Claude Code

SimonHeybrock and others added 2 commits October 7, 2026 12:41
Scientists need data that is never written as a run, e.g. alignment scans,
to reference later. Add an "Export data" button to the dashboard header that
opens a wizard modeled on the plot wizard: choose workflow and output,
choose between the current value and the buffered history of a time series
(optionally correlated with up to two other time series), choose sources and
window params, then preview and download a NeXus file.

An export can only contain what DataService holds. Previously a key kept
history only while a plot subscribed to it with a history extractor, so a
scan done without the right plot open had nothing to export. A permanent
subscriber now keeps history for every 0-D time-series output, within the
existing 20 MB per-key buffer cap.

scippnexus cannot write DataArrays as NXdata, so a small h5py writer
produces one NXdata group per source; tests check that scippnexus loads
the groups back to the exported data.

The correlation lookup is extracted from CorrelationHistogramPlotter so that
the plot and the export share it. Data-source resolution in plot_orchestrator
becomes public for the same reason.

Refs #1351

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From review of the export feature:

- f144 sources are often named like PV paths (slit_set_2/blade_bottom). HDF5
  read the slash as a path separator, so such an axis broke the NXdata
  layout. Export names now replace it; the writer rejects names with one.
- History exports kept only the first start_time, through the display
  extractor. They now read the buffer as stored, so every update keeps its
  own window, as needed to compute rates.
- Current-value exports no longer aggregate over a time window. That needs
  history which outputs other than 0-D time series do not have unless a plot
  asks for it (the export then crashed), and then only for the plot's
  window. The choice left is latest update vs. since run start, hidden when
  the output offers only one.
- An axis equal to the data or to the other axis is rejected instead of
  silently overwriting the correlated data.
- The size cap is checked before writing as well; write errors are shown
  in the wizard.
- correlate moves to its own module, so data export does not depend on the
  correlation plotter. Filenames carry a UTC timestamp like the file
  content.

Refs #1351

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@SimonHeybrock SimonHeybrock added the area:dashboard Panel/Bokeh/HoloViews UI, plotting, frontend config label Oct 7, 2026
Current-value exports again offer the plots' window controls: since start,
latest update, or a sum/mean over a window. The window is aggregated with
the plots' own extractor, from the buffer as stored, after checking that
the buffered updates reach back over the whole window. A window the buffer
does not cover is refused with what is buffered, instead of crashing (no
history buffered) or silently aggregating less (a plot buffering a shorter
window). 0-D time series are always covered; other outputs only while a
plot in window mode buffers them.

Refs #1351

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dashboard Panel/Bokeh/HoloViews UI, plotting, frontend config

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Save data to (NeXus) files

1 participant