Repository navigation
Export buffered dashboard data to NeXus files - #1355
Draft
SimonHeybrock wants to merge 3 commits into
Draft
SimonHeybrock wants to merge 3 commits into
SimonHeybrock wants to merge 3 commits into
Conversation
Scientists need data that is never written as a run, e.g. alignment scans, to reference later. Add an "Export data" button to the dashboard header that opens a wizard modeled on the plot wizard: choose workflow and output, choose between the current value and the buffered history of a time series (optionally correlated with up to two other time series), choose sources and window params, then preview and download a NeXus file. An export can only contain what DataService holds. Previously a key kept history only while a plot subscribed to it with a history extractor, so a scan done without the right plot open had nothing to export. A permanent subscriber now keeps history for every 0-D time-series output, within the existing 20 MB per-key buffer cap. scippnexus cannot write DataArrays as NXdata, so a small h5py writer produces one NXdata group per source; tests check that scippnexus loads the groups back to the exported data. The correlation lookup is extracted from CorrelationHistogramPlotter so that the plot and the export share it. Data-source resolution in plot_orchestrator becomes public for the same reason. Refs #1351 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From review of the export feature: - f144 sources are often named like PV paths (slit_set_2/blade_bottom). HDF5 read the slash as a path separator, so such an axis broke the NXdata layout. Export names now replace it; the writer rejects names with one. - History exports kept only the first start_time, through the display extractor. They now read the buffer as stored, so every update keeps its own window, as needed to compute rates. - Current-value exports no longer aggregate over a time window. That needs history which outputs other than 0-D time series do not have unless a plot asks for it (the export then crashed), and then only for the plot's window. The choice left is latest update vs. since run start, hidden when the output offers only one. - An axis equal to the data or to the other axis is rejected instead of silently overwriting the correlated data. - The size cap is checked before writing as well; write errors are shown in the wizard. - correlate moves to its own module, so data export does not depend on the correlation plotter. Filenames carry a UTC timestamp like the file content. Refs #1351 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Current-value exports again offer the plots' window controls: since start, latest update, or a sum/mean over a window. The window is aggregated with the plots' own extractor, from the buffer as stored, after checking that the buffered updates reach back over the whole window. A window the buffer does not cover is refused with what is buffered, instead of crashing (no history buffered) or silently aggregating less (a plot buffering a shorter window). 0-D time series are always covered; other outputs only while a plot in window mode buffers them. Refs #1351 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1351.
Scientists need data that is never written as a run, e.g. alignment scans, to reference later. This adds an Export data button to the dashboard header. It opens a wizard modeled on the plot wizard:
Only what DataService holds can be exported. Previously a key kept history only while a plot subscribed to it with a history extractor, so a scan done without the right plot open left nothing to export. A permanent
TimeseriesRetentionsubscriber now keeps history for every 0-D time-series output, within the existing 20 MB per-keyTemporalBuffercap. A window aggregation is served only if the buffered updates reach back over the whole window, and is otherwise refused with what is buffered. 0-D time series are always covered. Other outputs are covered only while a plot in window mode buffers them, and then only for that plot's window.Correlation. The file holds the unbinned per-update table, not the histogram: each data point carries the axis value in effect at its time as a coord, and each axis is also exported as its own time series. The lookup is shared with
CorrelationHistogramPlotter.NeXus. scippnexus cannot write DataArrays as
NXdata, sonexus_writer.pywrites oneNXdatagroup per source with h5py. Tests check that scippnexus loads every group back to the exported DataArray; files also open innexusformat(NeXpy). Source names with/(common for f144 PVs) are written with_, and the original names are kept as group attributes. The format is a field in the export params, so more formats can be added.Transfer.
pn.widgets.FileDownloadsends the file as base64 over the websocket, which runs on the IOLoop all sessions share. Exports are therefore capped at 50 MB.Placement of the modal. The modal's holder is passed into
PlotGridTabsas an overlay next to its own modal container. In the header, panes the wizard builds after load stayed invisible (#1154). Wrapping the main content to hold it broke plot geometry (caught byrender_geometry_test).Open question: retention memory
Keeping history for all time series is not free. Instruments have 87 (DREAM) to 167 (ODIN) time-series keys. At 1 Hz a key fills its 20 MB cap in about a week, so a long-running dashboard could grow to 1.7–3.3 GB. Only keys with running workflows get data, so the real figure is lower. A smaller cap for retention-only keys, or a global budget, belongs with #1274. Should that land before this, or is it a follow-up?
Test plan
/names) and the wizard steps.🤖 Generated with Claude Code