Archiving #214
Description
- As a systems/data administrator I would like scans data to be saved in a way that is appropriate for long-term, scientifically useful data
- As a scientist I would like scans data to be stored in a way that aligns with FAIR principles
This is a partial continuation of #24 , but focused on the archiving aspects.
For avoidance of any doubt, we cannot tie these bluesky scan output files to the concept of a DAE run.
On discussion with @ChrisM-S, an initial solution could be:
- Bluesky writes its files into
c:\data\<rb number>. This avoids depending on network at the time when bluesky is writing the file. - Bluesky marks files that it has finished with with a readonly flag
- Bluesky writes checksum files for all of the file(s) it has generated
- A cron job will copy/move the files from there into
\\isis\inst$\ndx<INST>\cycle_xxx\autoreduced- To do: do we want any structure under the
autoreduceddirectory
- To do: do we want any structure under the
- From there, the data will be replicated and archived using the existing mechanisms for autoreduced data
- File naming will stay as it is for the moment
Acceptance criteria
- Consult internally whether this is a workable solution
- Discuss this proposed solution with instrument scientists for feedback
- Write an ADR describing this decision, how/why we came to it, and what the tradeoffs are
- Write tickets to do technical work required