SQW compression methodology to speed up I/O

[cmarooney-stfc]

Currently SQw objects write and store the computed pixel information i.e. qx, qy, qz, E which are $\mathcal{O}\left(n_{\text{run}} n_{\text{Efix}} n_{\text{det}} n_{\text{Ebin}} \right) \times 4$ data.

Instead, it is possible to store only the $E_{fix}$, detectors, $E_{bin}$ arrays and the mapping of the pixels to the requisite indices in these arrays (already stored as det_id, en_id, run_id) to compute qx, qy, qz, E on-the-fly (these expansions are available in e.g. calculate_qw_pixels2). This instead results in $\mathcal{O}\left(n_{\text{run}} (n_{\text{Efix}} + n_{\text{det}} + n_{\text{Ebin}} ) \right)$ (given unique detector compression, this may in fact be smaller than this; signal and error and *_id will still need to be stored as $\mathcal{O}\left(n_{\text{run}} n_{\text{Efix}} n_{\text{det}} n_{\text{Ebin}} \right)$ )

Not only this, but it should be possible to eliminate zero-counts (possibly following a low-pass filter stage) from the storage by dropping these detector results from the mapping. This could be done by:

• splitting the detectors into "signalling"-"non-signalling" and dumping as separate blocks
• by deliberately ordering these to the bottom of the data arrays

It would then be possible to drop the signal & error components for these pixels and assume any "empty" detector signal is 0. This will inherently further reduce the volume of stored data.

Initial estimates for standard data suggest that these could result in savings ~5x with respect to current storage.