Monitoring for errors #49

Description

It would be nice to be able to inspect the system quickly and easily, certainly without having to fire up a shell.

Definition of Done

• A basic script looking for errors or inconsistencies is available (only this is in scope for Waveform project).
• We know within a reasonable amount of time that something (an upload or processing step) failed, so we can manually re-run, and caution users not to use the incomplete data
• Perhaps showing a summary of data that has been received (can't really check for errors, as we don't know what data's expected)
• Separately monitor compressed HL7 files vs 2 kinds of parquets vs csv files.

Dependencies

Comments

This could roll into the next project on open telemetry.
It would be useful to have shared error monitoring in SAFEHR rather than done separately for individual projects.

Suggested implementation

Mount the waveform-export and waveform-saved-messages directories read-only into the emap streamlit container. Remember that waveform-export is not a part of Emap and neither directory is guaranteed to have any contents.

Make a new streamlit page that has controls on it for date, hour, variable, maybe bed.

For HL7 monitoring

A query method with caching (careful not to cache current data that is fast changing...) that takes the above parameters and does a query along the lines of:

find waveform-save-messages/${date}*/${bed}/ -name '*.bz' perl -pe 's/\r/\n/g' | grep -Po 'NM\|\d+\|' | sort | uniq -c

(you could use pure python instead)

The purpose is to see at a glance what types of data were coming through at various times. I think a common use case is wanting to know what is coming through right now.

For export pipeline monitoring

Something in a similar spirit to the above but in the waveform-export directory.

To check for errors, you could compare the number of files in original-csv vs pseudonymised vs ftps-logs, or you could scan the snakemake log file.