DAQRunConsumer that buffers events from latest run

[erik-wallin]

This was discussed on Slack, but moving the discussion here.

Let's separate two different use-cases:

1. Sometimes you only want to write an event to disk if it fulfills some specific condition, e.g. having a pulse above a certain readout threshold. For this, you could pass a lambda to the RunConsumer that evaluates each event before writing it to disk.
2. If you are implementing a task with some sort of smart scan, e.g. when scanning over CALIB to see where the TOT threshold lies you'd like decrease or increase CALIB dynamically based on the result. Or more generally, if you want to implement something beyond basic grid scans. For this, the user needs to get a handle on the raw event packets themselves.

I propose that we modify DecodeAndWrite (or make another DAQRunConsumer) that

1. Buffers every event taken since the last startRun().
2. Has a function std::vectorpacking::SingleROCEventPacket& readBuffer(), that returns the whole buffer to the user.
3. Optionally, only writes an EventPacket to disk if it fulfills some user specified condition.
4. Clears the buffer the next time startRun() is called.

[tomeichlersmith]

The decode-and-write-run-summary branch has my mocked up solution. Instead of adding a readBuffer() function, it just takes a Lambda from the user to call at end_run. I prefer this because then we know that the function is being called at the appropriate time - at the end of the run and before the start of another - but it also is confusing to have Lambdas floating around.

Sidenote: this branch also only stores the samples from the 72 DAQ channels. I made this choice to try to save some memory, but it probably isn't worth that much if you want to change the table to be just a vector of SingleROCEventPacket objects instead of the vector of a 72-length array of Samples.

[erik-wallin]

Right, for many tasks it would be good to store more than one event. It would be best if it could store every event in the run, up to some MAX_BUFFER_SIZE.

But, only doing this with Lambdas seems a bit inconvenient?

Let's say you have a whole complicated analysis procedure of the data you just took, you're going to poke a whole bunch of registers based on the results, and make a ton of print-outs to the terminal. Doesn't it seem easier just to call readBuffer() in the task, without having to make a Lambda out of all of it? Especially since the Lambda you pass must have the correct return and argument types, it becomes a bit inflexible.

Since startRun() and endRun() are always called as a part of daq_run(), is don't see a possibility for readBuffer() to be read at the wrong time?

[tomeichlersmith]

You make good points, I drop my Lambda request 👍

[andpet1324]

I used another method using the existing implementation. In the writer, I referenced a vector called buffer to which I saved the event packet:

`std::vectorpflib::packing::SingleROCEventPacket buffer;
pflib::DecodeAndWriteToCSV writer{
...
[&](std::ofstream& f, const pflib::packing::SingleROCEventPacket& ep) {
...
buffer.push_back(ep);
}
};

After doing a daq_run, I then read out and saved the buffer data and cleared the buffer. This seems to work well from what I've seen, but I think it would be nice with a better implementation.

One problem this method has, is that when we have read out the data and calculated some value val, I want to save val with the writer. But if I reference val in the writer, I am saving the value for the current event in the next event, since the value gets calculated after running daq_run.

[andpet1324]

I put together a new class DecodeAndWriteToBuffer that inherits from DecodeAndWrite. The reason I had to create it was because I wanted to do a daq_run without creating a csv (see set_toa function https://github.com/LDMX-Software/pflib/blob/369190240b584142ca396b6cf25bfbdc5be49679/app/tool/tasks.cxx), and just read out ep data.

The class really simple. All it does is store a buffer vector with the event packets. When the function read_data is called, it returns the buffer and clears it. I also added a statement in end_run that clears the buffer if it gets too big.

It would be great with some feedback for improvements. I don't fully understand how the class gets handled. Will push once I'm sure how to handle set_toa.

Links to DecodeAndWriteToBuffer:
https://github.com/LDMX-Software/pflib/blob/369190240b584142ca396b6cf25bfbdc5be49679/src/pflib/DecodeAndWrite.cxx
https://github.com/LDMX-Software/pflib/blob/369190240b584142ca396b6cf25bfbdc5be49679/include/pflib/DecodeAndWrite.h

[tomeichlersmith]

This DecodeAndWriteToBuffer class looks good and I would like to comment on it. Could you open a PR with just that class addition?

Also, have you connected with @taylorjcolaizzi #167 - they are also working on TOA "alignment" (setting TOA thresholds) and so it would be good if y'all worked together on it.