Memory optimization

[Gijom]

Hello, I am facing some memory problems when reading big databases (which I would like to store completly in the memory). I noticed that the wfdb functions generally return float64 numpy arrays. However, if I understood correctly from the link below, the signals are coded on maximum 32 bits.
https://www.physionet.org/physiotools/wag/signal-5.htm

Here are my questions / remarks:

• does this mean it is safe to convert all signals to 32bits ?
• if this is safe, would it be possible to make your functions return float32 arrays ? Or even better to return the minimum size format depending on the header format field ?

G.

[cx1111]

I think the second option sounds reasonable. Can anyone think of any instances where returning numpy arrays with dtype=floatXX !=64 will do any harm with the average user's scripts?

[cx1111]

I'm nearly done with this, though I have some uncertainties regarding default logic and usability.

When digital values are returned, it is easy to use the numpy dtype corresponding to the same resolution or higher, ie. int16 for fmt 16 signals. But when physical values are requested, it may be a bit more ambiguous.

Consider a fmt 16 or fmt 32 signal. Converting those to float16 and float32 will make them lose some resolution if indeed the input signal was really captured with 16 or 32 bits.

In the end, I wonder what values the returnres argument I am adding should be allowed to be. I thought of:

• 64, 32, 16 to be very explicit. But this is annoying because even though there is only float16+, there is int8.
• 64, 32, 'minres' for a more automatic case where minres makes the package decide the optimal dtype. Once again, when it comes to floats, what would be the correct automatic formats? int32 --> float64 or float32? etc.

The default will be 64 in any case. Regarding my previous comment, people could easily create overflows/underflows if they unintentionally use a low resolution dtype.

[cx1111]

I've decided to allow it to be set to either 64, 32, 16 or 8. The user can mind the dtype since different cases may warrant different options.

Regarding your questions, for all formats aside from 32, it will be safe to convert to float32 with insignificant precision loss. And even with fmt 32 signals, it is unlikely that we have any (at this point in time on physionet) in which the capturing device actually recorded with 32 bit precision, so float32 would be generally safe in those cases too.

One last note, we have some ridiculously long signals in certain databases (many many days long). When processing these kinds of signals, reading them part by part will be inevitable since this package reads all values into memory.

I'll close this issue once I push the changes.

[Gijom]

Great, many thanks for your prompt changes!

[cx1111]

Sure, no problem. Let me know if my changes actually did anything :)

[cx1111]

I should go and edit the code to use np.add/subtract/divide/multiply during adc/dac conversion when only one version of the signal (digital/physical) is desired, to prevent extra memory use via temporary variable allocation.