Memory error while using widgets

[FlyingFordAnglia]

Hello!
I am encountering a MemoryError while running the following bit of code. I faced the issue only after upgrading the spikeinterface module (previous version was 0.101.2; updated to 0.102.3). The same code worked in the previous version. Once I uninstalled the latest version and installed an older version, the code ran without errors again.

recording_layers = dict(
    ps=recording_split_by_shank[0],
    spatial_filt_ps=shank_wise_recording_spatial_filt[0]
)

w = sw.plot_traces(recording_layers, mode='map', time_range=[19*60, 19*60+5], backend="ipywidgets")

The MemoryError I get points to the following line within the spikeinterface module (line 166 in spikeinterface/widgets/traces.py):

list_traces = [traces * scale for traces in list_traces]

It looks like the widget code tries to load the entire recording object into memory.

[chrishalcrow]

Hi @FlyingFordAnglia thanks for posting the issue. Nothing obvious comes to mind. Could you post some more of your code leading up to the plot_traces function call, just so we can see what preprocessing steps have been applied? And could you post some more of the error trace you get? Thanks!

[FlyingFordAnglia]

Hi, here is the code leading up to the widget:

import spikeinterface.extractors as se
import spikeinterface.core as si
import spikeinterface.widgets as sw
import spikeinterface.preprocessing as spre

base_dir = Path(r'<base dire name>') 
data_dir = os.path.join(base_dir, '<session dir name>')

recording = se.read_spikeglx(data_dir, stream_id=stream_id, load_sync_channel=False)

recording = recording.astype('float32')
recording_bandpass = spre.bandpass_filter(recording, freq_min=300, freq_max=9000)
recording_phase_shift = spre.phase_shift(recording=recording_bandpass)

recording_layers = dict(
    raw=recording,
    filt=recording_bandpass,
    phase_shift=recording_phase_shift
)

w = sw.plot_traces(recording_layers, mode='map', time_range=[19*60, 19*60+2], backend="ipywidgets")

This code works with version 0.101.2. When I upgrade to 0.102.3, I get the following error. No other packages were changed. I double checked that downgrading to previous version makes the code work again.

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
Cell In[17], line 8
      1 # %matplotlib widget
      2 recording_layers = dict(
      3     raw=recording,
      4     filt=recording_bandpass,
      5     phase_shift=recording_phase_shift
      6 )
----> 8 w = sw.plot_traces(recording_layers, mode='map', time_range=[19*60, 19*60+2], backend="ipywidgets")

File ~\miniconda3\envs\spike_interface\lib\site-packages\spikeinterface\widgets\traces.py:166, in TracesWidget.__init__(self, recording, segment_index, channel_ids, order_channel_by_depth, time_range, mode, return_scaled, cmap, show_channel_ids, events, events_color, events_alpha, color_groups, color, clim, tile_size, seconds_per_row, scale, vspacing_factor, with_colorbar, add_legend, backend, **backend_kwargs)
    160 cmap = cmap
    162 times_in_range, list_traces, frame_range, channel_ids = _get_trace_list(
    163     recordings, channel_ids, time_range, segment_index, return_scaled=return_scaled, times=times
    164 )
--> 166 list_traces = [traces * scale for traces in list_traces]
    168 # stat for auto scaling done on the first layer
    169 traces0 = list_traces[0]

File ~\miniconda3\envs\spike_interface\lib\site-packages\spikeinterface\widgets\traces.py:166, in <listcomp>(.0)
    160 cmap = cmap
    162 times_in_range, list_traces, frame_range, channel_ids = _get_trace_list(
    163     recordings, channel_ids, time_range, segment_index, return_scaled=return_scaled, times=times
    164 )
--> 166 list_traces = [traces * scale for traces in list_traces]
    168 # stat for auto scaling done on the first layer
    169 traces0 = list_traces[0]

MemoryError: Unable to allocate 102. GiB for an array with shape (71183037, 384) and data type float32

[JoeZiminski]

I wonder if it is related to line 681 here, the computation of start / end times will also now take into account the start time on the recording. Could you do recording.get_start_time(), I wonder if it is nonzero?

[FlyingFordAnglia]

I wonder if it is related to line 681 here, the computation of start / end times will also now take into account the start time on the recording. Could you do recording.get_start_time(), I wonder if it is nonzero?

When I do recording.get_start_time(), it is not zero. I get 5296.751773920088.

[JoeZiminski]

I think that may be the issue, I guess this start time is loaded from the meta data. It's not being subtracted from the times, and possibly making them meaningless and defaulting to the entire dataset, which is too large. To solve the issue, try adjusting your passed times relative to the start time of 5296.751773920088 rather than zero. Or, if this start time is not useful to you, you can run recording.reset_times(), let me know if that helps.

[FlyingFordAnglia]

I think that may be the issue, I guess this start time is loaded from the meta data. It's not being subtracted from the times, and possibly making them meaningless and defaulting to the entire dataset, which is too large. To solve the issue, try adjusting your passed times relative to the start time of 5296.751773920088 rather than zero. Or, if this start time is not useful to you, you can run recording.reset_times(), let me know if that helps.

Great! Running recording.reset_times() before all the preprocessing steps solves the issue. The widget runs as expected.
I think it would be good to mention this in the docs though, for anyone else who tries to use a widget (or any other function that requires start and end times) with the new version.

[alejoe91]

@JoeZiminski @h-mayorquin I think that in this case passin time_range=[19*60, 19*60+5] should raise an error! I'll add some stricter checks like @h-mayorquin did here #3942