Shape mismatch error while applying curation

[majidkhalili]

Hello,

While applying curation using results = sif.apply_curation(sorting_Analayzer, curation_dict, merging_mode='hard'), I get the following error. After execution, both compute_waveforms and Compute : spike_amplitudes + spike_locations progress bars reach 100%, and only then I get the error.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[25], line 8
      6 with open(savingPath + '\\Sorters\\Sorter_Analyzer_Kilosort4_PFC_VGA\\spikeinterface_gui\\curation_data.json', 'r') as file:
      7         curation_dict_PFC = json.load(file)
----> 8 sorting_Analayzer_RR_Analyzer_MergedNRemoved_PFC = sif.apply_curation(sorting_Analayzer_RR_Analyzer_PFC, curation_dict_PFC, merging_mode='hard')

File c:\Users\Majid\anaconda3\envs\spikeinterface_env\Lib\site-packages\spikeinterface\curation\curation_format.py:354, in apply_curation(sorting_or_analyzer, curation_dict, censor_ms, new_id_strategy, merging_mode, sparsity_overlap, verbose, **job_kwargs)
    342     analyzer = analyzer.remove_units(curation_dict["removed_units"])
    343     analyzer, new_unit_ids = analyzer.merge_units(
    344         curation_dict["merge_unit_groups"],
    345         censor_ms=censor_ms,
   (...)
    352         **job_kwargs,
    353     )
--> 354     apply_curation_labels(analyzer.sorting, new_unit_ids, curation_dict)
    355     return analyzer
    356 else:

File c:\Users\Majid\anaconda3\envs\spikeinterface_env\Lib\site-packages\spikeinterface\curation\curation_format.py:256, in apply_curation_labels(sorting, new_unit_ids, curation_dict)
    253             group_values.append(value)
    254     if len(set(group_values)) == 1:
    255         # all group has the same label or empty
--> 256         sorting.set_property(key, values=group_values, ids=[new_unit_id])
    257 else:
    259     for key in label_def["label_options"]:

File c:\Users\Majid\anaconda3\envs\spikeinterface_env\Lib\site-packages\spikeinterface\core\base.py:296, in BaseExtractor.set_property(self, key, values, ids, missing_value)
    291         assert (
    292             dtype_kind == existing_property.kind
    293         ), f"Mismatch between existing property dtype {existing_property.kind} and provided values dtype {dtype_kind}."
    295     indices = self.ids_to_indices(ids)
--> 296     self._properties[key][indices] = values
    297 else:
    298     indices = self.ids_to_indices(ids)

ValueError: shape mismatch: value array of shape (2,) could not be broadcast to indexing result of shape (1,)

If it is relevant, I get the following warnings before the error pops up and while computing the waveforms and apike amplitudes:

c:\Users\Majid\anaconda3\envs\spikeinterface_env\Lib\site-packages\spikeinterface\qualitymetrics\misc_metrics.py:908: UserWarning: Some units have too few spikes : amplitude_cutoff is set to NaN
  warnings.warn(f"Some units have too few spikes : amplitude_cutoff is set to NaN")
c:\Users\Majid\anaconda3\envs\spikeinterface_env\Lib\site-packages\numpy\core\_methods.py:206: RuntimeWarning: Degrees of freedom <= 0 for slice
  ret = _var(a, axis=axis, dtype=dtype, out=out, ddof=ddof,
c:\Users\Majid\anaconda3\envs\spikeinterface_env\Lib\site-packages\numpy\core\_methods.py:163: RuntimeWarning: invalid value encountered in divide
  arrmean = um.true_divide(arrmean, div, out=arrmean,
c:\Users\Majid\anaconda3\envs\spikeinterface_env\Lib\site-packages\numpy\core\_methods.py:198: RuntimeWarning: invalid value encountered in divide
  ret = ret.dtype.type(ret / rcount)

I am using spikeinterface==0.101.2.

Thanks in advance for your assistant!

[majidkhalili]

Update: When I removed the manual labels field from the json file, the problem got resolved and curation was applied.

[samuelgarcia]

Hi @majidkhalili
thank you for this I will have a look.

[samuelgarcia]

Hi could you try this branch #3601 ?

[majidkhalili]

Sure, I need some time to test, and I will get back to you on this.

[MariaShuja]

Hi @majidkhalili

Were the RuntimeWarnings also resolved? I get the same warnings when computing the params using

analyzer_KS4.compute(extensions_to_compute, extension_params=extension_params)

Not sure if I should be alarmed by these warnings.

[majidkhalili]

Hey Maria,

Which warnings are you getting?

[MariaShuja]

Hi Majid,

I am getting this:

/scicore/home/rinaldit/shujah0000/miniforge3/envs/SpikeInterface/lib/python3.12/site-packages/spikeinterface/sortingcomponents/peak_localization.py:226: RuntimeWarning: invalid value encountered in divide coms = np.dot(wf_data, local_contact_locations) / (np.sum(wf_data, axis=1)[:, np.newaxis]) /scicore/home/rinaldit/shujah0000/miniforge3/envs/SpikeInterface/lib/python3.12/site-packages/spikeinterface/sortingcomponents/peak_localization.py:226: RuntimeWarning: invalid value encountered in divide coms = np.dot(wf_data, local_contact_locations) / (np.sum(wf_data, axis=1)[:, np.newaxis]) /scicore/home/rinaldit/shujah0000/miniforge3/envs/SpikeInterface/lib/python3.12/site-packages/spikeinterface/sortingcomponents/peak_localization.py:226: RuntimeWarning: invalid value encountered in divide coms = np.dot(wf_data, local_contact_locations) / (np.sum(wf_data, axis=1)[:, np.newaxis]) /scicore/home/rinaldit/shujah0000/miniforge3/envs/SpikeInterface/lib/python3.12/site-packages/spikeinterface/sortingcomponents/peak_localization.py:226: RuntimeWarning: invalid value encountered in divide coms = np.dot(wf_data, local_contact_locations) / (np.sum(wf_data, axis=1)[:, np.newaxis])

[majidkhalili]

hmmmm,
I have never seen this warning before. Maybe some one from Spikeinterface team can help.

I would look at the peak locations to see how many invalid locations do I get, if they are just few of them, I wouldnt worry about it much.

[samuelgarcia]

THis warnings are due to zeros in peak localization not sure they are related to the curation machanism.

[zm711]

@majidkhalili were you able to test if our curation fixes helped you?

[majidkhalili]

@zm711 sorry for the very late response. I didnt have access to the data (due to large data size, I dont keep it on my computer). As mentioned above, I already solved the issue by removing the manual labels. But I can still test the new branch. Is there a way that I can install the above mentioned branch via pip so that I can created a dedicated test environment for this?