`unsigned_to_signed` does not update recording offset_to_uV

[ablot]

If I read a raw binary file that is saved in uint16 and correctly set the offset, calling unsigned_to_signed returns a recording with an invalid offset.

raw_rec = se.read_binary(
        data_file,
        sampling_frequency=30000.,
        dtype=np.uint16,
        num_channels=384,
        gain_to_uV=3.05176,
        offset_to_uV=-2048*3.05176,
    )
# we can use get_traces with return_in_uV, all is good

unsigned = si.unsigned_to_signed(raw_rec, bit_depth=12)
# now the trace in uV have an offset, exactly 2048*3.05176

Shouldn't the unsigned_to_signed subtract 2**bit_depth / 2 * gain from the offset? Or is it expected that the offset is unset when using this function?

[ablot]

Maybe related, for some reason using set_probe on an unsigned recording changes the value returned by get_traces, this does not happen if I call set_probe first and then convert to signed. I don't see anything in the probe mapping that would change the gain or offset.

recs = dict()
recs['raw'] = se.read_binary(
        ephys_path / f"np2-{prb_suffix}-ephys_{suffix}.raw",
        sampling_frequency=fs_hz,
        dtype=np.uint16,
        num_channels=num_channels,
    )
recs['unsigned_from_raw'] = si.unsigned_to_signed(recs['raw'], bit_depth=bit_depth)
print(recs['raw'].dtype)
recs['probe_from_unsigned'] = recs['unsigned_from_raw'].set_probe(npx_mapping, in_place=False)
# The other way round
recs['probe_from_raw']= recs['raw'].set_probe(npx_mapping, in_place=False)
recs['unsigned_from_probe']=  si.unsigned_to_signed(recs['probe_from_raw'], bit_depth=bit_depth)


for k, rec in recs.items():
    trace =  rec.get_traces(start_frame=0, end_frame=2, return_in_uV=False, channel_ids=[30])
    print(f"{k}: {trace[0,0]} (dtype: {rec.dtype})")

Prints:

raw: 3847 (dtype: uint16). [expected]
unsigned_from_raw: 1799 (dtype: int16) [expected]
probe_from_unsigned: -28921 (dtype: int16) [BAD expected is 1799]
probe_from_raw: 3847 (dtype: uint16) [expected]
unsigned_from_probe: 1799 (dtype: int16) [expected]

[ablot]

These are probably unrelated issues. The second is that calling self.clone in a UnsignedToSignedRecording does not keep the bit_depth property to the recording segments.

[h-mayorquin]

Hi, thanks for opening this issue!

I actually think that you are right here that unsigned_to_signed should set the offset, but I would like to know if we are coming at it from the same angle. To me, the idea is that the physical meaning of the signal is given by the traces_in_uV and that should be the invariant in this operation - the gains and offsets should be modified so that remains true after a transformation in ADC space. For example, with your data: before unsigned_to_signed the physical voltage is 3847 * 3.05176 + (-2048*3.05176) = 5490 µV, but after conversion it becomes 1799 * 3.05176 + (-2048*3.05176) = -760 µV (wrong!). The fix would be to update the offset to 0, so that 1799 * 3.05176 + 0 = 5490 µV (preserving the physical voltage).

That said, I am not sure we can keep this principle across all of our recordings (or if it is desirable or if there is a better principle). Some preprocessors do handle calibration adjustments (correct_lsb() adjusts gain, zscore() resets to 1.0/0.0), but most just copy metadata as-is, and others like filter() zero the offset for unclear reasons.

How are you thinking about it? Are you coming at it from another angle?

Also, it would be great if you could open another issue for the probe cloning thing mentioned in your second comment. That seems like a separate bug related to clone() not preserving the bit_depth parameter.

Guys, what do you think? @alejoe91 @samuelgarcia @chrishalcrow @zm711

[alejoe91]

Hi guys,

I think that the unsigned_to_signed is a special way of setting the offset. Let me try to clarify:

Case 1: an acquisition system produces int16, but with an offset due to some circuit conditioning. In this case, the offset_to_uV should be used to correct for the offset.

Case 2: an acquisition system produces uint16, with a certain bit depth. While this is true that this results in an offset, the unsigned_to_signed takes care of it, since this is due to the actual data type and ADC range. So in this case, my suggestion would be to use offsets_to_uV=0 in the read_binary and let the unsigned_to_signed do its job. When getting traces, the uint16 will be converted to int16, offset taken care of, and then the gain will be applied.

@h-mayorquin maybe we should clarify these points in the docs?

[alejoe91]

These are probably unrelated issues. The second is that calling self.clone in a UnsignedToSignedRecording does not keep the bit_depth property to the recording segments.

Let's open a separate issue for this!

[alejoe91]

@ablot was this clarified with my comment here? #4170 (comment)

[ablot]

Yes you can close, except if you think that we should add more doc or emit a warning when unsigned_to_signed is called on a recording with an offset_to_uV non null. Thank you!