[ntuple] add Real32Quant column type

[silverweed]

This Pull request:

adds the Real32Quant column type to RNTuple. This column type stores floating point values on disk as integers with a user-defined precision (from 3 to 32 bits) and a user-defined value range. This allows to reduce the storage space required to save floats with a well-defined range with more precision than a simple truncation.

The conversion is defined as (pseudocode):

def quantize(value, min, max, n_precision_bits)
{
  quantized_max = (1 << n_precision_bits) - 1;
  scale = quantized_max / (max - min);
  quantized = round((value - min) * scale);
  return quantized;
}

This change requires adding metadata to the on-disk information, more specifically in the Field Description (see specifications.md for more details).

Checklist:

• tested changes locally
• updated the docs (if necessary)

[github-actions]

Test Results

    13 files      13 suites   3d 3h 43m 58s ⏱️
 2 697 tests  2 697 ✅ 0 💤 0 ❌
32 953 runs  32 953 ✅ 0 💤 0 ❌

Results for commit 64700ac.

♻️ This comment has been updated with latest results.

[jblomer]

Very nice! In principle looks good to me.

[jblomer]

Should we define it? I.e. test and throw?

[silverweed]

I wanted to keep the specification as vague as possible to give us room for implementing it in the most performant way possible. Maybe this is not necessary, but I think we should have a discussion about it before settling for anything other than UB - we can always specify more in the future, but not go back.

[jblomer]

I think that's fine for the specification. But in our current implementation, should we / will we throw?

[silverweed]

Added in 6526106

[pcanal]

This column type stores floating point values on disk as integers with a user-defined precision (from 3 to 32 bits) and a user-defined value range.

This sounds very similar to Double32 ... We will need to explain clearly the differences and advantages ....

[jblomer]

LGTM, thanks! I'll leave the question where to throw in the Unpack() call stack up to you.

[pcanal]

I recommend holding off on merging until we have documented/understood the differences (if any) between this implementation and the related Double32_t implementation (capability and resulting onfile precision)

[silverweed]

@pcanal The implementation is, logically speaking, exactly the same (see TBufferFile.cxx:Read/WriteDouble32.
The main difference is design-wise, for the fact that RNTuple's quantization is:

• not bound to the type of the variable (you can quantize any float or double, not just a Double32_t
• not statically chosen (you can set the value range and the bit width at runtime rather than deciding once and for all via the variable comments).

We can discuss in more details, but in my opinion those are the two main points of difference. The implementation itself is trivial and it's akin to Read/WriteFastArrayDouble32 (but slightly more performant in principle, as it doesn't have to check the min/max/scale factor for each element - they are all the same within a call to Pack/Unpack).

As a last divergence point, Double32_t silently clamps the values that fall our of range, while Real32Quant will throw in that situation.

[pcanal]

(but slightly more performant in principle, as it doesn't have to check the min/max/scale factor for each element - they are all the same within a call to Pack/Unpack).

Isn't it the already same in the in Read/WriteFastArrayDouble32 ? (Furthermore for reading the common case is to call TBufferFile::ReadWithFactor/ReadWithNbits)

The implementation is, logically speaking, exactly the same

Thank you for clarifying :)

As a last divergence point, Double32_t silently clamps the values that fall our of range, while Real32Quant will throw in that situation.

(If not already done) this should be called out in the doc.

[silverweed]

Isn't it the already same in the in Read/WriteFastArrayDouble32?

Yes, I had misread the function the first time, you are right. But anyway performance is not really our concern as regards our motivation for Real32Quant :)

(If not already done) this should be called out in the doc.

Our idea is, at least for now, to say that the behavior in case of out-of-range is undefined, because we don't want to preclude ourselves to change our mind or our implementation in the future. At the moment we throw an exception but we might decide something different in the future, e.g. for performance reasons.
You can see the exact wording in this very commit (in the specifications.md file), so if you have some specific suggestion feel free to comment on it.

Edit: I was wrong, we're currently not specifying the behavior in the specifications.md but only in the code. Should we also explicitly say the behavior is undefined in the specs? I'm not sure it makes sense because the specs in principle only refer to the binary format; should they also say how a writer should behave when receiving wrong input from the user? @jblomer thoughts about this?

[jblomer]

Edit: I was wrong, we're currently not specifying the behavior in the specifications.md but only in the code. Should we also explicitly say the behavior is undefined in the specs? I'm not sure it makes sense because the specs in principle only refer to the binary format; should they also say how a writer should behave when receiving wrong input from the user? @jblomer thoughts about this?

I think that the specification is not the ideal place for that sort of error behavior documentation. I'd suggest a brief section in the architecture.md that summarizes the low-precision float options in RNTuple and highlights the differences and similarities to the already existing Double32_t and Float16_t.

[jblomer]

@pcanal is there still anything blocking this PR?

[pcanal]

Thanks.