Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ntuple] add Real32Quant column type #16390

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
+602 −38
Open

[ntuple] add Real32Quant column type #16390

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
c14659e
[ntuple] add Real32Quant column type
silverweed Aug 14, 2024
2e8f47b
[ntuple] add more tests for Real32Quant, set min bits to 1
silverweed Sep 10, 2024
488cc12
[ntuple] move ValueRange from Field desc to Column desc
silverweed Sep 10, 2024
2423ab5
[ntuple] add column ValueRange logic
silverweed Sep 10, 2024
7a6d557
[ntuple] fix specification table
silverweed Sep 10, 2024
9e6b1df
[ntuple] in QuantizeReal, round instead of flooring
silverweed Sep 10, 2024
de71160
[ntuple] fix ValueRange being always non-null on disk
silverweed Sep 10, 2024
4b6d742
[ntuple] add exhaustive tests for Real32Quant
silverweed Sep 11, 2024
095babf
[ntuple] improve specification
silverweed Sep 16, 2024
984bc4a
[ntuple] make Real32Quant throws on out of range in Pack/Unpack
silverweed Sep 18, 2024
d6c9c0f
[ntuple] add paragraph to architecture on low-prec floats
silverweed Sep 23, 2024
64700ac
[ntuple] simpler unquantization formula
silverweed Sep 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions tree/ntuple/v7/doc/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,6 +450,47 @@ Every fill context prepares a set of entire clusters in the final on-disk layout
When a fill context flushes data,
a brief serialization point handles the RNTuple meta-data updates and the reservation of disk space to write into.

Low precision float types
--------------------------
RNTuple supports encoding floating point types with a lower precision when writing them to disk. This encoding is specified by the
user per field and it is independent on the in-memory type used for that field (meaning both a `RField<double>` or `RField<float>` can
be mapped to e.g. a low-precision 16 bit float).

RNTuple supports the following encodings (all mutually exclusive):

- **Real16**/**SplitReal16**: IEEE-754 half precision float. Set by calling `RField::SetHalfPrecision()`;
- **Real32Trunc**: floating point with less than 32 bits of precision (truncated mantissa).
Set by calling `RField::SetTruncated(n)`, with $10 <= n <= 31$ equal to the total number of bits used on disk.
Note that `SetTruncated(16)` makes this effectively a `bfloat16` on disk;
- **Real32Quant**: floating point with a normalized/quantized integer representation on disk using a user-specified number of bits.
Set by calling `RField::SetQuantized(min, max, nBits)`, where $1 <= nBits <= 32$.
This representation will map the floating point value `min` to 0, `max` to the highest representable integer with `nBits` and any
value in between will be a linear interpolation of the two. It is up to the user to ensure that only values between `min` and `max`
are stored in this field. The current RNTuple implementation will throw an exception if that is not the case when writing the values to disk.

In addition to these encodings, a user may call `RField<double>::SetDouble32()` to set the column representation of a `double` field to
a 32-bit floating point value. The default behavior of `Float16_t` can be emulated by calling `RField::SetTruncated(21)` (which will truncate
a single precision float's mantissa to 12 bits).

Here is an example on how a user may dynamically decide how to quantize a floating point field to get the most precision out of a fixed bit width:
```c++
auto model = RNTupleModel::Create();
auto field = std::make_unique<RField<float>>("f");
// assuming we have an array of floats stored in `myFloats`:
auto [minV, maxV] = std::minmax_element(myFloats.begin(), myFloats.end());
constexpr auto nBits = 24;
field->SetQuantized(*minV, *maxV, nBits);
model->AddField(std::move(field));
auto f = model->GetDefaultEntry().GetPtr<float>("f");

// Now we can write our floats.
auto writer = RNTupleWriter::Recreate(std::move(model), "myNtuple", "myFile.root");
for (float val : myFloats) {
*f = val;
writer->Fill();
}
```

Relationship to other ROOT components
-------------------------------------

Expand Down
89 changes: 55 additions & 34 deletions tree/ntuple/v7/doc/specifications.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -415,7 +415,7 @@ the field has been created as a virtual field from another, non-projected source
If a projected field has attached columns,
these columns are alias columns to physical columns attached to the source field.

If `flag==0x04` (type checksum) is set, the field metadata contain the checksum of the ROOT streamer info.
If `flag==0x04` (_type checksum_) is set, the field metadata contain the checksum of the ROOT streamer info.
This checksum is only used for I/O rules in order to find types that are identified by checksum.

Depending on the flags, the following optional values follow:
Expand Down Expand Up @@ -459,6 +459,18 @@ Top-level fields have their own field ID set as parent ID.
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Flags | Representation Index |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ First element index (if flag 0x08 is set) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Min value (if flag 0x10 is set) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ Max value (if flag 0x10 is set) +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
```

The order of columns matter: every column gets an implicit column ID
Expand All @@ -477,37 +489,38 @@ The representation index is consecutive starting at zero.

The column type and bits on storage integers can have one of the following values

| Type | Bits | Name | Contents |
|------|------|--------------|-------------------------------------------------------------------------------|
| 0x01 | 64 | Index64 | Parent columns of (nested) collections, counting is relative to the cluster |
| 0x02 | 32 | Index32 | Parent columns of (nested) collections, counting is relative to the cluster |
| 0x03 | 96 | Switch | Tuple of a kIndex64 value followed by a 32 bits dispatch tag to a column ID |
| 0x04 | 8 | Byte | An uninterpreted byte, e.g. part of a blob |
| 0x05 | 8 | Char | ASCII character |
| 0x06 | 1 | Bit | Boolean value |
| 0x07 | 64 | Real64 | IEEE-754 double precision float |
| 0x08 | 32 | Real32 | IEEE-754 single precision float |
| 0x09 | 16 | Real16 | IEEE-754 half precision float |
| 0x16 | 64 | Int64 | Two's complement, little-endian 8 byte signed integer |
| 0x0A | 64 | UInt64 | Little-endian 8 byte unsigned integer |
| 0x17 | 32 | Int32 | Two's complement, little-endian 4 byte signed integer |
| 0x0B | 32 | UInt32 | Little-endian 4 byte unsigned integer |
| 0x18 | 16 | Int16 | Two's complement, little-endian 2 byte signed integer |
| 0x0C | 16 | UInt16 | Little-endian 2 byte unsigned integer |
| 0x19 | 8 | Int8 | Two's complement, 1 byte signed integer |
| 0x0D | 8 | UInt8 | 1 byte unsigned integer |
| 0x0E | 64 | SplitIndex64 | Like Index64 but pages are stored in split + delta encoding |
| 0x0F | 32 | SplitIndex32 | Like Index32 but pages are stored in split + delta encoding |
| 0x10 | 64 | SplitReal64 | Like Real64 but in split encoding |
| 0x11 | 32 | SplitReal32 | Like Real32 but in split encoding |
| 0x12 | 16 | SplitReal16 | Like Real16 but in split encoding |
| 0x1A | 64 | SplitInt64 | Like Int64 but in split + zigzag encoding |
| 0x13 | 64 | SplitUInt64 | Like UInt64 but in split encoding |
| 0x1B | 64 | SplitInt32 | Like Int32 but in split + zigzag encoding |
| 0x14 | 32 | SplitUInt32 | Like UInt32 but in split encoding |
| 0x1C | 16 | SplitInt16 | Like Int16 but in split + zigzag encoding |
| 0x15 | 16 | SplitUInt16 | Like UInt16 but in split encoding |
| 0x1D |10-31 | Real32Trunc | IEEE-754 single precision float with truncated mantissa |
| Type | Bits | Name | Contents |
|------|------|--------------|-----------------------------------------------------------------------------------------------|
| 0x01 | 64 | Index64 | Parent columns of (nested) collections, counting is relative to the cluster |
| 0x02 | 32 | Index32 | Parent columns of (nested) collections, counting is relative to the cluster |
| 0x03 | 96 | Switch | Tuple of a kIndex64 value followed by a 32 bits dispatch tag to a column ID |
| 0x04 | 8 | Byte | An uninterpreted byte, e.g. part of a blob |
| 0x05 | 8 | Char | ASCII character |
| 0x06 | 1 | Bit | Boolean value |
| 0x07 | 64 | Real64 | IEEE-754 double precision float |
| 0x08 | 32 | Real32 | IEEE-754 single precision float |
| 0x09 | 16 | Real16 | IEEE-754 half precision float |
| 0x16 | 64 | Int64 | Two's complement, little-endian 8 byte signed integer |
| 0x0A | 64 | UInt64 | Little-endian 8 byte unsigned integer |
| 0x17 | 32 | Int32 | Two's complement, little-endian 4 byte signed integer |
| 0x0B | 32 | UInt32 | Little-endian 4 byte unsigned integer |
| 0x18 | 16 | Int16 | Two's complement, little-endian 2 byte signed integer |
| 0x0C | 16 | UInt16 | Little-endian 2 byte unsigned integer |
| 0x19 | 8 | Int8 | Two's complement, 1 byte signed integer |
| 0x0D | 8 | UInt8 | 1 byte unsigned integer |
| 0x0E | 64 | SplitIndex64 | Like Index64 but pages are stored in split + delta encoding |
| 0x0F | 32 | SplitIndex32 | Like Index32 but pages are stored in split + delta encoding |
| 0x10 | 64 | SplitReal64 | Like Real64 but in split encoding |
| 0x11 | 32 | SplitReal32 | Like Real32 but in split encoding |
| 0x12 | 16 | SplitReal16 | Like Real16 but in split encoding |
| 0x1A | 64 | SplitInt64 | Like Int64 but in split + zigzag encoding |
| 0x13 | 64 | SplitUInt64 | Like UInt64 but in split encoding |
| 0x1B | 64 | SplitInt32 | Like Int32 but in split + zigzag encoding |
| 0x14 | 32 | SplitUInt32 | Like UInt32 but in split encoding |
| 0x1C | 16 | SplitInt16 | Like Int16 but in split + zigzag encoding |
| 0x15 | 16 | SplitUInt16 | Like UInt16 but in split encoding |
| 0x1D |10-31 | Real32Trunc | IEEE-754 single precision float with truncated mantissa |
| 0x1E | 1-32 | Real32Quant | Real value contained in a specified range with an underlying quantized integer representation |

The "split encoding" columns apply a byte transformation encoding to all pages of that column
and in addition, depending on the column type, delta or zigzag encoding:
Expand All @@ -529,6 +542,9 @@ not cluster-wise.
The "Real32Trunc" type column is a variable-sized floating point column with lower precision than `Real32` and `SplitReal32`.
It is a IEEE-754 single precision float with some of the mantissa's least significant bits truncated.

The "Real32Quant" type column is a variable-sized real column that is internally represented as an integer within
a specified range of values. For this column type, flag 0x10 (column with range) is always set (see paragraphs below).

Future versions of the file format may introduce additional column types
without changing the minimum version of the header or introducing a feature flag.
Old readers need to ignore these columns and fields constructed from such columns.
Expand All @@ -539,14 +555,18 @@ The flags field can have one of the following bits set
| Bit | Meaning |
|----------|-------------------------------------------------------------------|
| 0x08 | Deferred column: index of first element in the column is not zero |
| 0x10 | Column with a range of possible values |

If flag 0x08 (deferred column) is set, the index of the first element in this column is not zero, which happens if the column is added at a later point during write.
In this case, an additional 64bit integer containing the first element index follows the flags field.
In this case, an additional 64bit integer containing the first element index follows the representation index field.
Compliant implementations should yield synthetic data pages made up of 0x00 bytes when trying to read back elements in the range $[0, firstElementIndex-1]$.
This results in zero-initialized values in the aforementioned range for fields of any supported C++ type, including `std::variant<Ts...>` and collections such as `std::vector<T>`.
The leading zero pages of deferred columns are _not_ part of the page list, i.e. they have no page locator.
In practice, deferred columns only appear in the schema extension record frame (see Section Footer Envelope).

If flag 0x10 (column with range) is set, the column metadata contains the inclusive range of valid values
for this column (used e.g. for quantized real values). The range is represented as a min and a max value, specified as IEEE 754 little-endian double precision floats.

If the index of the first element is negative (sign bit set), the column is deferred _and_ suppressed.
In this case, no (synthetic) pages exist up to and including the cluster of the first element index.
See Section "Page List Envelope" for further information about suppressed columns.
Expand Down Expand Up @@ -839,7 +859,8 @@ Such cases are marked as `R` in the table.
| Real16 | | | | | | | | | | | | W | W |
| (Split)Real32 | | | | | | | | | | | | W* | W |
| (Split)Real64 | | | | | | | | | | | | | W* |
| Real32Trunc | | | | | | | | | | | | W* | |
| Real32Trunc | | | | | | | | | | | | W | |
| Real32Quant | | | | | | | | | | | | W | W |

Possibly available `const` and `volatile` qualifiers of the C++ types are ignored for serialization.
The default column for serialization is denoted with an asterix.
Expand Down
6 changes: 6 additions & 0 deletions tree/ntuple/v7/inc/ROOT/RColumn.hxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -333,6 +333,11 @@ public:
assert(fElement);
return fElement->GetBitsOnStorage();
}
std::optional<std::pair<double, double>> GetValueRange() const
{
assert(fElement);
return fElement->GetValueRange();
}
std::uint32_t GetIndex() const { return fIndex; }
std::uint16_t GetRepresentationIndex() const { return fRepresentationIndex; }
DescriptorId_t GetOnDiskId() const { return fOnDiskId; }
Expand All @@ -344,6 +349,7 @@ public:

void SetBitsOnStorage(std::size_t bits) { fElement->SetBitsOnStorage(bits); }
std::size_t GetWritePageCapacity() const { return fWritePage.GetCapacity(); }
void SetValueRange(double min, double max) { fElement->SetValueRange(min, max); }
}; // class RColumn

} // namespace ROOT::Experimental::Internal
Expand Down
9 changes: 9 additions & 0 deletions tree/ntuple/v7/inc/ROOT/RColumnElementBase.hxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
#include <cstddef> // for std::byte
#include <cstdint>
#include <memory>
#include <optional>
#include <string>
#include <type_traits>
#include <typeinfo>
Expand All @@ -54,6 +55,8 @@ protected:
/// Size of the C++ value that corresponds to the on-disk element
std::size_t fSize;
std::size_t fBitsOnStorage;
/// This is only meaningful for column elements that support it (e.g. Real32Quant)
std::optional<std::pair<double, double>> fValueRange = std::nullopt;

explicit RColumnElementBase(std::size_t size, std::size_t bitsOnStorage = 0)
: fSize(size), fBitsOnStorage(bitsOnStorage ? bitsOnStorage : 8 * size)
Expand Down Expand Up @@ -89,6 +92,11 @@ public:
throw RException(R__FAIL(std::string("internal error: cannot change bit width of this column type")));
}

virtual void SetValueRange(double, double)
{
throw RException(R__FAIL(std::string("internal error: cannot change value range of this column type")));
}

/// If the on-storage layout and the in-memory layout differ, packing creates an on-disk page from an in-memory page
virtual void Pack(void *destination, const void *source, std::size_t count) const
{
Expand All @@ -103,6 +111,7 @@ public:

std::size_t GetSize() const { return fSize; }
std::size_t GetBitsOnStorage() const { return fBitsOnStorage; }
std::optional<std::pair<double, double>> GetValueRange() const { return fValueRange; }
std::size_t GetPackedSize(std::size_t nElements = 1U) const { return (nElements * fBitsOnStorage + 7) / 8; }
}; // class RColumnElementBase

Expand Down
Loading
Loading