You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
bfloat16 is very widely used for machine learning, much more widely used than IEEE 754-2008 float16.
bfloat16 is not natively supported by NumPy, but TensorStore, JAX, and TensorFlow all add bfloat16 support to NumPy when they are loaded.
However, the NumPy type string syntax does not easily accommodate bfloat16.
TensorStore already supports bfloat16 with the zarr format as an extension. In .zarray the "dtype" is specified as "bfloat16" (which always means bfloat16 little endian). This deviates from the usual NumPy type string syntax, but was chosen because it allows unmodified zarr-python to read bfloat16 data written by TensorStore, provided that a bfloat16 data type (as provided by JAX, TensorFlow, and TensorStore) has been registered with NumPy, due to the way zarr-python resolves data types.
It would be great if zarr v3 specifically supported bfloat16 to allow for better interoperability.
The text was updated successfully, but these errors were encountered:
The syntax is now consistent with "bfloat16" as of #155. I'd propose to add bfloat16 as a dtype extension, as it could not easily be supported by zarr-python without requiring jax/tensorflow. Also cross-linking numpy/numpy#19808, which proposes to add bfloat16 to numpy directly.
bfloat16 is very widely used for machine learning, much more widely used than IEEE 754-2008 float16.
bfloat16 is not natively supported by NumPy, but TensorStore, JAX, and TensorFlow all add bfloat16 support to NumPy when they are loaded.
However, the NumPy type string syntax does not easily accommodate bfloat16.
TensorStore already supports bfloat16 with the zarr format as an extension. In
.zarray
the"dtype"
is specified as"bfloat16"
(which always means bfloat16 little endian). This deviates from the usual NumPy type string syntax, but was chosen because it allows unmodified zarr-python to read bfloat16 data written by TensorStore, provided that a bfloat16 data type (as provided by JAX, TensorFlow, and TensorStore) has been registered with NumPy, due to the way zarr-python resolves data types.It would be great if zarr v3 specifically supported bfloat16 to allow for better interoperability.
The text was updated successfully, but these errors were encountered: