Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v3: support for bfloat16 #130

Open
jbms opened this issue Feb 8, 2022 · 1 comment
Open

v3: support for bfloat16 #130

jbms opened this issue Feb 8, 2022 · 1 comment
Labels
data-type protocol-extension Protocol extension related issue

Comments

@jbms
Copy link
Contributor

jbms commented Feb 8, 2022

bfloat16 is very widely used for machine learning, much more widely used than IEEE 754-2008 float16.

bfloat16 is not natively supported by NumPy, but TensorStore, JAX, and TensorFlow all add bfloat16 support to NumPy when they are loaded.

However, the NumPy type string syntax does not easily accommodate bfloat16.

TensorStore already supports bfloat16 with the zarr format as an extension. In .zarray the "dtype" is specified as "bfloat16" (which always means bfloat16 little endian). This deviates from the usual NumPy type string syntax, but was chosen because it allows unmodified zarr-python to read bfloat16 data written by TensorStore, provided that a bfloat16 data type (as provided by JAX, TensorFlow, and TensorStore) has been registered with NumPy, due to the way zarr-python resolves data types.

It would be great if zarr v3 specifically supported bfloat16 to allow for better interoperability.

@jstriebel
Copy link
Member

The syntax is now consistent with "bfloat16" as of #155. I'd propose to add bfloat16 as a dtype extension, as it could not easily be supported by zarr-python without requiring jax/tensorflow. Also cross-linking numpy/numpy#19808, which proposes to add bfloat16 to numpy directly.

@jstriebel jstriebel added protocol-extension Protocol extension related issue data-type labels Nov 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data-type protocol-extension Protocol extension related issue
Projects
None yet
Development

No branches or pull requests

2 participants