feature(data-types): add data-types from https://github.com/zarr-developers/zarr-python/pull/2874 #5

jhamman · 2025-04-15T19:44:54Z

This PR adds the definition of the 4 new data types from zarr-developers/zarr-python#2874. They include:

datetime64
fixed-length-ascii
fixed-length-bytes
fixed-length-ucs4

Closes #4

…-bytes, and fixed-length-ucs4 data-types from zarr-python

d-v-b · 2025-04-15T19:53:50Z

data-types/datetime64/README.md

+
+## Permitted fill values
+
+The value of the `fill_value` metadata key must be a signed 64-bit integer.


If we want to stick with the integer representation, we should clarify that this is a JSON representation of an integer between -2^63 and 2^63 - 1, inclusive. But as this data type is totally new for zarr v3 and we don't have to worry about backwards compatibility in the json format w.r.t. v2, we could consider a more date-like human-readable alternative to an int. I'd love to get feedback from heavy datetime users on what they would expect.

d-v-b · 2025-04-15T19:59:02Z

data-types/datetime64/README.md

+}
+```
+
+## Notes


If numpy compatibility is still the design goal for this dtype, I think it would make sense to say something about compatibility with the numpy datetime dtype, e.g. with a link to the numpy docs .

d-v-b · 2025-04-15T20:09:18Z

data-types/fixed-length-ascii/README.md

@@ -0,0 +1,33 @@
+# Fixed-length ASCII data type
+
+Defines a data type for fixed-length ASCII strings.


I think my initial name for this dtype (fixed-length-ascii) was misleading -- it models the numpy S dtype, which represents any fixed-length byte string, which includes non-ascii characters. So we should probably rename this to fixed-length-bytes or equivalent.

i think the key distinction between this dtype (modelling numpy S*) and the fixed-length bytes dtype defined elsewhere in this PR (modelling numpy V*) is that scalars with this dtype are intended to be interpreted as strings, not opaque sequences of bytes. We should probably decide if we really want two dtypes for this.

Numpy also assumes NUL-padding, i.e. that any NUL characters at the end can be ignored.

d-v-b · 2025-04-15T20:25:46Z

data-types/fixed-length-ascii/README.md

+
+## Permitted fill values
+
+The value of the `fill_value` metadata key must be a string.


we will need more constraints here, such as as length constraint. for reference, in zarr v2 the fill value was a base64-encoded ascii string but it didn't constrain the length.

I agree that the length constraint makes sense. But not critical for merging this, imo.

jbms · 2025-04-15T21:00:42Z

data-types/datetime64/README.md

+
+## Notes
+
+Valid values for the `unit` configuration option include: `["Y", "M", "W", "D", "h", "m", "s", "ms", "us", "μs", "ns", "ps", "fs", "as"]`


This should say something about the epoch (presumably unix epoch) and time standard (presumably UTC) it uses.

d-v-b · 2025-04-15T21:00:45Z

instead of two fixed-length string data types (ascii and ucs4), we could consider a single fixed-length string dtype that's parametrized by the string encoding.

jbms · 2025-04-15T21:03:44Z

data-types/datetime64/README.md

+
+## Notes
+
+Valid values for the `unit` configuration option include: `["Y", "M", "W", "D", "h", "m", "s", "ms", "us", "μs", "ns", "ps", "fs", "as"]`


Note that the units listed after nanoseconds cannot represent dates beyond 1970 within 64-bits and therefore are presumably not very useful.

The first part of this sentence is correct.

The second is not. There is a lot of use cases in the (paleo-)climate space for dates that go much further back in time.

To be clear, the range of 64-bit picoseconds is just a few months forward or backward from the epoch.

Presumably picoseconds would only be useful either with more than 64 bits or for a time duration rather than time point.

is it helpful if we define the goal of this datetime type as "give numpy datetime64 users a zarr data type they can use without thinking"? In this case, we optimize for compatibility and accept whatever decisions the numpy folks have made. We can always define a separate datetime data type that avoids perceived defects in the numpy datetime dtype, and we can point users who don't need maximal numpy compatibility to the better data type.

Another comment regarding datetime: numpy supports a special NaT (not a time) value. Is that intended to be supported here?

i'm asking because, whether it's sensible or not, ps is a unit numpy datetime64 supports. Instead of litigating that decision, or any other numpy decision, we could optimize for compatibility here, which would simplify the task of this PR considerably.

Another comment regarding datetime: numpy supports a special NaT (not a time) value. Is that intended to be supported here?

yes i believe that should be supported, and so it should be defined in this PR

Those units don't hurt anything they just are very unlikely to be used intentionally. Numpy supports the same units for both datetime and timedelta --- are you intending to also add timedelta as well?

Those units don't hurt anything they just are very unlikely to be used intentionally. Numpy supports the same units for both datetime and timedelta --- are you intending to also add timedelta as well?

yes, timedelta is planned.

jbms · 2025-04-15T21:04:12Z

data-types/datetime64/schema.json

+            "type": "object",
+            "properties": {
+              "unit": {
+                "type": "string"


could be listed as enum rather than string

jbms · 2025-04-15T21:17:20Z

Regarding the datetime type generally, I think I'd be happier if that were handled by some sort of unit attribute that was independent of the data type.

For how this is handled by the udunits2 package, see: https://docs.unidata.ucar.edu/udunits/current/udunits2lib.html#Time

In udunits the unit would be specified as "seconds since 1970-01-01".

The complication with making it a data type is that you have to specify how it is handled by every single codec.

jbms · 2025-04-15T21:18:52Z

All of these need to specify which array -> bytes codecs support them (presumably just bytes), and how they are encoded.

jbms · 2025-04-15T21:23:53Z

data-types/fixed-length-ascii/README.md

+
+## Permitted fill values
+
+The value of the `fill_value` metadata key must be a string.


Why is the length specified in bits? What does it mean if you specify a length that is not a multiple of 8?

I think specifying the length in bits is a zarr-python implementation detail that we shouldn't have to deal with in the spec. i think a length in bytes is better here.

I also think that a "native" length key would be more intuitive for all of these dtypes. With "native", I mean 1-byte-lengths for ascii and bytes and 4-byte-lengths for utf-32.

jbms · 2025-04-15T21:25:09Z

data-types/fixed-length-ucs4/README.md

@@ -0,0 +1,33 @@
+# Fixed-length Unicode string data type
+
+Defines a data type for fixed-length Unicode strings.


Probably should also include a recommendation NOT to use this type except for compatibility with existing datasets, because UTF-32 encoding is basically never a good choice.

Probably should include a note that it is intended to only contain valid unicode strings, and that implementations may return an error if given invalid data (e.g. utf-16 surrogate pairs, out-of-range characters) if that is what is intended.

Should include a note that NUL characters at the end are to be ignored, if that is what is intended.

jbms · 2025-04-15T21:25:49Z

data-types/fixed-length-ucs4/README.md

+    "data_type": "fixed-length-ucs4",
+    "fill_value": "",
+    "configuration": {
+      "length_bits": 24


Using 24, which is not a multiple of 32, is confusing.

jbms · 2025-04-15T21:26:05Z

data-types/fixed-length-ucs4/schema.json

+        "configuration": {
+          "type": "object",
+          "properties": {
+            "length_bits": {


why is the length in bits, and what happens if it is not a multiple of 32?

jbms · 2025-04-15T21:26:50Z

data-types/fixed-length-ucs4/README.md

@@ -0,0 +1,33 @@
+# Fixed-length Unicode string data type
+
+Defines a data type for fixed-length Unicode strings.


I think utf-32 is now more commonly used and may be a better choice than ucs-4.

To be clear, I mean the name "utf-32".

jhamman · 2025-04-16T04:00:38Z

Thanks @d-v-b and @jbms for the initial reviews. I'll take a second pass tomorrow.

normanrz · 2025-04-16T19:17:09Z

data-types/datetime64/README.md

+    "data_type": "datetime64",
+    "fill_value": -9223372036854775808,
+    "configuration": {
+      "unit": "s"
+    },


Suggested change

"data_type": "datetime64",

"fill_value": -9223372036854775808,

"configuration": {

"unit": "s"

},

"data_type": { "name": "datetime64", "configuration": {"unit":"s"} },

"fill_value": -9223372036854775808,

normanrz · 2025-04-16T19:18:44Z

data-types/datetime64/schema.json

+        "required": ["name", "configuration"],
+        "additionalProperties": false
+      },
+      { "const": "datetime64" }


If the configuration (with its unit field) is mandatory, the short-hand should not be permissible.

normanrz · 2025-05-06T17:15:08Z

Since you mentioned that you are still working on this, I turned this PR into a draft PR, in accordance with #13. Feel free to switch to a regular PR as soon as you think it is ready.

…ions into zarr-python-dtypes

…extensions into zarr-python-dtypes

fill out numpy `U` dtype

normanrz

There are still a few open discussion items that I think would be nice to resolve.
However, the PR already meets the formal criteria for merging. Please let me know.

data-types/fixed-length-utf32/schema.json

normanrz · 2025-06-02T15:05:26Z

data-types/fixed-length-utf32/schema.json

+  "oneOf": [
+    {
+      "type": "object",
+      "properties": {
+        "name": {
+          "const": "fixed_length_utf32"
+        },
+        "configuration": {
+          "type": "object",
+          "properties": {
+            "length_bits": {
+              "type": "integer"
+            }
+          },
+          "required": ["length_bytes"],
+          "additionalProperties": false
+        }
+      },
+      "required": ["name", "configuration"],
+      "additionalProperties": false
+    },
+    { "const": "fixed_length_utf32" }
+  ]


Since a configuration with length_bytes is required, the short form should not be permissible. The same applies to the other dtypes.

Suggested change

"oneOf": [

{

"type": "object",

"properties": {

"name": {

"const": "fixed_length_utf32"

},

"configuration": {

"type": "object",

"properties": {

"length_bits": {

"type": "integer"

}

},

"required": ["length_bytes"],

"additionalProperties": false

}

},

"required": ["name", "configuration"],

"additionalProperties": false

},

{ "const": "fixed_length_utf32" }

]

{

"type": "object",

"properties": {

"name": {

"const": "fixed_length_utf32"

},

"configuration": {

"type": "object",

"properties": {

"length_bytes": {

"type": "integer"

}

},

"required": ["length_bytes"],

"additionalProperties": false

}

},

"required": ["name", "configuration"],

"additionalProperties": false

}

normanrz · 2025-06-02T15:08:36Z

data-types/fixed-length-ascii/README.md

+
+## Permitted fill values
+
+The value of the `fill_value` metadata key must be a string.


I agree that the length constraint makes sense. But not critical for merging this, imo.

normanrz · 2025-06-02T15:11:13Z

data-types/fixed-length-ascii/README.md

+
+## Permitted fill values
+
+The value of the `fill_value` metadata key must be a string.


I also think that a "native" length key would be more intuitive for all of these dtypes. With "native", I mean 1-byte-lengths for ascii and bytes and 4-byte-lengths for utf-32.

Co-authored-by: Norman Rzepka <code@normanrz.com>

d-v-b · 2025-06-03T16:10:33Z

A question about fill values: the v2 spec required that string fill values be encoded via bas64. Do we want to use the same encoding here?

normanrz · 2025-06-03T17:38:15Z

A question about fill values: the v2 spec required that string fill values be encoded via bas64. Do we want to use the same encoding here?

I was wondering the same thing. For ascii it seems a bit pointless. For bytes and utf-32 it might be more intuitive/safer to encode, though.

d-v-b · 2025-06-03T18:00:21Z

I imagine base64 encoding the ascii was necessary for unprintable characters, and the same constraint might lead us to base64 encode unicode, too, but the json spec defines a scheme for escaping any unicode character, so maybe we should just use that, for ascii and unicode data types alike? This has the advantage of working automatically with JSON libraries:

json.loads(b'{"foo": "\u0000"}')
{'foo': '\x00'}

feature(data-types): add datetime64, fixed-length-ascii, fixed-length…

89ea29d

…-bytes, and fixed-length-ucs4 data-types from zarr-python

d-v-b reviewed Apr 15, 2025

View reviewed changes

jbms reviewed Apr 15, 2025

View reviewed changes

jhamman mentioned this pull request Apr 16, 2025

adds extensions that zarr-python defines #1

Merged

normanrz reviewed Apr 16, 2025

View reviewed changes

abarciauskas-bgse mentioned this pull request Apr 25, 2025

ODD PI 25.3 Objective 5: Foundational Zarr-Python and Xarray Contributions NASA-IMPACT/veda-odd#165

Open

3 tasks

mkitti mentioned this pull request Apr 27, 2025

Reserve characters in names #7

Open

normanrz marked this pull request as draft May 6, 2025 17:12

d-v-b mentioned this pull request May 19, 2025

compatibility with zarr dtypes refactor pydata/xarray#10333

Open

d-v-b added 2 commits May 27, 2025 14:24

Merge branch 'main' of https://github.com/zarr-developers/zarr-extens…

ba09d7e

…ions into zarr-python-dtypes

remove datetime stuff, rename ucs4 to utf32, fill out schema

1be0c9e

jhamman marked this pull request as ready for review May 29, 2025 16:19

jhamman and others added 3 commits May 29, 2025 09:20

Merge branch 'main' of https://github.com/zarr-developers/zarr-extens…

aa21bfa

…ions into zarr-python-dtypes

Merge branch 'zarr-python-dtypes' of https://github.com/jhamman/zarr-…

46ac0b8

…extensions into zarr-python-dtypes

Merge pull request #1 from d-v-b/zarr-python-dtypes

13f8a4a

fill out numpy `U` dtype

normanrz reviewed Jun 2, 2025

View reviewed changes

d-v-b and others added 2 commits June 2, 2025 19:36

Update data-types/fixed-length-utf32/schema.json

bbc69d1

Co-authored-by: Norman Rzepka <code@normanrz.com>

add null terminated clarification

7894f9b


		## Permitted fill values

		The value of the `fill_value` metadata key must be a signed 64-bit integer.

		@@ -0,0 +1,33 @@
		# Fixed-length ASCII data type

		Defines a data type for fixed-length ASCII strings.


		## Notes

		Valid values for the `unit` configuration option include: `["Y", "M", "W", "D", "h", "m", "s", "ms", "us", "μs", "ns", "ps", "fs", "as"]`

		@@ -0,0 +1,33 @@
		# Fixed-length Unicode string data type

		Defines a data type for fixed-length Unicode strings.

feature(data-types): add data-types from https://github.com/zarr-developers/zarr-python/pull/2874 #5

Are you sure you want to change the base?

feature(data-types): add data-types from https://github.com/zarr-developers/zarr-python/pull/2874 #5

Uh oh!

Conversation

jhamman commented Apr 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d-v-b commented Apr 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbms commented Apr 15, 2025

Uh oh!

jbms commented Apr 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jhamman commented Apr 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

normanrz commented May 6, 2025

Uh oh!

normanrz left a comment

Choose a reason for hiding this comment

Uh oh!