Buffer Prototype Argument #1910
Conversation
…er_prototype_argument
…er_prototype_argument
|
@normanrz if you can find time to review/merge this, it would be much appreciated. |
src/zarr/array_spec.py
Outdated
| dtype: np.dtype[Any] | ||
| fill_value: Any | ||
| order: Literal["C", "F"] | ||
| prototype: Prototype |
There was a problem hiding this comment.
Is the Prototype pickleable?
There was a problem hiding this comment.
The default_prototype, which is the one we use in ArrayV3Metadata, is pickleable.
AFAIK, it is not possible to type hint pickable but added the requirement in the docstring.
There was a problem hiding this comment.
The thing that has to be pickleable is the class, not the object itself. I'm having a hard time imagining any scenario in which a class is not pickleable.
|
I wonder if we should use the runtime config to configure the buffer implementation instead of a Prototype object that gets passed through? |
Is it possible/practical to set the runtime config for each |
Runtime config can be changed, but it is a global setting so as you pointed out it might change during chunk processing. Do you think it is useful to change the buffer class for multiple getitem calls? |
From the users perspective, using a buffer class per zarr arrays seems fine but mixing zarr arrays that are backed by different buffer classes is common. |
src/zarr/buffer.py
Outdated
| return prototype.buffer.from_bytes(func(buf.as_numpy_array())) | ||
|
|
||
|
|
||
| class Prototype(NamedTuple): |
There was a problem hiding this comment.
I am unsure whether the name Prototype conveys what the purpose of this class is. Is that a common way of calling such a construct? Maybe we could make it less general: BufferPrototype. @d-v-b any opinion on this?
There was a problem hiding this comment.
I think Template would be a better name.
edit: on the other hand, "prototype" is an established programming pattern: https://en.wikipedia.org/wiki/Prototype-based_programming
There was a problem hiding this comment.
i think prototype works, but I think the need for this class suggests that we might want to re-consider the hard boundary we have drawn between buffers and ndbuffers.
There was a problem hiding this comment.
I have no strong opinion, Template or BufferPrototype are also good names or maybe BufferTemplate ?
There was a problem hiding this comment.
No strong opinion, either, but if we had to decide, I'd go with BufferPrototype.
There was a problem hiding this comment.
Renamed to BufferPrototype
|
I'm afraid my recent PRs introduced some conflicts here. Would you mind cleaning these up? |
…er_prototype_argument
…er_prototype_argument
* v3: (22 commits) [v3] `Buffer` ensure correct subclass based on the `BufferPrototype` argument (zarr-developers#1974) Fix doc build (zarr-developers#1987) Fix doc build warnings (zarr-developers#1985) Automatically generate API reference docs (zarr-developers#1918) Update `RemoteStore.__str__` and add UPath tests (zarr-developers#1964) [v3] Elevate codec pipeline (zarr-developers#1932) 0 dim arrays: indexing (zarr-developers#1980) `parse_shapelike` allows 0 (zarr-developers#1979) Clean up typing and docs for indexing (zarr-developers#1961) add json indentation to config (zarr-developers#1952) chore: update pre-commit hooks (zarr-developers#1973) Bump pypa/gh-action-pypi-publish in the actions group (zarr-developers#1969) chore: update pre-commit hooks (zarr-developers#1957) Update release.rst (zarr-developers#1960) doc: update release notes for 3.0.0.alpha (zarr-developers#1959) Basic working FsspecStore (zarr-developers#1785) Feature: Top level V3 API (zarr-developers#1884) Buffer Prototype Argument (zarr-developers#1910) Create issue-metrics.yml fixes bug in transpose (zarr-developers#1949) ...
In order to pass array information down through the whole stack, codecs and stores now take a
prototypeargument. This ways, an user can control what buffers and arrays are created in the stack.This replaces the
factoryargument