Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRAFT] V3 spec implmentation. #568

Closed
wants to merge 2 commits into from

Conversation

Carreau
Copy link
Contributor

@Carreau Carreau commented Jun 2, 2020

This is mostly opened to foster discussion and need code from https://github.com/Carreau/zarr-spec-v3-impl.

IN the above mentioned repository I'm working on looking at what an
implementation of the spec v3 could look to inform us on the possible
transition and compatiblity shims.

So far I have a rough implementation of an in memory v3 store as well as
multiple utilities.

  1. A base class that automatically provide sync version of all async
    method of a class. I'm playing with the idea of having most method
    async as this may be useful in some context.

For example when creating a array at /a/b/c/d/e/f/g/h/i/j/k you want
to check that None of the parents are arrays, which can be done with N
async requests.

  1. An adapted class that wraps a v3 store and provide a v2 API.
    My though is that most code is currently v2 compatible and it would be
    useful for legacy codebase and early testing of store.

  2. a class that wrap 2 stores, a reference and a tested one, replicate
    operation on both stores, and abort if it sees any difference in
    behavior. This could help to catch changes in behavior.

The tests in this PR start to test the v3 memorystore and compare it to
the v2 memorystore.

@pep8speaks
Copy link

pep8speaks commented Jun 2, 2020

Hello @Carreau! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

Line 26:1: E402 module level import not at top of file
Line 218:75: E203 whitespace before ':'

Comment last updated at 2020-08-26 16:50:57 UTC

@Carreau
Copy link
Contributor Author

Carreau commented Jun 26, 2020

Question on v3 implementation route. Do you want a complete rewrite in a separate module ? Or do you want to still use the current Group/Array class as well as the utilities functions in zarr.storage / zarr.meta ...etc.

Many of these function will need to be mave "version aware", some of them might be able to get the current version as they are given a store. But other (decode/encode_array_metadata) are not and should be likely given a kwarg.

This is mostly opened to foster discussion and need code from https://github.com/Carreau/zarr-spec-v3-impl.

IN the above mentioned repository I'm working on looking at what an
implementation of the spec v3 could look to inform us on the possible
transition and compatiblity shims.

So far I have a rough implementation of an in memory v3 store as well as
multiple utilities.

1) A base class that automatically provide  sync version of all async
method of a class. I'm playing with the idea of having most method
async as this may be useful in some context.

For example when creating a array at   /a/b/c/d/e/f/g/h/i/j/k you want
to check that None of the parents are arrays, which can be done with N
async requests.

2) An adapted class that wraps a v3 store and provide a v2 API.
My though is that most code is currently v2 compatible and it would be
useful for legacy codebase and early testing of store.

3) a class that wrap 2 stores, a reference and a tested one, replicate
operation on both stores, and abort if it sees any difference in
behavior. This could help to catch changes in behavior.

The tests in this PR start to test the v3 memorystore and compare it to
the v2 memorystore.
This is mostly opened to foster discussion and need code from https://github.com/Carreau/zarr-spec-v3-impl.

IN the above mentioned repository I'm working on looking at what an
implementation of the spec v3 could look to inform us on the possible
transition and compatiblity shims.

So far I have a rough implementation of an in memory v3 store as well as
multiple utilities.

1) A base class that automatically provide  sync version of all async
method of a class. I'm playing with the idea of having most method
async as this may be useful in some context.

For example when creating a array at   /a/b/c/d/e/f/g/h/i/j/k you want
to check that None of the parents are arrays, which can be done with N
async requests.

2) An adapted class that wraps a v3 store and provide a v2 API.
My though is that most code is currently v2 compatible and it would be
useful for legacy codebase and early testing of store.

3) a class that wrap 2 stores, a reference and a tested one, replicate
operation on both stores, and abort if it sees any difference in
behavior. This could help to catch changes in behavior.

The tests in this PR start to test the v3 memorystore and compare it to
the v2 memorystore.
@Carreau
Copy link
Contributor Author

Carreau commented Sep 28, 2020

Pr was at 96732fc97e86fdb07f84d8b307a11d1f311a0fc3 , will rebase /squash and force-push.

@joshmoore
Copy link
Member

This has been completed by #898 (plus follow ons like #1007)

@joshmoore joshmoore closed this May 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants