Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new API design doc #447

Merged
merged 16 commits into from
Apr 25, 2022
Merged

Add new API design doc #447

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
ee800d4
Add new API design doc
jwodder Mar 3, 2021
1702bb2
Add get_raw_metadata()
jwodder Mar 4, 2021
6340c9b
Update for first round of comments
jwodder Mar 5, 2021
ffd6ac1
Remove the "_all" from "get_all_*" methods
jwodder Mar 8, 2021
9e2d414
Change `DandiMeta` to `DandisetMeta`
jwodder Mar 11, 2021
58500c1
Specify `get_metadata` default for `download()`
jwodder Mar 11, 2021
6d731db
Make `set_metadata()` modify objects in-place
jwodder Mar 11, 2021
6661c28
RemoteAsset.uuid → identifier
jwodder Mar 11, 2021
20cc7dc
Replace Asset.sha256 with `get_digest()`
jwodder Mar 11, 2021
3f62f3d
Add an implementation note
jwodder Mar 12, 2021
21f361a
Adjust progress dict fields
jwodder Mar 12, 2021
7d3266f
Add some methods
jwodder Apr 6, 2021
201b081
Updates
jwodder Jun 4, 2021
af512df
Remove "created" attribute from Asset classes
jwodder Jun 7, 2021
6f84091
Update User name fields
jwodder Jun 7, 2021
2b32d56
Don't do that.
jwodder Jun 7, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Change DandiMeta to DandisetMeta
  • Loading branch information
@jwodder
jwodder committed Jun 4, 2021
commit 9e2d41417c14a4a827b88601dc6633157ba567d5
12 changes: 6 additions & 6 deletions doc/design/python-api-1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Designs for an improved Python API
* `get_assets() -> Iterator[Asset]`
* `get_assets_under_path(path: str) -> Iterator[Asset]` — Returns all assets whose paths are either under the given folder or equal the given filepath
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UX sugarings we should consider add on top

@assets -> get_assets()
@metadata -> get_metadata()
__iter__() -> .get_assets()

? they might be tricky with subclassing though since IIRC we cannot just bind the property to get _assets in super class, since then subclasses .assets would us it instead of overloaded... May be properties should be lazy (cached value) invocations over the methods? That would give them a better purpose

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not clear on what exactly you're proposing here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

smth like

@property
def assets(self):
     if self._assets is None:
            self._assets = self.get_assets()
     return self._assets

def __iter__(self):
     for asset in self.assets:
          yield asset

but we can think about all the sugaring later

* `get_asset_by_path(path: str) -> Optional[Asset]` — Finds the unique asset with the given exact path; returns `None` if no such asset
* `get_metadata() -> DandiMeta`
* `get_metadata() -> DandisetMeta`
* `Asset`:
* `path: str`
* `sha256: str`
Expand All @@ -37,7 +37,7 @@ Designs for an improved Python API
* Add a `get_draft_dandiset(dandiset_id: Union[str, RemoteDandiset]) -> DraftDandiset` method
* Add a `get_dandisets() -> Iterator[RemoteDandiset]` method
* These objects' `Version`s are the `most_recent_version`s
* Retype `create_dandiset` to `create_dandiset(metadata: DandiMeta) -> RemoteDandiset`
* Retype `create_dandiset` to `create_dandiset(metadata: DandisetMeta) -> RemoteDandiset`
* Add a `create_local_dandiset(dirpath: Union[str, Path], name: str, description: str) -> LocalDandiset` method that, in addition to calling `create_dandiset()`, also creates a `dandiset.yaml` file in `dirpath`?
* Add a `get_current_user() -> User` method
* Add a `search_users(username: str) -> Iterator[User]` method
Expand All @@ -63,7 +63,7 @@ Designs for an improved Python API
* `get_users() -> Iterator[User]`
* `set_users(users: List[User]) -> None`
* Should this method also accept strings giving the usernames directly?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not bother interfacing users calls for now at all at this point

* `get_metadata() -> DandiMeta`
* `get_metadata() -> DandisetMeta`
* `get_raw_metadata() -> dict` — useful when the metadata is invalid
* `publish() -> Version` (Should this be moved to `DraftDandiset`?)
* `get_assets() -> Iterator[RemoteAsset]`
Expand All @@ -77,7 +77,7 @@ Designs for an improved Python API
* TO DO: Add an argument for controlling whether to create a directory under `target_dir` with the same name as the Dandiset ID?

* Methods of the `DraftDandiset` class (a subclass of `RemoteDandiset` used only for mutable draft versions):
* `set_metadata(metadata: DandiMeta) -> DraftDandiset` — returns the new, modified object
* `set_metadata(metadata: DandisetMeta) -> DraftDandiset` — returns the new, modified object
* Methods for uploading an individual asset:
* `upload_file(filepath: Union[str, Path], metadata: BareAssetMeta, show_progress=True, existing="refesh", validation="require") -> RemoteAsset`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

metadata should not be provided but extracted as part of the *upload_file*

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea of this method is to support generic file uploads where the user crafts the metadata themselves. If we only want to support upload of files that can be represented with LocalAsset instances, we can get rid of it.

* `iter_upload_file(filepath: Union[str, Path], metadata: BareAssetMeta, existing="refesh", validation="require") -> Iterator[UploadProgressDict]`
Expand Down Expand Up @@ -146,11 +146,11 @@ Designs for an improved Python API
* `done%: float` — percentage of bytes downloaded (Rename to "download"? or "pct"?)
* `checksum: str` — `"differs"`, `"ok"`, or `"-"`

* **To discuss:** Should methods that accept `DandiMeta` and `(Bare|Remote)AssetMeta` instances also accept raw `dict`s? If so, should any validation be done on these `dict`s?
* **To discuss:** Should methods that accept `DandisetMeta` and `(Bare|Remote)AssetMeta` instances also accept raw `dict`s? If so, should any validation be done on these `dict`s?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for starters we can just add a dedicated @classmethod from_dict(cls, d: dict, validate:bool=True) -> cls (possible?) and require such conversion explicitly. If we keep running into needing that too often -- we would rethink

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pydantic already provides parse_obj() (validating) and construct() (non-validating) methods for that.


* `dandi.dandiset`: Expand the ability for `Dandiset` (renamed to `LocalDandiset` and made a subclass of the new `Dandiset` base class) to represent a Dandiset on disk:
* `path: Path` (or `dandiset_path`?)
* `set_metadata(metadata: DandiMeta) -> None` ?
* `set_metadata(metadata: DandisetMeta) -> None` ?
* `validate_metadata() -> None`
* `validate_assets() -> ???` ?
* Make `organize()` a method of this class?
Expand Down