Skip to content

Commit

Permalink
Rename 'node' to 'container' and shorten URLs. (bluesky#478)
Browse files Browse the repository at this point in the history
* Rename structure family 'node' to 'container', shorten URLs.

* Update tree util.

* Make test of inlined HFD5 content more thorough.

* Update URLs.

* Satisfy linters.

* Update renamed queries in docs.

* Rename client node module and Node class.
  • Loading branch information
danielballan authored and dylanmcreynolds committed Jul 7, 2023
1 parent bca5c0b commit c062d8a
Show file tree
Hide file tree
Showing 42 changed files with 1,196 additions and 1,162 deletions.
4 changes: 2 additions & 2 deletions docs/source/explanations/compression.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,8 +142,8 @@ Finally, in this example. the server decides that the raw, compressed content is
so small (304 bytes) that is isn't not worth compressing.

```
$ http -p Hh :8000/node/metadata/
GET /node/metadata/ HTTP/1.1
$ http -p Hh :8000/metadata/
GET /metadata/ HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Expand Down
18 changes: 9 additions & 9 deletions docs/source/explanations/structures.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ a string label for each dimension.
This `(10, 10)`-shaped array fits in a single `(10, 10)`-shaped chunk.

```
$ http :8000/node/metadata/small_image | jq .data.attributes.structure
$ http :8000/metadata/small_image | jq .data.attributes.structure
```

```json
Expand Down Expand Up @@ -101,7 +101,7 @@ This `(10000, 10000)`-shaped array is subdivided into 4 × 4 = 16 chunks,
which is why the size of each chunk is given explicitly.

```
$ http :8000/node/metadata/big_image | jq .data.attributes.structure
$ http :8000/metadata/big_image | jq .data.attributes.structure
```

```json
Expand Down Expand Up @@ -142,7 +142,7 @@ This is a 1D array where each item has internal structure,
as in numpy's [strucuted data types](https://numpy.org/doc/stable/user/basics.rec.html)

```
$ http :8000/node/metadata/structured_data/pets | jq .data.attributes.structure
$ http :8000/metadata/structured_data/pets | jq .data.attributes.structure
```

```json
Expand Down Expand Up @@ -241,7 +241,7 @@ order, but we cannot make requests like "rows 100-200". (Dask has the same
limitation, for the same reason.)

```
$ http :8000/node/metadata/long_table | jq .data.attributes.structure
$ http :8000/metadata/long_table | jq .data.attributes.structure
```

```json
Expand Down Expand Up @@ -354,7 +354,7 @@ In certain cases, it is efficient to in-line all the information about the node'
"links": {
"block": "http://localhost:8000/api/v1/array/block/structured_data/xarray_dataset/lat?block={index_0},{index_1}",
"full": "http://localhost:8000/api/v1/array/full/structured_data/xarray_dataset/lat",
"self": "http://localhost:8000/api/v1/node/metadata/structured_data/xarray_dataset/lat"
"self": "http://localhost:8000/api/v1/metadata/structured_data/xarray_dataset/lat"
},
"meta": null
},
Expand Down Expand Up @@ -401,7 +401,7 @@ In certain cases, it is efficient to in-line all the information about the node'
"links": {
"block": "http://localhost:8000/api/v1/array/block/structured_data/xarray_dataset/lon?block={index_0},{index_1}",
"full": "http://localhost:8000/api/v1/array/full/structured_data/xarray_dataset/lon",
"self": "http://localhost:8000/api/v1/node/metadata/structured_data/xarray_dataset/lon"
"self": "http://localhost:8000/api/v1/metadata/structured_data/xarray_dataset/lon"
},
"meta": null
},
Expand Down Expand Up @@ -453,7 +453,7 @@ In certain cases, it is efficient to in-line all the information about the node'
"links": {
"block": "http://localhost:8000/api/v1/array/block/structured_data/xarray_dataset/precipitation?block={index_0},{index_1},{index_2}",
"full": "http://localhost:8000/api/v1/array/full/structured_data/xarray_dataset/precipitation",
"self": "http://localhost:8000/api/v1/node/metadata/structured_data/xarray_dataset/precipitation"
"self": "http://localhost:8000/api/v1/metadata/structured_data/xarray_dataset/precipitation"
},
"meta": null
},
Expand Down Expand Up @@ -505,7 +505,7 @@ In certain cases, it is efficient to in-line all the information about the node'
"links": {
"block": "http://localhost:8000/api/v1/array/block/structured_data/xarray_dataset/temperature?block={index_0},{index_1},{index_2}",
"full": "http://localhost:8000/api/v1/array/full/structured_data/xarray_dataset/temperature",
"self": "http://localhost:8000/api/v1/node/metadata/structured_data/xarray_dataset/temperature"
"self": "http://localhost:8000/api/v1/metadata/structured_data/xarray_dataset/temperature"
},
"meta": null
},
Expand Down Expand Up @@ -547,7 +547,7 @@ In certain cases, it is efficient to in-line all the information about the node'
"links": {
"block": "http://localhost:8000/api/v1/array/block/structured_data/xarray_dataset/time?block={index_0}",
"full": "http://localhost:8000/api/v1/array/full/structured_data/xarray_dataset/time",
"self": "http://localhost:8000/api/v1/node/metadata/structured_data/xarray_dataset/time"
"self": "http://localhost:8000/api/v1/metadata/structured_data/xarray_dataset/time"
},
"meta": null
}
Expand Down
6 changes: 3 additions & 3 deletions docs/source/how-to/api-keys.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ We can use in other web clients as well. For example, using [HTTPie](https://htt
we can see that unauthenticated requests are refused

```
$ http http://localhost:8000/api/v1/node/metadata/
$ http http://localhost:8000/api/v1/metadata/
HTTP/1.1 401 Unauthorized
content-length: 30
content-type: application/json
Expand All @@ -117,7 +117,7 @@ but passing the API key in the `Authorization` header as `Apikey YOUR_KEY_HERE`
(Note the use of `'` quotes.)

```
$ http http://localhost:8000/api/v1/node/metadata/ 'Authorization:Apikey 48e8f8598940fa0f3e80b406def606e17e815a2c76fe21350a99d6d9935371d11533b318'
$ http http://localhost:8000/api/v1/metadata/ 'Authorization:Apikey 48e8f8598940fa0f3e80b406def606e17e815a2c76fe21350a99d6d9935371d11533b318'
HTTP/1.1 200 OK
content-length: 320
content-type: application/json
Expand All @@ -130,7 +130,7 @@ set-cookie: tiled_csrf=InE4mplUO0goPxf4V07tVuLSLUvDqhgtALTHYoC3T3s; HttpOnly; Pa
```

The API key can also be passed in the URL like
`http://localhost:8000/api/v1/node/metadata/?api_key=YOUR_KEY_HERE`. Using the
`http://localhost:8000/api/v1/metadata/?api_key=YOUR_KEY_HERE`. Using the
`Authorization` header is preferred (more secure) but in some situations, as in
pasting a link into a web browser, the URL is the only option.

Expand Down
2 changes: 1 addition & 1 deletion docs/source/how-to/client-logger.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Requests (`->`) and responses (`<-`) will now be logged to the console, like so.
16:49:22.486 <- 200 server:nginx/1.18.0 (Ubuntu) date:Tue, 01 Feb 2022 21:49:22 GMT content-type:application/json content-length:761 connection:keep-alive etag:35b70c6412c39db8b7b5132ddf61973c expires:Tue, 01 Feb 2022 21:59:22 GMT content-encoding:gzip vary:Accept-Encoding server-timing:tok;dur=0.1, pack;dur=0.0, compress;dur=0.1;ratio=3.1, app;dur=3.9 set-cookie:tiled_csrf=-fyaLez0YkradgcEVYBJh4QotR5MNyzouV0SV0NWHmM; HttpOnly; Path=/; SameSite=lax
16:49:22.492 -> GET 'https://tiled-demo.blueskyproject.io/?root_path=true' 'host:tiled-demo.blueskyproject.io' 'accept:application/x-msgpack' 'accept-encoding:gzip,blosc' 'connection:keep-alive' 'user-agent:python-tiled/0.1.0a49' 'cookie:tiled_csrf=-fyaLez0YkradgcEVYBJh4QotR5MNyzouV0SV0NWHmM'
16:49:22.531 <- 200 server:nginx/1.18.0 (Ubuntu) date:Tue, 01 Feb 2022 21:49:22 GMT content-type:application/x-msgpack content-length:773 connection:keep-alive etag:35b70c6412c39db8b7b5132ddf61973c expires:Tue, 01 Feb 2022 21:59:22 GMT content-encoding:gzip vary:Accept-Encoding server-timing:tok;dur=0.1, pack;dur=0.0, compress;dur=0.1;ratio=2.7, app;dur=4.5
16:49:22.535 -> GET 'https://tiled-demo.blueskyproject.io/node/metadata/' 'host:tiled-demo.blueskyproject.io' 'accept:application/x-msgpack' 'accept-encoding:gzip,blosc' 'connection:keep-alive' 'user-agent:python-tiled/0.1.0a49' 'cookie:tiled_csrf=-fyaLez0YkradgcEVYBJh4QotR5MNyzouV0SV0NWHmM'
16:49:22.535 -> GET 'https://tiled-demo.blueskyproject.io//metadata/' 'host:tiled-demo.blueskyproject.io' 'accept:application/x-msgpack' 'accept-encoding:gzip,blosc' 'connection:keep-alive' 'user-agent:python-tiled/0.1.0a49' 'cookie:tiled_csrf=-fyaLez0YkradgcEVYBJh4QotR5MNyzouV0SV0NWHmM'
16:49:22.572 <- 200 server:nginx/1.18.0 (Ubuntu) date:Tue, 01 Feb 2022 21:49:22 GMT content-type:application/x-msgpack content-length:292 connection:keep-alive etag:821dd2a8b431ecd016f94cacd44af74f server-timing:tok;dur=0.0, pack;dur=0.0, app;dur=3.8

>>> t = c['generated']['short_table']
Expand Down
30 changes: 15 additions & 15 deletions docs/source/how-to/custom-clients.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,28 @@
# Custom Python Client Objects

To provide an "upgraded" and more finely-tuned user experience for certain
kinds of dataset, TIled can be configured to use custom Python objects.
kinds of dataset, Tiled can be configured to use custom Python objects.
This is transparent and automatic from the point view of the user.

In the Python client, when a user accesses a given item, Tiled inspects the
item to decide what type of object to use to represent it.
In simple cases, this is just based on the `structure_family`: `"array"` goes
to `tiled.client.array.ArrayClient`; `"dataframe"` goes to
`tiled.client.dataframe.DataFrameClient`; `"node"` goes to
`tiled.clide.node.Node`. Those classes then manage further commication with
Tiled server to access their contents.
`tiled.clide.node.Container`. Those classes then manage further communication
with Tiled server to access their contents.

Each item always has exactly one `structure_family`, and it's always from a
fixed list. In addition, it may have a list of `specs`, labels which are meant
to communicate some more specific expectations about the data that may or may
not have meaning to a given client. If a client does not recognize some spec,
it can still access the metadata and data and performed Tiled's essential
fucntions. If it does recognize a spec, it can provide an upgraded user
functions. If it does recognize a spec, it can provide an upgraded user
experience.

## Example

Supose data labeled with the `xdi` spec is guaranteed to have a metadata
Suppose data labeled with the `xdi` spec is guaranteed to have a metadata
dictionary containing the following two entries:

```py
Expand Down Expand Up @@ -64,7 +64,7 @@ display as in:
<...>
````

It is conventional to include angle brakets `<>` when the string is not valid
It is conventional to include angle brackets `<>` when the string is not valid
Python code, as opposed to

````py
Expand All @@ -85,7 +85,7 @@ data labeled with the `xdi` spec. We'll register it manually for development
and testing. Then we'll see how to configure it seamlessly for the user.
```py
from tiled.client.node import DEFAULT_STRUCTURE_CLIENT_DISPATCH
from tiled.client.container import DEFAULT_STRUCTURE_CLIENT_DISPATCH
from tiled.client import from_uri
custom = dict(DEFAULT_STRUCTURE_CLIENT_DISPATCH["numpy"])
Expand Down Expand Up @@ -149,7 +149,7 @@ recognizes none of the specs, or if there are no specs, it falls back to using t
structure family. Specs should generally be sorted from most specific to least
specific, so that Tiled uses the most finely-tuned client object available.

## More Possiblities and Design Guidelines
## More Possibilities and Design Guidelines

There are many other useful things we could do with a custom client that is purpose-built
for a specific kinds of data and/or metadata. We can add convenience properties
Expand Down Expand Up @@ -179,13 +179,13 @@ with other scientific Python libraries.
_change_ the behavior of the existing methods, attributes, and properties in
the base class. This is a well-known
[principle](https://en.wikipedia.org/wiki/Liskov_substitution_principle) in
software design generally, and it is especialy crucial here. If a user runs code
in a software environment where the library with the custom objects happens
to be missing, we want the user to immediate notice the missing methods, not
get confusingly different results from the "standard" method and the
customized one. In addition, it is helpful if the "vanilla" Tiled documentation
and user knowledge transfers to the custom classes with _additions_ but no
confusing _changes_.
software design generally, and it is especially crucial here. If a user runs
code in a software environment where the library with the custom objects
happens to be missing, we want the user to immediate notice the missing
methods, not get confusingly different results from the "standard" method
and the customized one. In addition, it is helpful if the "vanilla" Tiled
documentation and user knowledge transfers to the custom classes with
_additions_ but no confusing _changes_.
2. If something custom will do I/O (i.e. download metadata or data from the
server) make it method, not a property. Properties that do "surprise" I/O
may block over a slow network and can be very confusing. The same guideline
Expand Down
6 changes: 3 additions & 3 deletions docs/source/reference/authentication.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ The content of `tokens.json` looks like
Make an authenticated request using that access token.

```
$ http GET :8000/api/v1/node/metadata/ "Authorization:Bearer `jq -r .access_token tokens.json`"
$ http GET :8000/api/v1//metadata/ "Authorization:Bearer `jq -r .access_token tokens.json`"
HTTP/1.1 200 OK
content-length: 239
content-type: application/json
Expand All @@ -106,7 +106,7 @@ set-cookie: tiled_csrf=1-Cpa1WcwggakZ91FtNsscjM8VO1N1znmuILlL5hGY8; HttpOnly; Pa
"id": "",
"links": {
"search": "http://localhost:8000/api/v1/node/search/",
"self": "http://localhost:8000/api/v1/node/metadata/"
"self": "http://localhost:8000/api/v1//metadata/"
},
"meta": null,
"type": "tree"
Expand All @@ -121,7 +121,7 @@ When the access token expires (after 15 minutes, by default) requests will be
rejected like this.

```
$ http GET :8000/api/v1/node/metadata/ "Authorization:Bearer `jq -r .access_token tokens.json`"
$ http GET :8000/api/v1//metadata/ "Authorization:Bearer `jq -r .access_token tokens.json`"
HTTP/1.1 401 Unauthorized
content-length: 53
content-type: application/json
Expand Down
2 changes: 1 addition & 1 deletion docs/source/reference/http-api-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The routes are generally spelled like ``GET /{action}/{path}/``, like GitHub
repository URLs, with the path following the structure of the Tree
entries.

The ``GET /node/metadata/{path}`` route provides the metadata about one node.
The ``GET //metadata/{path}`` route provides the metadata about one node.
The ``GET /node/search/{path}`` route provides paginated access to the children of
a given node, with optional filtering (search). The ``GET /node/full/{path}`` route
provides all the metadata and data below a given node.
Expand Down
28 changes: 14 additions & 14 deletions docs/source/reference/python-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,10 @@ as well as:
.. autosummary::
:toctree: generated
tiled.client.node.Node.get
tiled.client.node.Node.keys
tiled.client.node.Node.items
tiled.client.node.Node.values
tiled.client.container.Container.get
tiled.client.container.Container.keys
tiled.client.container.Container.items
tiled.client.container.Container.values
```

The views returned by `.keys()`, `.items()`, and `.values()`
Expand Down Expand Up @@ -63,11 +63,11 @@ Beyond the Mapping interface, Node adds the following attributes
.. autosummary::
:toctree: generated
tiled.client.node.Node.metadata
tiled.client.node.Node.references
tiled.client.node.Node.sorting
tiled.client.node.Node.uri
tiled.client.node.Node.specs
tiled.client.container.Container.metadata
tiled.client.container.Container.references
tiled.client.container.Container.sorting
tiled.client.container.Container.uri
tiled.client.container.Container.specs
```

It adds these methods, which return a new Node instance.
Expand All @@ -76,8 +76,8 @@ It adds these methods, which return a new Node instance.
.. autosummary::
:toctree: generated
tiled.client.node.Node.search
tiled.client.node.Node.sort
tiled.client.container.Container.search
tiled.client.container.Container.sort
```

It adds these methods for downloading and refreshing cached data.
Expand All @@ -86,8 +86,8 @@ It adds these methods for downloading and refreshing cached data.
.. autosummary::
:toctree: generated
tiled.client.node.Node.download
tiled.client.node.Node.refresh
tiled.client.container.Container.download
tiled.client.container.Container.refresh
```

It adds this method, which returns the unique metadata keys,
Expand All @@ -97,7 +97,7 @@ structure_families, and specs of its children along with their counts.
.. autosummary::
:toctree: generated
tiled.client.node.Node.distinct
tiled.client.container.Container.distinct
```

## Structure Clients
Expand Down
9 changes: 6 additions & 3 deletions docs/source/reference/queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,14 @@ Follow the links in the table below for examples specific to each query.
tiled.queries.In
tiled.queries.NotIn
tiled.queries.Regex
tiled.queries.Spec
tiled.queries.Specs
tiled.queries.StructureFamily
tiled.queries.SpecQuery
tiled.queries.SpecsQuery
tiled.queries.StructureFamilyQuery
```

(Some have the word `Query` at the end of their name to avoid confusion with
other objects in the Tiled codebase.)

## Query expressions

The `Key` object can be used to construct queries in a readable way using
Expand Down
10 changes: 5 additions & 5 deletions tiled/_tests/test_catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,15 +45,15 @@ async def test_nested_node_creation(a):
await a.create_node(
key="b",
metadata={},
structure_family=StructureFamily.node,
structure_family=StructureFamily.container,
specs=[],
references=[],
)
b = await a.lookup_adapter(["b"])
await b.create_node(
key="c",
metadata={},
structure_family=StructureFamily.node,
structure_family=StructureFamily.container,
specs=[],
references=[],
)
Expand Down Expand Up @@ -84,7 +84,7 @@ async def test_sorting(a):
await a.create_node(
key=letter,
metadata={"letter": letter, "number": number},
structure_family=StructureFamily.node,
structure_family=StructureFamily.container,
specs=[],
references=[],
)
Expand Down Expand Up @@ -130,7 +130,7 @@ async def test_search(a):
await a.create_node(
key=letter,
metadata={"letter": letter, "number": number, "x": {"y": {"z": letter}}},
structure_family=StructureFamily.node,
structure_family=StructureFamily.container,
specs=[],
references=[],
)
Expand All @@ -152,7 +152,7 @@ async def test_search(a):
await d.create_node(
key=letter,
metadata={"letter": letter, "number": number},
structure_family=StructureFamily.node,
structure_family=StructureFamily.container,
specs=[],
references=[],
)
Expand Down
4 changes: 2 additions & 2 deletions tiled/_tests/test_distinct.py
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ def test_distinct(context):
{"value": ["MyDataFrame"], "count": None},
],
"structure_families": [
{"value": "node", "count": None},
{"value": "container", "count": None},
{"value": "array", "count": None},
{"value": "dataframe", "count": None},
],
Expand All @@ -107,7 +107,7 @@ def test_distinct(context):
{"value": ["MyDataFrame"], "count": 5},
],
"structure_families": [
{"value": "node", "count": 22},
{"value": "container", "count": 22},
{"value": "array", "count": 10},
{"value": "dataframe", "count": 10},
],
Expand Down
Loading

0 comments on commit c062d8a

Please sign in to comment.