Skip to content

Commit

Permalink
Strip out references (bluesky#481)
Browse files Browse the repository at this point in the history
* Drop mention of References in docs.

* Remove references.
  • Loading branch information
danielballan committed Jul 5, 2023
1 parent 2c4aac7 commit 465c66c
Show file tree
Hide file tree
Showing 29 changed files with 77 additions and 331 deletions.
50 changes: 13 additions & 37 deletions docs/source/explanations/metadata.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,37 @@
# Metadata, Specs, and References
# Metadata and "Specs"

## Metadata

Each Tiled node carries an optional dictionary of metadata. This is fully
under the control of the user: Tiled itself is not opinionated about its
content and does not reserve any names for its own use.

```{note}
The metadata dictionary may be arbitrarily nested, and it may contain
anything that Tiled can transmit as JSON or [msgpack](https://msgpack.org).
All JSON-serializable types (dicts, lists, numbers, strings) are supported.
All JSON-serializable types are supported:

- strings
- numbers
- objects (dictionaries)
- arrays (lists)

The following types are additionally supported:

* Dates, represented as `datetime.datetime` objects,
- Dates, represented as `datetime.datetime` objects,
are supported natively by msgpack. JSON does not support dates, so they are
converted to [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) strings
when clients request JSON.
* Numpy objects are tolerated as input, but they are converted to (nested)
- Numpy objects are tolerated as input, but they are converted to (nested)
lists before encoding to msgpack or JSON. Metadata is not a suitable
place to store large arrays.
* `bytes` objects are supported natively by msgpack. If JSON is requested,
- `bytes` objects are supported natively by msgpack. If JSON is requested,
conversion to unicode is attempted.
* `uuid.UUID` objects are converted into the standard hyphen-separated hex
- `uuid.UUID` objects are converted into the standard hyphen-separated hex
representation.
```

## Specs

Every node in Tiled has exactly one "structure family" ("node", "array",
Every node in Tiled has exactly one "structure family" ("container", "array",
"dataframe", etc.) It's useful to think of the structure family as a coarse,
lowest-common-denominator description of how to query and interpret the data.

Expand All @@ -51,31 +55,3 @@ the specs in the list, or if no specs are given, it can fall back to the
structure family to obtain a workable description of the data. Specs are just
an upgrade: "If you know what this means, you can use it to assume additional
constraints."

## References

Each Tiled structure carries an optional List of references, labeling links to
external resources. Their usage is open-ended. They may be used as links to:

* other structures or search results within this Tiled server or other Tiled servers
* data stored in other data storage systems unrelated to Tiled
* documents or publications related to the data

References have a label and a URL. Labels do not need to be unique. Example:

```py
references = [
{"label": "raw_data", "url": "https://..."},
{"label": "raw_data", "url": "https://..."},
{"label': "doi", "url": "https://..."},
]
```

Each item must be a dict where the key is a string, and the value is a valid
URL. The formal definiton of URL is broad: it can include references to web
resources (e.g. `https://...`), local files (`file://localhost/...`), and
more.

In the future, we may expand this to integrate with richer standards for
describing relationships between data, such as RDF, OWL, XLink, and JSON-LD.
If you are interested in this, please open a GitHub issue describing your use case.
1 change: 0 additions & 1 deletion docs/source/reference/python-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ Beyond the Mapping interface, Node adds the following attributes
:toctree: generated
tiled.client.container.Container.metadata
tiled.client.container.Container.references
tiled.client.container.Container.sorting
tiled.client.container.Container.uri
tiled.client.container.Container.specs
Expand Down
1 change: 0 additions & 1 deletion scripts/json_indexes.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ async def test(a):
"nested": {"number": i, "number_as_string": str(i)},
},
specs=[],
references=[],
structure_family="array",
)
# await a.create_metadata_index("nested_number_as_string", "nested.number_as_string")
Expand Down
6 changes: 0 additions & 6 deletions tiled/_tests/test_catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,15 +47,13 @@ async def test_nested_node_creation(a):
metadata={},
structure_family=StructureFamily.container,
specs=[],
references=[],
)
b = await a.lookup_adapter(["b"])
await b.create_node(
key="c",
metadata={},
structure_family=StructureFamily.container,
specs=[],
references=[],
)
c = await b.lookup_adapter(["c"])
assert b.segments == ["b"]
Expand Down Expand Up @@ -86,7 +84,6 @@ async def test_sorting(a):
metadata={"letter": letter, "number": number},
structure_family=StructureFamily.container,
specs=[],
references=[],
)

# Default sorting is _not_ ordered.
Expand Down Expand Up @@ -132,7 +129,6 @@ async def test_search(a):
metadata={"letter": letter, "number": number, "x": {"y": {"z": letter}}},
structure_family=StructureFamily.container,
specs=[],
references=[],
)
assert "c" in await a.keys_range(0, 5)
assert await a.search(Eq("letter", "c")).keys_range(0, 5) == ["c"]
Expand All @@ -154,7 +150,6 @@ async def test_search(a):
metadata={"letter": letter, "number": number},
structure_family=StructureFamily.container,
specs=[],
references=[],
)
assert await d.search(Eq("letter", "c")).keys_range(0, 5) == ["c"]
assert await d.search(Eq("number", 12)).keys_range(0, 5) == ["c"]
Expand All @@ -171,7 +166,6 @@ async def test_metadata_index_is_used(a):
"bool": bool(i),
},
specs=[],
references=[],
structure_family="array",
)
# Check that an index (specifically the 'top_level_metdata' index) is used
Expand Down
25 changes: 25 additions & 0 deletions tiled/_tests/test_specs.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
from ..adapters.mapping import MapAdapter
from ..client import Context, from_context
from ..server.app import build_app
from ..structures.core import Spec


def test_specs():
tree = MapAdapter({}, specs=[Spec("spec_test")])
with Context.from_app(build_app(tree)) as context:
client = from_context(context)
assert client.specs == [Spec("spec_test")]


def test_specs_give_as_str():
tree = MapAdapter({}, specs=["spec_test"])
with Context.from_app(build_app(tree)) as context:
client = from_context(context)
assert client.specs == [Spec("spec_test")]


def test_specs_with_version():
tree = MapAdapter({}, specs=[Spec("spec_test", version="1.1")])
with Context.from_app(build_app(tree)) as context:
client = from_context(context)
assert client.specs == [Spec("spec_test", version="1.1")]
70 changes: 0 additions & 70 deletions tiled/_tests/test_specs_and_references.py

This file was deleted.

2 changes: 1 addition & 1 deletion tiled/_tests/test_validation.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ def lower_case_dict(d):
return out, modified


def validate_foo(metadata, structure_family, structure, spec, references):
def validate_foo(metadata, structure_family, structure, spec):
if structure_family != "dataframe":
raise ValidationError(f"structure family for spec {spec} must be dataframe")

Expand Down
Loading

0 comments on commit 465c66c

Please sign in to comment.