Hierarchically organized STAC APIs

[moradology]

In the most recent release of the STAC API specification (i.e. v1.0.0-beta.5), two features have been added which imply a hierarchical representation of of STAC documents within STAC APIs: children and browseable. A recently added draft on stac_fastapi implements these features with an in memory representation of the hierarchy which defines behavior for both of these features.

In this draft, the hierarchy is specified separately from the ingested contents of the database. For instance, the data which defines the Joplin collection and its various related items is supplied via a json file and an environment variable which tells the server where a valid hierarchy definition can be located.

Assuming this structure is sufficient and desiring a more robust implementation which leverages the performance and benefits for implementers of other STAC libraries which might lean on pgstac, it would be great to push this information and associated behaviors down to postgres.

Below are some TODOs in the stac_fastapi draft which are next to functions/behavior which could likely find its home in pgstac

[moradology]

Function to look up catalog from the new catalogs table: https://github.com/stac-utils/stac-fastapi/pull/336/files#diff-1976a4ae1097a62eeb3fcf5486db05da340c8187aa529bb246488e6b5237d9c5R41-R42

[moradology]

Functions for ingesting hierarchy to table of catalogs: https://github.com/stac-utils/stac-fastapi/pull/336/files#diff-1976a4ae1097a62eeb3fcf5486db05da340c8187aa529bb246488e6b5237d9c5R180

[moradology]

Generate partially completed link json in postgres to avoid pulling all data from collections and catalogs: https://github.com/stac-utils/stac-fastapi/pull/336/files#diff-48c3b9c11c264e04cde80885bcab3d58d0a21f0e3e245e52c51e7893ee969ec7R103

[moradology]

Add optional catalog argument to determine which collections should be included in search: https://github.com/stac-utils/stac-fastapi/pull/336/files#diff-48c3b9c11c264e04cde80885bcab3d58d0a21f0e3e245e52c51e7893ee969ec7R310-R311

[moradology]

Add function to return all collection children (or simply all children?) for a given catalog: https://github.com/stac-utils/stac-fastapi/pull/336/files#diff-48c3b9c11c264e04cde80885bcab3d58d0a21f0e3e245e52c51e7893ee969ec7R375-R376

[cfis]

@bitner @moradology - We would like to have supported hierarchies in pgstac. Would you be interested in a PR that implements this? Something along these lines:

Part 1:

• Add a ltree (https://www.postgresql.org/docs/current/ltree.html) column to the collections table. In my experience that makes querying tree like structures much easier versus than writing recursive queries
• Update APIs to allow insertion of child collections
• Add additional APIs for querying a collection's children

Part 2:
Add ability to store catalogs. Since catalogs/collections can be intermixed, probably the easiest solution is just add a type field to the collections table that can be set to catalog or collection. This would then allow the ability to have mutliple/nested catalogs and collections.

Thoughts?

[bitner]

Also see my response on #206.

I know that there is the Browsable and Children extensions for Stac API, these definitely apply to links. Right now, pgstac does not touch the links, but stac-fastapi-pgstac does mangle the links in order to make sure that they are relative to the url the api is being served from. As described above, there is plenty of work that could be done in stac-fastapi-pgstac. I still have not seen any specs or examples for how any API endpoints should work for nested STAC API's - but this would definitely be a primary issue for the API layer (ie stac-fastapi-pgstac).

Pgstac is primarily set up to provide an implementation of search.

Within stac-fastapi-pgstac, all other endpoints are essentially built by creating a search. For example, the endpoint '/collections/{collection}/items' is just translated to pgstac as this search collection={collection}' and is absolutely no different than the return from this endpoint: /search?collection={collection}`

I am definitely happy to review any PRs that can help address this issue, but this is not something that is currently on my radar with my work at all (obviously, a contract with DevSeed can certainly change that priority). But even then, the first step would be making sure that there are full and complete examples as part of STAC API extensions that describe the full way that endpoints (url patterns) and query parameters would work to accomplish this in an API context. This has definitely been done for static stac endpoints, but there are a lot more conflicts with url patterns when it comes to the API context. The pgstac part of any solution here is probably a fairly easy lift, the API side of things is where things are apt to get tricky and need to be figured out before we even tackle the server side