Support incremental indexing

[captaincoordinates]

Incremental indexing can mean:

• repeatedly indexing the same source to identify additions, removals, and changes
• indexing multiple sources (catalogs) and having them contribute to the same aggregate catalog
  • this requires the ability to namespace collections to avoid collection naming collisions between catalogs
• both of the above

Notes

• index files should be versioned according to the indexer version that created them, so that behaviour can accommodate differently-versioned files
• need to understand how to retrieve existing index files if remotely located
  • how to return them updated?
• understand what happens if paging happens over a data change
• new table required to match root catalog URIs to unique identifiers, allows one root URI without an identifier but not two

[captaincoordinates]

Proposed new indexing process notes:

• create a load ID (UUID)
• indexer can optionally be started with a URI for an existing manifest.json file
  • if provided, and the resource is remote (HTTP / S3) pull into a local directory
  • if provided, cannot also provide an index-config path. Index-config will be stored in the manifest and hashed so that it is not modified between incremental loads
• check the manifest's indexer version is compatible with the current indexer version, if not error-out
• create all tables and views in an in-memory DuckDB instance as normal
• populate the index_history (previously audit) table from existing parquet file if it exists / is available locally
• created _previous versions of collections and items tables from existing parquet files if exist / are available locally
• load source data into the new tables
  • hash every item and collection during load
  • add catalog source to all collections
  • if other catalogs were previously loaded and are not loaded this time, copy those catalogs' data in also
  • include a loaded_by reference to load ID in each collection and item
• calculate load stats, i.e. how many collections / items added, removed, updated, unchanged between new and _previous tables
• export new tables to parquet files, include load_id in file names
• update manifest.json to include these new filenames
• manifest.json records how many catalogs exist in the index
  • if more than one, also records a unique prefix for each catalog
  • -or- maybe a catalogs table with prefix column
• if original manifest was remote and writeable (S3)
  • upload new parquet files first, then when complete upload new manifest.json
  • remove all but previous parquet files, based on index_history table

For paging question:

• paging JWT includes the last load ID at the time of query generation. Server errors with HTTP 409 Conflict and forces the user to re-query (this needs to be described in the paging response configurations)

Other:

• add a new /catalogs endpoint to demonstrate how multiple catalogs are supported

Future:

• Support one or more STAC URIs for individual update within the index (e.g. could be triggered by an S3 PUT event). Does not need to be supported in this change, but this change should proceed with the expectation that this change is coming.