Skip to content

Update stage out (upload data to S3) to read catalog.json #130

Closed
@ngachung

Description

@ngachung

New Upload procedure:

  • It will look for CATALOG_FILE Env variable for Catalog file location whcih is a valid STAC catalog.
  • STAC catalog to have list of granules metadata files in "links"
    Example:
{
    "type": "Catalog",
    "stac_version": "1.0.0",
    "id": "sample-id",
    "description": "Reference: https://github.com/radiantearth/stac-spec/blob/master/examples/catalog.json",
    "links": [
        {
            "href": "/absolute/path/to/stac/granules/json/file",
            "rel": "child",
            "type": "application/json",
            "title": "<granules-id>"
        },
        {
            "href": "/absolute/path/to/stac/granules/json/file",
            "rel": "child",
            "type": "application/json",
            "title": "<granules-id>"
        }
    ]
}
  • each metadata file is a valid STAC item.
  • read each stac metadata to find necessary files in "assets"
  • push the files to S3 using collection-id.
    • may need to pull the collection from U-DS to see if necessary files are there.
  • update STAC metadata "assets" with S3 URL.
  • upload STAC metadata file.

Need to update

  • in catalog stage: it needs a new provider name.
  • need a metadata parse from stac to CMR for Cumulus
  • new step function workflow with the above provider name to take care of these files.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions