Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update stage out (upload data to S3) to read catalog.json #130

Closed
ngachung opened this issue Apr 3, 2023 · 0 comments · Fixed by #138
Closed

Update stage out (upload data to S3) to read catalog.json #130

ngachung opened this issue Apr 3, 2023 · 0 comments · Fixed by #138
Assignees
Labels

Comments

@ngachung
Copy link
Collaborator

ngachung commented Apr 3, 2023

New Upload procedure:

  • It will look for CATALOG_FILE Env variable for Catalog file location whcih is a valid STAC catalog.
  • STAC catalog to have list of granules metadata files in "links"
    Example:
{
    "type": "Catalog",
    "stac_version": "1.0.0",
    "id": "sample-id",
    "description": "Reference: https://github.com/radiantearth/stac-spec/blob/master/examples/catalog.json",
    "links": [
        {
            "href": "/absolute/path/to/stac/granules/json/file",
            "rel": "child",
            "type": "application/json",
            "title": "<granules-id>"
        },
        {
            "href": "/absolute/path/to/stac/granules/json/file",
            "rel": "child",
            "type": "application/json",
            "title": "<granules-id>"
        }
    ]
}
  • each metadata file is a valid STAC item.
  • read each stac metadata to find necessary files in "assets"
  • push the files to S3 using collection-id.
    • may need to pull the collection from U-DS to see if necessary files are there.
  • update STAC metadata "assets" with S3 URL.
  • upload STAC metadata file.

Need to update

  • in catalog stage: it needs a new provider name.
  • need a metadata parse from stac to CMR for Cumulus
  • new step function workflow with the above provider name to take care of these files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants