Identify collection configuration for DAAC collection name, what files to archive, DAAC SNS, etc.

[ngachung]

Create an API endpoint for user to provide collection archive configuration values
Create schema for storing collection archive configuration

https://github.com/podaac/cloud-notification-message-schema/tree/master/samples

Original Issue: unity-sds/unity-project-management#172

[wphyojpl]

Message formats:
submission format: https://github.com/podaac/cloud-notification-message-schema?tab=readme-ov-file#notification-message-fields

response format: https://github.com/podaac/cloud-notification-message-schema?tab=readme-ov-file#response-message-fields

[wphyojpl]

I'm wondering if we need to create "collections" to DAAC before we can start sending granules..

If so, will DAACs accept our way of creating too many collections, and when do we create the collections in the DAAC?

[ngachung]

The DAAC collection creation is an involved process resulting in a formal agreement between mission and DAACs on what the collection names are. For our purpose, we just care that the data producer has worked that out with the DAACs and they know the DAAC collection identifier.

[wphyojpl]

{
  "collection": "<DAAC collection name that user has created, and provided via configuration>",
  "identifier": "< Granule ID starting with Unity identifiers or simply a UUID? example: URN:NASA:UNITY:UDS_LOCAL_TEST:DEV:UDS_COLLECTION___2404251100:abcd.1234.efgh.test_file05>",
  "submissionTime": "<Lambda will generate this: 2024-05-16T16:04:44.776376>",
  "provider": "<Should be provided by user. But this is not mandatory. example: unity. NO. It will be tenant which is pulled from identifier>",
  "version": "<one of 1.0, 1.1, 1.2, 1.3>",
  "product": {
    "name": "<recommendation is Granule ID: URN:NASA:UNITY:UDS_LOCAL_TEST:DEV:UDS_COLLECTION___2404251100:abcd.1234.efgh.test_file05.. No. Everything after tenant/venue.>",
    "dataVersion": "<This is not mandatory. Should we include? Currently, it is simply the date. example: 2404251100. It has to be the DAAC version. User needs to provide it>",
    "files": [
      {
        "name": "abcd.1234.efgh.test_file05.data.stac.json",
        "type": "data",
        "uri": "s3://unity-dev-cumulus-unity-william-test-1/URN:NASA:UNITY:UDS_LOCAL_TEST:DEV:UDS_COLLECTION___2404251100/URN:NASA:UNITY:UDS_LOCAL_TEST:DEV:UDS_COLLECTION___2404251100:abcd.1234.efgh.test_file05/abcd.1234.efgh.test_file05.data.stac.json",
        "checksumType": "md5",
        "checksum": "<Currently checksum and size of some files are unknown and -1. It needs to be fixed or omitted>",
        "size": -1
      },
      {
        "name": "abcd.1234.efgh.test_file05.nc.cas",
        "type": "metadata",
        "uri": "s3://unity-dev-cumulus-unity-william-test-1/URN:NASA:UNITY:UDS_LOCAL_TEST:DEV:UDS_COLLECTION___2404251100/URN:NASA:UNITY:UDS_LOCAL_TEST:DEV:UDS_COLLECTION___2404251100:abcd.1234.efgh.test_file05/abcd.1234.efgh.test_file05.nc.cas",
        "checksumType": "md5",
        "checksum": "unknown",
        "size": -1
      },
      {
        "name": "abcd.1234.efgh.test_file05.nc.stac.json",
        "type": "metadata",
        "uri": "s3://unity-dev-cumulus-unity-william-test-1/URN:NASA:UNITY:UDS_LOCAL_TEST:DEV:UDS_COLLECTION___2404251100/URN:NASA:UNITY:UDS_LOCAL_TEST:DEV:UDS_COLLECTION___2404251100:abcd.1234.efgh.test_file05/abcd.1234.efgh.test_file05.nc.stac.json",
        "checksumType": "md5",
        "checksum": "unknown",
        "size": -1
      }
    ]
  }
}

• Should the configuration filter be based on granules name regex + collection name regex?
  • Or ONly based on granules name regex?
• Should there be an option to select which of the files to be stored in DAAC? example: based on file extensions like .nc, .h5, .xml, but not .stac.json
• Will there be different SNSes depending on the configuration?

[wphyojpl]

https://www.figma.com/board/LlVM66Q9qV0UQn7JQ6NHuw/DAAC-Archiver-Workflow?t=YRZ4LkzMCEHC1xMv-0

[wphyojpl]

Are we going to listen to all messages from SNS?
Should we filter the messages that are not unity? How to create a filter?
How should the filter be? is it based on granule id which starts with Unity identifier?

[wphyojpl]

• daac config endpoint should be /data/<collect-id>/daac
• User can optionally add "type" + "file extension" to choose selective files to be archived to DAAC.
• There is only 1 SNS at this moment, but it will be changed at run time.
• Needs to cehck if sqs from shared services account can connect to arbitrary SNS?

[wphyojpl]

• Use Percolator for future extension. Currently, only "match" collection ID.
• Shared Service has SNS which is shared via settings page.
• DAAC's lambda will send message to UDS SNS, and UDS will submit to DAAC's SNS.

[wphyojpl]

How strict should the control be with users CRUD-ing the daac configs?