[Bug]: Granule DAPA API returns granule with running and failed status

[ngachung]

DAPA granules endpoint should not return "incomplete" items i.e. items where status != completed.

ASIPS team discovered an issue when searching for items

unity_sds_client.resources.Collection(collection_id=urn:nasa:unity:asips:int:P1590000-6T___1)
Traceback (most recent call last):
  File "/home/steved/code/mdps-tools/mdps_tools/unity/broken.py", line 14, in <module>
    objs = ds.get_collection_data(c, limit=10)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/steved/mambaforge/envs/mdps/lib/python3.12/site-packages/unity_sds_client/services/data_service.py", line 78, in get_collection_data
    ds = Dataset(dataset['id'], collection.collection_id, dataset['properties']['start_datetime'], dataset['properties']['end_datetime'], dataset['properties']['created'], properties=dataset['properties'])
                                                          ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
KeyError: 'start_datetime'

start_datetime doesn't exist for granules in running status. Why the granules are set to running status even though the step function completed successful is a separate issue.

I thought this was fixed with #92 but it seems to have resurfaced?

UDS version 9.4.1.

[wphyojpl]

It's possible that this is resurfaced as we moved away from cumulus DB, and store the entire granules in our own DB (ES).

But only "completed" granules should be triggered to store in DB via "cumulus-report-granules-topic" .

[wphyojpl]

I have checked further. When an incoming event doesn't have start_datetime, there are no files as well, and granules indexer would reject that event.

There is a chance that if start_datetime is missing in cumulus event, it would also be missing in the indexer, but they are usually present for "running" events.

We can choose to also ignore "running" events