Stac

[abarciauskas-bgse]

Would appreciate users' feedback on this page only: https://docs.maap-project.org/en/stac/searching.html (docs/source/searching.rst). Reviewing the published (HTML) page and provided feedback as comments on this draft PR is preferrable.

@nmt28 who else can / should we ask to review this PR? I may also ask the ESA team.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[freitagb]

@abarciauskas-bgse - comments/suggestions below:

1. cmr.earthdata.nasa.gov url is not hyperlinked in second paragraph of MAAP's CMR section
2. We should consider a support-like email address and/or form for "contacting the data team"
3. Update the text "You will probably use the collection title or ID to search." to "You should use the collection title or ID to search (examples below)." It removes some ambiguity and points to the examples if they're like me and stop reading when they don't know what something is :-D
4. Should the search/*.ipynb's be linked (e.g. search/searching_the_stac_catalog.ipynb). Could maybe have the text .ipynb and when clicked on it opens the correct hyperlink?
5. Check Data access via STAC section for typos

[nmt28]

Some minor headings under which example is which will be useful for those not so familiar with the different catalogs. Otherwise it looks clear to me.

Not sure who we should include on the ESA side. But it's worth tagging Alexey for the GIs side of things

[abarciauskas-bgse]

I have made the updates suggested by @nmt28 and @freitagb and also updated all the pages in the search directory, however we need to address https://github.com/NASA-IMPACT/active-maap-sprint/issues/423#issuecomment-1492713403 before we can release this documentation, as it relies on pystac_client.

[wildintellect]

This is still true. Should it be described differently?

[abarciauskas-bgse]

I was thinking the same thing but I don't think we should keep documentation just because it's still true, and I don't think this serves a purpose anymore if we're deprecating MAAP CMR altogether. It could make sense to create a list of all the collections we are no longer maintaining with the change, but I thought that would be work with very little extra benefit than the spreadsheet we have.

[wildintellect]

I'm working on rebasing this to develop after the re-org

~/maap-documentation# git merge develop
Removing docs/source/visualization/srtm-stac-mosaic.ipynb
CONFLICT (modify/delete): docs/source/tutorials.rst deleted in develop and modified in HEAD. Version HEAD of docs/source/tutorials.rst left in tree.
Auto-merging docs/source/technical_tutorials/search/searching_edsc_gui.ipynb
CONFLICT (content): Merge conflict in docs/source/technical_tutorials/search/searching_edsc_gui.ipynb
CONFLICT (rename/delete): docs/source/search/deprecated_collection_table.ipynb deleted in HEAD and renamed to docs/source/technical_tutorials/search/deprecated_collection_table.ipynb in develop. Version develop of docs/source/technical_tutorials/search/deprecated_collection_table.ipynb left in tree.
CONFLICT (rename/delete): docs/source/search/cmr_collection_table.ipynb deleted in HEAD and renamed to docs/source/technical_tutorials/search/cmr_collection_table.ipynb in develop. Version develop of docs/source/technical_tutorials/search/cmr_collection_table.ipynb left in tree.
CONFLICT (file location): docs/source/tutorials/AfriSAR_AGB_1681/AfriSAR_AGB_Maps_1681.ipynb added in HEAD inside a directory that was renamed in develop, suggesting it should perhaps be moved to docs/source/science/AfriSAR_AGB_1681/AfriSAR_AGB_Maps_1681.ipynb.

[wildintellect]

@abarciauskas-bgse I updated the branch by merging in develop. Please verify. Also asked @rtapella to give this a review to make sure all the updates for the new structure are in place.

[abarciauskas-bgse]

I'm getting a new error when trying to use pystac client

from pystac_client import Client
URL = 'https://stac.maap-project.org/'
cat = Client.open(URL)
for collection in cat.get_all_collections():
    print(collection)

output:

--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) /tmp/ipykernel_4605/3207962672.py in 1 URL = 'https://stac.maap-project.org//' 2 cat = Client.open(URL) ----> 3 for collection in cat.get_all_collections(): 4 print(collection)

/opt/conda/lib/python3.7/site-packages/pystac/catalog.py in get_all_collections(self)
366 """Get all collections from this catalog and all subcatalogs. Will traverse
367 any subcatalogs recursively."""
--> 368 yield from self.get_collections()
369 for child in self.get_children():
370 yield from child.get_collections()

/opt/conda/lib/python3.7/site-packages/pystac_client/client.py in get_collections(self)
261 for col in page["collections"]:
262 collection = CollectionClient.from_dict(
--> 263 col, root=self, modifier=self.modifier
264 )
265 call_modifier(self.modifier, collection)

/opt/conda/lib/python3.7/site-packages/pystac_client/collection_client.py in from_dict(cls, d, href, root, migrate, preserve_dict, modifier)
66 modifier: Optional[Callable[[Modifiable], None]] = None,
67 ) -> "CollectionClient":
---> 68 result = super().from_dict(d, href, root, migrate, preserve_dict)
69 # error: Cannot assign to a method [assignment]
70 # python/mypy#2427

/opt/conda/lib/python3.7/site-packages/pystac/collection.py in from_dict(cls, d, href, root, migrate, preserve_dict)
641
642 assets: Optional[Dict[str, Any]] = {
--> 643 k: Asset.from_dict(v) for k, v in d.get("assets", {}).items()
644 }
645 links = d.pop("links")

AttributeError: 'NoneType' object has no attribute 'items'

so I'll have to fix that before merging this to develop. I first assumed it's because the AfriSAR_UAVSAR_KZ collection is missing assets but none of the collections have assets listed. This is using pystac-client 0.5.1 (but I had the same issue with 0.6 and 0.4)

Any ideas @jjfrench @emileten @wildintellect @sbquinlan ?

[emileten]

@abarciauskas-bgse the following collections are opened by pystac without any problem :

<CollectionClient id=AfriSAR_UAVSAR_Coreg_SLC>
<CollectionClient id=BIOSAR1>
<CollectionClient id=AfriSAR_AGB_Maps_1681>
<CollectionClient id=Landsat8_SurfaceReflectance>
<CollectionClient id=ABLVIS1B>
<CollectionClient id=AfriSAR_UAVSAR_Ungeocoded_Covariance>
<CollectionClient id=Global_PALSAR2_PALSAR_FNF>
<CollectionClient id=GEDI02_A>
<CollectionClient id=AFLVIS2>
<CollectionClient id=AFRISAR_DLR>
<CollectionClient id=icesat2-boreal>
<CollectionClient id=AfriSAR_UAVSAR_Normalization_Area>
<CollectionClient id=ESACCI_Biomass_L4_AGB_V3_100m_2010>
<CollectionClient id=ESACCI_Biomass_L4_AGB_V3_100m_2018>
<CollectionClient id=ESACCI_Biomass_L4_AGB_V3_100m_2017>
<CollectionClient id=AfriSAR_UAVSAR_Geocoded_SLC>
<CollectionClient id=Global_PALSAR2_PALSAR_Mosiac>
<CollectionClient id=AFRISAR_DLR2>
<CollectionClient id=AfriSAR_UAVSAR_KZ>

Your error is triggered by the next collection, which is GEDI_CalVal_Field_Data. The bug is due to its item_assets field being empty. This collection, along with a couple collections that come next, have this bug and were inserted very recently. Checking this !

Note : you can reproduce the problem with cat.get_collection('GEDI_CalVal_Field_Data')