Explicit STAC Types in JSON [was] Ability to identify STAC file types with tools like file

[schwehr]

Probably not something the average user is going to care about, but in exploring STAC files on my local machine, I find it hard sometimes to know which type of STAC file I'm looking at. Or if I'm trying to find the catalog, collection, or item, it might be hard to know from the command line. e.g. from Earth Engine:

file catalog.json MODIS_[MN]*
catalog.json:                            JSON data
MODIS_MCD43A4_006_BAI.json:              HTML document, ASCII text, with very long lines
MODIS_MCD43A4_006_EVI.json:              HTML document, ASCII text, with very long lines
MODIS_MCD43A4_006_NDSI.json:             HTML document, ASCII text, with very long lines
MODIS_MCD43A4_006_NDVI.json:             ASCII text
MODIS_MCD43A4_006_NDWI.json:             HTML document, ASCII text, with very long lines
MODIS_MOD09GA_006_BAI.json:              HTML document, ASCII text, with very long lines
MODIS_MOD09GA_006_EVI.json:              HTML document, ASCII text, with very long lines
MODIS_MOD09GA_006_NDSI.json:             HTML document, ASCII text, with very long lines
MODIS_MOD09GA_006_NDVI.json:             ASCII text
MODIS_MOD09GA_006_NDWI.json:             HTML document, ASCII text, with very long lines
MODIS_MOD13A1.json:                      HTML document, ASCII text, with very long lines
MODIS_MOD44W_MOD44W_005_2000_02_24.json: JSON data
MODIS_MYD09GA_006_BAI.json:              HTML document, ASCII text, with very long lines
MODIS_MYD09GA_006_EVI.json:              HTML document, ASCII text, with very long lines
MODIS_MYD09GA_006_NDSI.json:             HTML document, ASCII text, with very long lines
MODIS_MYD09GA_006_NDVI.json:             ASCII text
MODIS_MYD09GA_006_NDWI.json:             HTML document, ASCII text, with very long lines
MODIS_MYD13A1.json:                      HTML document, ASCII text, with very long lines
MODIS_NTSG_MOD16A2_105.json:             JSON data

This is a bummer! It would be awesome if file was able to return STAC, the version, and one of {Catalog, Collection, Item}. There might be a better way, but this could possibly work for file:

• Mandate stac_version be 1st
• Mandate stac_type (or some similar name) be 2nd

That runs a bit counter to the idea of the general flexibility of json, which would also be a bummer. If the idea is loosened to just letting readers know the type (like if I can read the json in some other language that doesn't have a stac library), then adding something that gives the type might be useful.

Thoughts?

[schwehr]

@m-mohr Can you give a little more than just a thumbs down? The entire idea? unix file (which isn't the most flexible tool)? The specific suggestion?

If you were to faced with a file of STAC json (possibly unsorted) and didn't have a STAC library, how would you distinguish the types?

Some use cases I'm thinking of: Use sends me some stac files to be added to Earth Engine. Or all stac found by crawling the internet. Or someone puts some stac files and data in a GCS bucket and we want to figure out what it is and what issues there might be.

[m-mohr]

Sorry for the thumbs down without a comment. I started with the reaction and started to comment, but got distracted and forgot it then.

There are a couple of suggestions in the best practices guide, which should make it a bit better. Does your catalog follow them? The only thing that I think could be improved there is to allow collection.json for collections instead of using catalog.json.
If those best practices are not followed, then other best practices are unlikely to be followed, too and I don't think we should enforce any naming scheme.
The second issue is that putting version numbers in the names might not be the best idea as your file names change for every STAC version then. That doesn't seem right and makes upgrading painful.

[schwehr]

Sorry, I will try to clarify. Hopefully this will be more obvious.

I was not thinking of file names at all, but the json contents. e.g. I might just be getting a string blob of json without any file as an RPC, from some database, or who knows where.

I will look more closely at best-practices.md, but those don't seem to be along the lines of what I was intending with this issue. Users are likely going to generate and pass around STAC files. The names of files may be wonky at best. Users are not likely to create a tree as described in Catalog Layout

e.g. with Earth Engine, we have > 500 collections, so naming them collection.json means we suddenly need a tree structure with one file in each directory. Users pass us yaml entries right now that don't have context. When we switch to STAC, they won't be in place for a user contribution. They may sometimes be malformed or copied from other sources without modifying them to fit into our structure.

Does that make more sense? e.g. Since json doesn't allow shebangs (#!) or // magic my original solution was saying what if the json always started as:

{
  "stac_version": "1.0.0-beta.2",
  "stac_type": "Item",

or

{
  "stac_version": "1.0.0-beta.2",
  "stac_type": "Collection",

or

{
  "stac_version": "1.0.0-beta.2",
  "stac_type": "Catalog",

Then I could submit a regex to the unix file magic database that would pick out the values for stac_version and stac_type. Then file foo*.json might give something like:

foo1.json         STAC Catalog, version 1.0.0-beta.2
foo2.json         STAC Collection, version 1.0.0-beta.2
foo3.json         STAC Item, version 1.0.0-beta.2

And it would be trivial for any language with a JSON reader to lookup the stac_type field from the JSON and know what they were dealing with.

[schwehr]

Note that the unix file command won't look very far into a file for the magic strings it is trying to match. If it did, it would get crazy slow.

[cholmes]

Interesting - I didn't know you could give hints like that to unix.

I'm definitely +1 on a recommendation (though probably not a requirement) to put stac_version at the top. stac_type I'd be for also recommending up top if we already had it, but I don't think we have that yet? Curious if there's other use cases for it. I do think it might help with validators, since I feel like staclint I've pointed at an item file and it tries to validate it as a catalog. So if there's a general use for having a stac type I'd be into it.

[m-mohr]

JSON Objects don't have an order, that's coming from the JSON spec. So as long as you don't write JSON files by hand, you can't really guarantee in which order the properties occur in the file or you need a custom JSON writer that somehow sorts by property name. Some writers may do that already, but that's nothing we can really ask for. For example, the Google Earth Engine catalogs (I'm reading 0.6.2) changed the order of the properties in the past quite often. So seems your writer can't guarantee an order.

For the validation, it would be cleaner to have a type field, at the moment it's really a mess with reporting as we need to validate against all schemas and guess what could be most appropriate. We could use type:Feature for Items already and just add type:Catalog and type:Collection or just say: no type=Catalog (and Collection is a catalog). I'm a bit hesistant to put another field (which must be required to be useful for validation und thus is breaking) in the 1.0 spec.

Users are not likely to create folders

Strange, most larger catalogs I've seen have folders?!

Conclusion: I don't think we can really do what you are asking for. I'd suggest using a library such as PySTAC.

[cholmes]

I'm a bit hesitant to put another field (which must be required to be useful for validation und thus is breaking) in the 1.0 spec.

Well, this is the beta period, so it's the last possible time we have to do something like this. I agree we should make sure it's really a win, and not consider it'll break things. But it's a pretty easy upgrade path. I like the idea of using type:Feature.

[m-mohr]

So for validation I don't see a big win. I'd say for validation one could rely on the availibity of the type field in items and all other files would be validated against either collection or catalog. Of course it would be a bit more straightforward in validation to have type:Catalog and type:Collection, but I don't feel that it would be worth the breaking change. In the end, a catalog is a collection. What do @lossyrob and @jbants think?

What other advantages would it have?

[lossyrob]

Having a stac_type field would make identification of the object type on deserialization a little less complicated in PySTAC, but not a great deal. Currently it uses this logic to check for properties that are required in one but not others of { Catalog, Collection, Item } in order to identify the object type. However this is a bit brittle in that if someone adds an extension that breaks the "in one and not the others" assumption, this logic will break. Checking a stac_type property would make this logic more resilient. Still, I'm not sure it's worth the additional field on every object.

[m-mohr]

This is outdated with regards to ItemCollections

Maybe we should come up with a best practice how to detect the files. It seems that you are using a different mechanism as I'm using in the Node Validator and there's likely more out there. I feel it should be the same across the board.

Something like:

if type is defined
  if type is 'Feature' and stac_version is defined // stac_version in items is only available since 0.8, check for (stac_version or assets) to support pre-0.8 data
    => Item
  else if type is 'FeatureCollection' and stac_version is defined
    => ItemCollection
  else
    => Invalid (GeoJSON)
else if stac_version is defined // available since STAC 0.6 and I guess we don't support any thing older?
  if extent is defined or license is defined // or makes more sense for validation, and would likely make more sense for PyStac 
    => Collection
  else
    => Catalog
else
  => Invalid (JSON)

Would that also work for PySTAC, @lossyrob ?

[m-mohr]

I've implemented this in the Node Validator, seems to work so far.

[ycespb]

Hi, what if "providers" occur as property ? According to the spec this would mean that the type is a collection and not a catalog ? would be good to allow a provider for catalog also (without having to add extent information which would make it a Catalog object)...

[m-mohr]

Then it's likely that clients may not recognize it as it's undefined for catalogs. As it's the "spatio-temporal" asset catalog, collections expect related extents. Why would you not want to provide them? Catalogs are for grouping and the provider metadata schould be in the Collection or Item.

[ycespb]

Is not a problem. We can provide "extents" even though they may not be very meaningful. It looks to me that most "catalogs" will then actually become "collections", and very few "catalogs" (which are not also "Collections") will remain. Note that in DCAT, https://www.w3.org/TR/vocab-dcat-2/ the equivalent of provider can be attached to either a dcat:Catalog or dcat:Dataset and not only a dcat:Dataset.