Support OGC API - Features

[jjspace]

The Spec

This issue is to track adding full support for the OGC API - Features spec as well as any discussion around the process.

The spec currently includes 5 parts with parts 4 and 5 currently under draft. The initial plan will be to support Part 1: Core with others as follow up tasks. I currently do not expect us to support Part 2 which adds CRSs which Cesium already does not do much work with.

• OGC API - Features - Part 1: Core corrigendum, version 1.0.1 standard
  • Latest Editor's Draft of next version of OGC API - Features - Part 1: Core
  • ISO 19168-1:2020 Geographic information - Geospatial API for features - Part 1: Core standard
• OGC API - Features - Part 2: Coordinate Reference Systems by Reference corrigendum, version 1.0.1 standard
  • Latest Editor's Draft of next version of OGC API - Features - Part 2: CRS by Reference
• OGC API - Features - Part 3: Filtering, version 1.0.0
• OGC API - Features - Part 4: Create, Replace, Update and Delete draft
• OGC API - Features - Part 5: Schemas draft

Format/Encoding

The spec does not mandate any specific encoding or format but strongly favors HTML and GeoJSON with additional suggestions for GML. We will focus the initial implementation only around GeoJSON (especially since we already support it) with other formats being optional secondary efforts in the future as requested.

Side note: the API does not dictate a specific way to designate what format to serve. I believe most servers will implement this with the content type in headers but another suggestion from the Spec is to use a f=[format] query parameter which I've observed utilized in other libraries and servers so my current assumption is this is one of the more generally accepted methods.

Other implementations

Just listing off some other implementations I've found. Mainly from this list

• ogcapi-js - lightweight and focused on the Features API
  • Works well but I don't see any way to control authentication beyond adjusting query params for every request. May not work with header auth?
• ogc-client - supports multiple OGC specs
  • Similar auth concerns. Allows setting global fetch options for the whole client which could make it hard to use multiple sources
  • The API seems more focused on exploring a service completely and may not fit our needs and use case as closely
  • Only works in browser as it relies on window internally

Example data

Just an assortment of data sources from servers that we know support the OGC Features API and we can use for testing during implementation

• National Land Survey of Finland seems to have a bunch of data
  • Free but needs an API key - Instructions for using the API key
  • Geographic names (OGC API Features)
  • Technical description – Topographic Database (OGC API Features)
• https://demo.ldproxy.net/ Demos using ldproxy but has a handful of different data that we can view
  • Data has a mix of areas, lines and points which is useful
  • Supports an HTML representation so it's really easy to explore the data without another viewer. Just add ?f=json to get the GeoJSON representation
  • Some of these are very large datasets, for example this one of road network nodes has over 10,000 points which could be useful for stress testing or any loading optimizations
• ArcGIS also seems to have OGC Features as an option but I have not explored it much yet
  • We may be able to use the API key we already use
  • Work with OGC Features services—ArcGIS Server | Documentation for ArcGIS Enterprise
  • Tutorial: Style features | CesiumJS | Esri Developer
• Please mention more if you know of good ones!

Dataset sizes/Number of features

These are the sources + collection ids of the links listed above. As you can see we have access to quite a few different sizes of data, especially large datasets, which will be handy for testing.

strassen
      33,754   nullpunkte - Zero points
      31,591   abschnitteaeste - Sections and Branches
      31,464   unfaelle - Accidents
      31,464   unfaelle2 - Accidents (alternative representation)
      11,256   netzknoten - Network nodes
vinyard
       1,585   vineyards - Vineyards
finland-topo (Topographic Database)
  17,246,159   virtavesikapea - Stream narrow
   9,993,979   maastokuvionreuna - Terrain pattern edge
   6,866,135   korkeuskayra - Elevation contour
   5,727,082   rakennusreunaviiva - Building edge line
   5,622,883   rakennus - Building
   3,730,942   tieviiva - Road line
   2,087,881   metsamaankasvillisuus - Forest land vegetation
   1,553,229   kalliosymboli - Rock symbol
   1,550,716   harvalouhikko - Sparse quarry
   1,411,551   suo - Swamp
   1,289,728   kallioalue - Rock area
   1,172,353   paikannimi - Place name
     629,270   soistuma - Swamp
     546,290   vesikivi - Water stone
     505,938   kivi - Stone
     493,539   maasto2kuvionreuna - Terrain pattern 2 edge
     425,224   maatalousmaa - Agricultural land
     212,147   korkeuskayrankorkeusarvo - Elevation contour height value
     209,310   jyrkanne - Cliff
     207,358   muuavoinalue - Other open area
     205,303   viettoviiva - Slope line
     196,532   virtausnuoli - Flow arrow
     167,285   kivikko - Gravel
     159,502   sahkolinja - Power line
     151,233   jarvi - Lake
     149,238   selite - Explanation
     140,752   syvyyskayra - Lake Depth contour
     128,037   vesikuoppa - Water pit
     127,920   syvyyspiste - Depth point
     119,244   vedenpinnankorkeusluku - Water surface elevation value
     109,804   sahkolinjansymboli - Power line symbol
     108,676   karttasymboli - Map symbol
      93,546   luiska - Ramp
      89,391   suurjannitelinjanpylvas - High voltage line pole
      85,670   kaislikko - Reed bed
      83,484   niitty - Meadow
      68,883   muuntaja - Transformer
      64,184   syvyyskayransyvyysarvo - Depth crossing depth value
      57,753   allas - Basin
      53,580   rakennelma - Structure
      46,921   tiesymboli - Road symbol
      45,170   suojelualueenreunaviiva - Protected area boundary
      36,616   lahde - Spring (water source)
      35,196   tienroteksti - Road text
      33,145   vesikivikko - Water gravel
      30,645   tervahauta - Tar pit
      28,772   osoitepiste - Address point
      27,053   aita - Fence
      21,883   luonnonsuojelualue - Nature conservation area
      20,576   virtavesialue - Stream area
      19,095   rautatie - Railway
      18,942   matalikko - Shoal
      15,855   urheilujavirkistysalue - Sports and recreation area
      15,742   koski - Rapids
      13,706   maaaineksenottoalue - Soil extraction area
      13,565   hietikko - Sand
      12,428   rauhoitettukohde - Protected area
      12,188   pistolaituriviiva - Dockline
      11,986   autoliikennealue - Vehicle traffic area
      11,776   maatuvavesialue - Groundwater area
      10,479   vesikulkuvayla - Waterway
       8,800   portti - Gate
       8,446   masto - Mast
       7,991   varastoalue - Storage area
       7,583   mastonkorkeus - Mast height
       5,771   taajaanrakennetunalueenreuna - Edge of the densely built area
       5,680   turvalaite - Safety device
       5,423   taajaanrakennettualue - Densely built area
       4,857   suojanne - Shelter
       3,917   muuntoasema - Transformer station
       3,595   puisto - Park
       3,508   kansallispuisto - National park
       3,413   puurivi - Tree line
       3,299   rautatiensymboli - Railway symbol
       2,875   muistomerkki - Monument
       2,389   ilmaradankannatinpylvas - Aerial railway support pole
       2,265   uittolaite - Rafting device
       2,228   vesikulkuvaylanteksti - Waterway text
       1,933   tulentekopaikka - Campfire site
       1,885   ilmarata - Aerial railway
       1,867   tuulivoimala - Wind turbine
       1,845   tulvaalue - Flood area
       1,791   puu - Tree
       1,790   pato - Dam
       1,558   valtakunnanrajapyykki - National boundary marker
       1,549   louhos - Quarry
       1,534   taytemaa - Fill land
       1,380   hautausmaa - Cemetery
       1,304   savupiippu - Chimney
       1,254   suojametsanreunaviiva - Protective forest boundary
       1,242   lahestymisvalo - Approach light
       1,202   kunnanhallintoraja - Municipal boundary
       1,190   nakotorni - Watchtower
       1,164   meri - Sea
         828   lentokenttaalue - Airport area
         807   metsamaanmuokkaus - Forest land cultivation
         659   vesiasteikko - Water scale
         639   aallonmurtaja - Breakwater
         632   rajavyohykkeentakaraja - Border zone back boundary
         559   rautatieliikennepaikka - Railway traffic area
         493   savupiipunkorkeus - Chimney height
         465   vesikulkuvaylankulkusuunta - Waterway direction
         378   kellotapuli - Bell tower
         361   vesitorni - Water tower
         339   ampumaalue - Shooting range
         309   kunnanhallintokeskus - Municipal administration center
         308   kunta - Municipality
         258   metsanraja - Forest boundary
         206   ankkuripaikka - Anchorage
         195   merkittavaluontokohde - Notable natural site
         190   sisaistenaluevesienulkoraja - Internal water boundary
         137   sulkuportti - Lock gate
         101   ulkojasisasaaristonraja - Outer archipelago boundary
          95   satamaalue - Port area
          82   kaatopaikka - Landfill
          76   hylky - Wreck
          69   aluemerenulkoraja - Area sea boundary
          40   kalliohalkeama - Rock fracture
          36   uittoranni - Rafting shore
          20   luonnonpuisto - Nature reserve
          19   suojaalueenreunaviiva - Protection area boundary
          18   suojaalue - Protection area
          14   hylynsyvyys - Wreck depth
          13   retkeilyalue - Recreation area
           3   aidansymboli - Fence symbol
           0   pelastuskoodipiste - Rescue code point
           0   suojametsa - Protective forest
           0   tunnelinaukko - Tunnel entrance
daraa
       3,694   TransportationGroundCrv - Transportation - Ground (Curves)
         575   UtilityInfrastructurePnt - Utility Infrastructure (Points)
         101   AgricultureSrf - Agricultural (Surfaces)
          82   StructureSrf - Structure (Surfaces)
          73   InformationPnt - Information (Points)
          54   TransportationGroundPnt - Transportation - Ground (Points)
          52   SettlementSrf - Settlement (Surfaces)
          48   SettlementPnt - Settlement (Points)
          38   StructurePnt - Structure (Points)
          33   o2s_l - Other (Curves)
          29   MilitarySrf - Military (Surfaces)
          27   StructureCrv - Structure (Curves)
          26   HydrographyCrv - Hydrography (Curves)
          20   AeronauticCrv - Aeronautic (Curves)
          20   HydrographySrf - Hydrography (Surfaces)
          17   o2s_p - Other (Points)
          14   CultureSrf - Cultural (Surfaces)
          11   UtilityInfrastructureCrv - Utility Infrastructure (Curves)
           9   FacilitySrf - Facility (Surfaces)
           8   AeronauticSrf - Aeronautic (Surfaces)
           8   VegetationSrf - Vegetation (Surfaces)
           6   RecreationSrf - Recreation (Surfaces)
           5   CulturePnt - Cultural (Points)
           5   TransportationGroundSrf - Transportation - Ground (Surfaces)
           4   TransportationWaterCrv - Transportation - Water (Curves)
           3   FacilityPnt - Facility (Points)
           3   RecreationPnt - Recreation (Points)
           3   o2s_a - Other (Surfaces)
           2   HydrographyPnt - Hydrography (Points)
           1   AgriculturePnt - Agricultural (Points)
           1   IndustrySrf - Industry (Surfaces)
           1   PhysiographyCrv - Physiography (Curves)
           1   RecreationCrv - Recreation (Curves)
finland-places (Geographic names)
   1,401,034   mapnames - MapName
     817,962   placenames - PlaceName
     817,962   placenames_simple - PlaceNameSimple
     804,789   places - Place

[jjspace]

I've created an example sandcastle to explore loading different "providers" for OGC Features APIs and visualizing them. Currently it uses the finland and ldproxy data above. This lets us easily explore the data itself but also start to get a feel for the process of managing said data with our underling GeoJsonDataSource objects. As I start implementing our new Provider class/classes I can swap out parts of this sandcastle and still see what it's like to view the data.

I've started working on this in the ogc-features branch which currently has that sandcastle in the developer gallery

Some initial thoughts and observations from this process that we should discuss:

1. GeoJSON doesn't de-duplicate data
   1. With the "pagination" and bbox and other filtering options of the Features API it's very easy/likely to load the same features multiple times. Ideally we should not create duplicated entities but I'm not sure what the best approach to that is
   2. The Features API dictates that every feature should have a unique id even if the GeoJSON spec itself doesn't, can we use this?

   The Core requirements class only requires that the feature URI is unique. Implementations MAY apply stricter rules and, for example, use unique id values per dataset or collection.

   3. In the examples I've seen thus far all features do include that id property
2. How can we access the extra properties of responses like numMatched?

   Each page may include information about the number of selected and returned features (numberMatched and numberReturned) as well as links to support paging (link relation next).

   1. I've also seen some of the samples return an itemCount property on the top level /collections and /collections/collectionId} routes. This is not defined in the spec but could be useful if it's fairly standardized in actual implementations.
   2. When loading data to GeoJsonDataSource we generally provide the url not the json itself and when it's parsed it ignores these extra values. Is there an easy way to expose these instead? If we load the json and pass it to the data source instead we could grab them but I have concerns over memory at that point.
3. How do we handle paging?
   1. How much does our provider control internally vs the consumer?
   2. Should we always try loading all data and only do paging/bbox opti internally?
   3. Or should it be exposed to the user and they control requesting more pages?
4. How should we handle the data source? All parsed into a single GeoJsonDataSource? or multiple?
   1. If multiple, how should we handle them? Also think this would complicate the de-duplication effort
5. Query params and Headers may be a little tricky or a pain to preserve and pass through to every request
   1. iTwin will need an auth header
   2. The Finland data needs an api-key query param (and breaks with the f=[format] query param)
   3. Probably not tooo much different than what we do for 3DTiles but needs to be thought of. This seemed to be one of the the main pitfalls of the libraries I tried.

CC @ggetz any thoughts?

[jjspace]

Bumped into an old issue with the Vineyard sample data that I opened a fix for in #12460

The bigger complication/annoyance I discovered today is that the spec does not provide a defined requirement for how to do paging other than the limit param. The ldproxy pages use a combination of offset and limit to do paging so I assumed offset was standard but the Finland data does not support offset and will actually 400 if it's in the request. The Finland data instead seems to rely on next links with an obscure next code

The only true requirements for the /items route we can rely on to exist are limit, bbox, and datetime. The response from the /items route SHALL include a links property or header but it's only required to include the self and alternate links. It SHOULD include a next link but isn't required.

This will all make paging or requesting more data complicated... Probably need to discuss further

[ggetz]

How should we handle the data source? All parsed into a single GeoJsonDataSource? or multiple? If multiple, how should we handle them? Also think this would complicate the de-duplication effort.

Geospatial users are often used to dealing with the concept of layers. Given that, I think it's make sense that a "layer" in the OGC Features API vocab should map to some kind of "layer" concept in the API.

• Under the hood we may want to be smart about how we batch things together, but the user may be looking to handle an object similar to an ImageryLayer.
• It'd be helpful to survey what other libraries do, like Leaflet GeoJSON / OGC API Leaflet Plugin

Do we want to expose a GeoJsonDataSource to the user, or treat it as an internal implementation detail?

• For now, using a GeoJsonDataSource is certainly the most direct approach. But consider if we want to separate the GeoJSON parsing from a GeoJsonDataSource instance itself.
• We also may want to consider performance implications. The Entity API has a lot of overhead with the property system which may not be needed if we are loading static features. We may may not address that here, but we should give ourselves the flexibility to adjust in the future if needed.

The Features API dictates that every feature should have a unique id even if the GeoJSON spec itself doesn't, can we use this

I think we should be able to rely on this as it's required by the spec. It's probably a requirement because of use cases like this.

How can we access the extra properties of responses like numMatched

Is this something you're thinking of using internally for page management, or to expose via the API? Is there a need for this to live on GeoJsonDataSource at all?

How do we handle paging? bbox queries?

I would think user's would like to load a layer and have it "just work" without having to page manually. There may be cases where a user wants to manually specify some constraints on what pages are loaded or a maximum extent, but by default I think most users would want that abstracted away.

[jjspace]

Some small updates

First I added some size stats to the initial comment for all the collections from Finland and ldproxy that I've been testing. A few notable ones:

• Finland Topography - virtavesikapea - Stream narrow has a whopping 17,246,159 features!
• The strassen set from ldproxy which is roads has multiple collections around 30,000 features which are a mix of lines and points pertaining to very similar data
• The Finland Geo Names collections are all over 800,000 features and I noticed they contain a lot of metadata about each so could be good for testing memory capacity

Second I threw together a first guess at the structure we should follow for implementing the necessary classes. I discussed this with @ggetz some but nothing here is final. Parts of this will probably mimic or be similar to how we handle Imagery Providers vs Imagery Layers.

classDiagram
    class FeatureProvider {
        requestFeatures(bbox, time) Feature[]
        Credit
    }
    class OGCFeatureProvider {
        baseUrl
        collectionId
        maxItems
        limitPerRequest
        requestFeatures()
    }
    class FeatureLayerCollection {
        FeatureLayer[] layers
        moveUp/Down
        add/remove(layer: FeatureLayer)
        add/remove events
        update(frameState)
    }
    class FeatureLayer {
        FeatureProvider provider
        Feature[] features
        PrimitiveCollection primitives
        EntityCollection entities : Temporary
        id
        name
        show
        update(frameState)
        boundingRect
        _updateBoundingRect()
        add(feature: Feature)
        addGeoJson(geoJson)
    }
    class Feature {
        id
        metadata
        FeatureGeometry geometry
        credits
        toEntity() Enable easy editing?
        static fromEntity()
    }
    class FeatureGeometry {
        Rectangle rectangle
    }
    class EntityCollection
    class Entity
    FeatureProvider <|-- OGCFeatureProvider
    FeatureProvider <-- FeatureLayer
    FeatureLayerCollection --> FeatureLayer
    FeatureLayer <-- Feature
    Feature <-- FeatureGeometry
    FeatureLayer <.. EntityCollection: Temp
    EntityCollection <-- Entity

Loading

First version

classDiagram
    class FeatureProvider {
        Feature[] loadData()
    }
    class OGCFeatureProvider {
        baseUrl
        collectionId
        bbox
        maxItems
        limitPerRequest
    }
    class RuntimeProvider
    class Primitive
    class FeatureCollection {
        show
        FeatureProvider provider
        update()
    }
    class GeoJsonDataSource
    class Entity
    class Feature {
        toEntity() Enable easy editing
        static fromEntity()
    }
    FeatureProvider <|-- RuntimeProvider
    FeatureProvider <|-- OGCFeatureProvider
    Primitive <|-- FeatureCollection
    FeatureProvider --> FeatureCollection
    FeatureCollection --> Feature
    FeatureCollection ..> GeoJsonDataSource: Temp
    GeoJsonDataSource --> Entity

Loading

[ggetz]

Thanks for the writeup @jjspace! A few clarifying comments from me on the proposed API diagram:

1. It's not obvious to me what a RuntimeProvider is here. Can you add some details?
2. It'd be helpful for FeatureCollection to have a function which returns a Rectangle defining the bounds, similar to ImageryLayer.getImageryRectangle.
3. Taking a look at the Feature API, it looks like there's also a way to specify temporal bounds as well as spatial ones. It seems to closely align with availability as we use it in the Entity API. How should that work here?
4. We should put some thought into how attribution is managed (i.e., Credit objects), just like terrain, imagery layers, and 3D Tiles.
5. It might make sense for FeatureCollection to have a name or id property. In the case of the Feature API, that would correspond to the Collection ID. Are there any other "top-level" properties which it makes sense for a FeatureCollection to have?
6. What properties does a Feature have? A geometry? Metadata properties? An identifier?
7. What parameters should FeatureProvider.loadData take? And to be super pedantic, I think this should be called requestData to be consistent with the verbiage the rest of the API uses for similar operations.
8. In other libraries which implement similar functionality, the equivalent of a FeatureCollection is usually added to a parent collection. Do we need something similar along the lines of ImageryLayerCollection?
9. How do we manage z-ordering? Both within a FeatureCollection and within a potential parent collection?

[jjspace]

@ggetz Some answers to your questions. I've also updated the diagram above

1. It was added simply as an example that there could be other "Feature Provider" classes that also implement the base class. I do not expect it to be implemented as part of this effort so I've dropped it from the diagram
2. Agreed, this will be added
3. I'm not entirely sure yet, probably at the FeatureLayer level and maybe also at the Feature level? I haven't seen data that uses this yet so I didn't dig into the spec about it. I assume we can probably somewhat closely match the Entities API as you mentioned
4. As I understand it in the Imagery system credits live on the Provider and the Imagery objects themselves. I think this would align with the FeaturesProvider and Features themselves. I haven't looked as close to how the scene pulls out the actual credits values so maybe that won't work. I've added them there for now.
5. Agreed, I would expect most of the classes in this system to have some sort of id to actually identify them
6. Updated some, I think the way you had in your poc branch probably does make sense, at least metatdata
7. Yes to renaming to requestData. Not 100% on params yet, probably where the current view is (bbox) and the time if we want to do availability. This may be too much of an OGC Features API focused choice so I'm not sure yet
8. In my initial diagram and mental model I think I was missing a part here and getting too caught up on the "feature collection" as defined by the OGC spec being the same as what we call a collection. Recontextualizing it in my head based on how imagery and primitives vs datasources work I think your previous suggestions to create a FeatureLayer as "a grouping of similar features" makes sense. Then we can create a FeatureLayerCollection and not have it be a FeatureCollectionCollection.
9. This can happen in the FeatureLayerCollection same as ImageryLayerCollection manages it for ImageryLayer