Enhance geo*_grid results with geometry information

[nyurik]

The new geotile_grid, as well as the older geohash_grid return bucket keys in an algorithm-specific format. This make adaption slower because it requires each client to have a copy of the key decoding algorithm in order to figure out the extent of each bucket.

I would like to propose for all geo grids to include an additional value for each bucket - the geometry of the bucket. For example, if the bucket is a rectangular shape (geotile_grid, geohash_grid, pluscode_grid, etc), the response would look like this:

{
  "key": "0/0/0", // geotile_grid key representing the whole world
  "geo": [[-180,-90],[-180,90],[180,90],[180,-90]],  // GeoJSON-encoded four corners
  "count": 42,
}

For the non-rectangular geo grids like geohex_grid, the "geo" value would look different, e.g. have 6 geo points for the hex cell.

Benefits: simpler client - rapid developer adaption because there is no need to decode complex key into geoshape, client does not have to know the details of the hashing algorithm.
Cons: slightly larger resultset (each bucket will have at least 4 lat/lng pairs)

Optionally, we could make this into an by-request feature - e.g. the user has to request extra geo data with an optional request parameter.

[elasticmachine]

Pinging @elastic/es-analytics-geo

[nreese]

+1

This is a great idea that will make consumption much easier for developers.

[polyfractal]

I think we discussed this idea in #36413, and decided not to implement returning bounding boxes. The main reason is that it's relatively trivial for clients to parse the hash into their favorite geometry using off-the-shelf libraries, and it adds a lot of overhead to the response. E.g. in the above example, it nearly doubles the size of the response (36 bytes up to 88)

[nyurik]

@polyfractal should it be revisited now that we have multiple hashing algorithms? The complexity of some algos may prevent rapid adaption, e.g. when we implement hex grid, it would be far from trivial to find a library in the language of the client (javascript, python, go, ...?) to convert the key into the extent - instead users will have to implement all of the mathematics required for such conversion. Plus we could mitigate the size increase by adding a flag parameter.

[nreese]

it adds a lot of overhead to the response. E.g. in the above example, it nearly doubles the size of the response (36 bytes up to 88)

How about making the geometry info optional and only turn it on with a flag in the request? That will remove the overhead for users that do not need the geometry.

The main reason is that it's relatively trivial for clients to parse the hash into their favorite geometry using off-the-shelf libraries

Even though we think it may be easy to convert the geometry, it is not always easy for developers with different backgrounds, especially developers without geospatial backgrounds. Forcing users to convert the geometry adds another barrier of entry for consuming Elastic APIs. Each barrier of entry makes it more likely that a developer will choose another solution over Elastic's.

[polyfractal]

Plus we could mitigate the size increase by adding a flag parameter.

How about making the geometry info optional and only turn it on with a flag in the request? That will remove the overhead for users that do not need the geometry.

++ if we decide to implement this, I think an opt-in flag would be the best way to handle it. I'll re-add team-discuss and we can talk about it at the next weekly meeting. :)

Historically, we tend to avoid pure "syntactic sugar" features that can be done client-side. I don't think it's too unreasonable to expect someone consuming a low-level geospatial API to understand how to convert the hashes into geometry (e.g. they are already building a mapping or geospatial application if they are using the API...).

In #36901 we added documentation showing how to do a simple conversion in JS, we could do similar for popular languages/libraries if needed.

But yeah, we can chat about this again and see what the full team thinks now that there are more tiles. :)

[talevy]

We discussed this as a team, and concluded that it would be worth exploring a scripting solution first before introducing dedicated solutions for increasing the amount of derivative content returned within aggregations. #39957 explains the approach