New package to normalize spatial data for web plotting.

[bhaskarvk]

This is a bit of a future planning, but here is the main idea.
Currently there is code in the leaflet package that extracts data from sp and sf objects and converts it into a dataframe that is then passed to the Javascript side (by converting it into a JSON).
This code is fairly generic and not really dependent on anything leaflet specific. It makes a lot of sense to take out this code and make it a package of its own. That way we can build other web plotting R packages to wrap say d3.geo or mapboxGL or cesium and reuse a major chunk of the code that takes data from spatial objects and passes it to Javascript.

I have some discussions about this with @jcheng5 and agrees that this is a good idea. There are some questions I have for the r-spatial community.

a) Do you think this is a good idea ?
b) If so then do you think it makes sense for this proposed package to live in r-spatial repo ?
c) If b) is 'yes' what sort of licensing and copyright arrangement we need in place between RStudio and r-spatial ?

cc @tim-salabim @edzer

[tim-salabim]

@bhaskarvk

a) I think this is a great idea
b) I don't see why this is not suited for r-spatial
c) I am no expert on licensing and all that is related, thus I don't really feel qualified to provide a definite answer on this one. I would love to get some more input here.

Two comments/questions from my side (that I can think of right now):

• I am not overly familiar with geojsonio but isn't that what this package is about?
• Should we aim at implementing this in C++ via Rcpp in order to handle large data performantly? Especially with regard to potentially implementing webgl rendering.

[bhaskarvk]

@tim-salabim geojsonio is for reading/writing geo/topo JSONs from the file-system. What I am proposing is a common package that will take any spatial R object (sp, sf, geo/topo JSONs either as lists, char strings or R objects) and making them available to a htmlwidget in a consistent manner.
That way we can easily make new web GIS plotting pkgs that wrap mapboxGL / Cesium / OpenLayers etc.

It may well be that this package will rely on sp/sf/geojson/geojsonio packages to read the data but what differentiates it is the consistent manner in which it makes this spatial data available to the widget side.

So then you can have code that can look like

leaflet() %>% 
  addPolygons(some<sp|sf>polygon-data,...)

# OR

mapboxGL() %>%
  addPolygons(some<sp|sf>polygon-data,...)

# OR

openLayers() %>%
  addPolygons(some<sp|sf>polygon-data,...)

w/o having to duplicate the code which reads the spatial objects in individual leaflet/mapboxGL/Cesium packages.

[tim-salabim]

relevant rstudio/leaflet#452

[edzer]

For the part (sf -> data.frame, sp -> data.frame), I think it makes more sense to have this as part of the sf and sp APIs, i.e. inside the packages.

For instance, rstudio/leaflet#452 does not happen when leaflet uses st_coordinates on the sfc object, instead of calling do.call(rbind, sfc) which wrongly assumes sfc is not an empty list.

[bhaskarvk]

Yes that's an acceptable solution as well. I just think it belongs outside of leaflet.

[edzer]

Fair enough. Which functions in leaflet does this concern?

[mdsumner]

I've been working on this in spbabel and in a superseded form of that in https://github.com/mdsumner/sc

The discussion and rationale there is my best overview of the landscape, but I've learnt quite a lot more since those were written.

The form must be relational, composed of multiple data frames - that's the only way to store all the types that are needed, and it's the only way to store topology at all (you can't do that with nesting, even if you nest indexes you still need a common pool table for those indexes to refer to).

It's desperately needed that we have a common agreed form for these data in R, and I think specific packages should all contain decompositions to a generic form for their specific types. I've learnt enough in those and related projects for progressing my work but I'm very happy to pursue this in a more general form that transcends all and any specific implementations currently in use. The OSM work by rOpenSci has similar challenges, and osmdata in particular is an important use-case.

@mpadge this is part of the general problem we've been talking about :)

Finally, I'm absolutely delighted to hear this is seen as important and I'm extremely happy to help in any way I can. This is essential for the R community to move forward on, and I look forward to seeing how my explorations will fit into this, thanks @bhaskarvk !

[bhaskarvk]

@edzer All the code in leaflet's R/normalize*.R files is what I was thinking.

I might be going off on a tangent to the theme of this discussion, but, what's r-spatial/the community's thoughts on using encoded polylines to represent geometries?

Whenever I plot a map in googleway I always encode my spatial objects first as it reduces the size of the object being plotted (and the encoded polylines are natively supported in Google Map's API).

I've been playing about with a spatialdatatable package to do the encoding. I don't know how far I'm going to take this package, but if there's appetite to include it in r-spatial then I'll carry on.

[edzer]

I see this similar to s2 cells and geohash; dedicated optimizations where you can afford some rounding and bandwidth is an issue. This one aims at communicating with the google maps stack.

Your package does this, as well as the integration with data.table. Does it make sense to somehow separate that?

If you believe it will attract a larger user community, we could move the package here.

I think separating the encoding/normalising is probably a good idea and would be a better fit for this 'new package' (whatever it turns out to be). And I think there will definitely be a better way of doing the encoding from sf - to polyline than my nested lapply's. I also started to look into boost and CGAL, but haven't progressed with it.

The reason I started writing spatialdatatable was to speed up the geosphere calculations, and also make them naturally usable inside data.table[ ] syntax.

I've made a start by creating googlePolylines to handle the encoding and decoding of (primarily) sf objects into encoded polylines. As mentioned, the encoded lines reduce precision, but can speed-up plotting

I've seen plugins for leaflet to use these polylines too so there may be some opportunity for integration.

Thanks for the reminder @tim-salabim !

Given my recent updates to mapdeck I think I've got a solid base of code to make this 'normalised data' package, so I'm happy to get this going.

anyone got a good suggestion for a package name?