CartoLayer support in pydeck

[Jesus89]

Hi @ajduberstein @kylebarron.

I'm working on extending pydeck functionality to add support for CartoLayer.

I would like to discuss with you what is the best way to approach this development.

Option 1

The fastest way for adding support for that layer is to include @deck.gl/carto in the jupyter-widget module. There are two reasons in favor of this approach:

• @deck.gl/carto is already a dependency of the module jupyter-widget
• @deck.gl/google-maps is already included in the module

With this approach, the layer will be ready to use like any other:

import pydeck as pdk

layer = pdk.Layer(
    'CartoLayer',
    ...)

pdk.Deck([layer])

Option 2

Another option is to create a library in deck.gl/bindings/pydeck-carto with a CartoLayer class that extends pydeck.Layer, similar to this one.

With this approach, we gain flexibility with a different API for the Layer, but also with the possibility of adding more CARTO-specific features in the future.

import pydeck as pdk
from pydeck_carto import CartoLayer

layer = CartoLayer(
    ...)

pdk.Deck([layer])

I would love to hear your opinion and suggestions. Thanks in advance!

[kylebarron]

I think the main question is how much you'll want to update CartoLayer. Pydeck isn't seeing much development, so if you want to update CartoLayer faster than Pydeck sees releases, then it could be easier to distribute as a separate package so that you can completely control the release cadence.

[ajduberstein]

@kylebarron in my mind the path forward is to allow other developers aside from me to release new versions of pydeck. That's simply adding new maintainers to https://readthedocs.org/, our PyPI page, and following our release checklist.

@Jesus89 I'm personally fine with adding the Carto module directly to pydeck. Also happy to add anyone that @alasarr or any member of our technical steering committee wants as a maintainer on the project.

[ibgreen]

I would prefer that we create an installable module similar to the earthengine-layers module. It is very valuable to maintain a clear separation between the core and commercial integrations.

I do get that it is more convenient both for CARTO devs and users to not do that. but please consider that while CARTO is currently the only non-basemap integration module in the deck.gl repo, my bet is that this will soon change and things will get weird if we don't keep these modules separate.

[alasarr]

Thanks for all the comments! I think I would prefer Option 1 because of these reasons:

• The general philosophy of CARTO module is to make the smoother integration for deck.gl users, so CartoLayer interfaces should be like any other layer. With option 2 we're breaking this.

• CARTO's users sometimes use other layers (not always CartoLayer). For example, if they've data in CARTO and want to use an H3, Quadkey, IconLayer... they can use this method. It's not a good idea for me to break this API. It'll be more difficult for the user, depending on the layer they'll need to use a different API.

• It was already available with the old CartoSQLLayer and not working from 8.6, so for me, it's a bug we introduced.. CartoSQLLayer layer was announced for deprecation at 8.5 release and finally deprecated in 8.6.

• If CARTO includes this layer in pydeck, CARTO will bring more users to pydeck (we're migrating CARTOframes visualization module to pydeck), so we will also bring support and more people to contribute with pydeck.

I would prefer that we create an installable module similar to the earthengine-layers module. It is very valuable to maintain a clear separation between the core and commercial integrations.

I think in deck.gl we've a good separation with the different packages (@deck.gl/carto, @deck.gl/mapbox, @deck.gl/arcgis, @deck.glgoogle-maps...). Not an expert on python (maybe I'm wrong), but what I feel is that we don't have a system like this in pydeck.

I propose we move forward with option 1, which will get the ball rolling and bring more users/contributors to pydeck. Then, we can focus on a long-term solution for this.

[alasarr]

Would be possible option 2, but keep the same API for CartoLayer and other layers of deck.gl? I think that solution will make everybody happy

[ibgreen]

I think in deck.gl we've a good separation with the different packages (@deck.gl/carto, @deck.gl/mapbox, @deck.gl/arcgis, @deck.glgoogle-maps...). Not an expert on python (maybe I'm wrong), but what I feel is that we don't have a system like this in pydeck.

Well, the Unfolded team did the initial work in earthengine-layers, showing how to build the required plumbing for separate Python modules. I feel that was the hardest part. There might be a little bit more plumbing required to support loading multiple add-on Python modules, but the basic pattern should hold. The main trick was to create the JS bundle in the proper format, I believe we used IIFE, the config files are all open source.

• I am open to the idea of breaking out a few more modules if that makes a separate pydeck_carto module more palatable.
• It could make sense to break out the basemap modules.
• However I don't think we need a granular 1-1 mapping of the various deck.gl layer modules. They can all go into the core pydeck as there are no real bundle size concerns.
• (BTW I don't think there is deck.gl/arcgis integration in pydeck today).

[alasarr]

I had no idea you did such good progress there!

[Jesus89]

Hi all. Thanks for your quick responses.

Taking into account all the information provided, I think, although option 1 seems a quick win in the short term, it can be a problem in the long term. Option 2 gives us more flexibility, so we can add more features in pydeck-carto without making the pydeck core bigger.

@kylebarron Regarding the frequency of updates in CartoLayer, I think it will be not too high. Our main goal is to be able to use the current version of the CartoLayer in pydeck.

@ajduberstein We'll be glad to help with the releases and maintenance of pydeck too.

@ibgreen Thanks for commenting on that point. I think it makes sense to make a separation between core and extra modules.

@alasarr Yes. I made a test and we can have the same API for CartoLayer by editing pdk.settings.custom_libraries. Using the CartoLayer class has some advantages though.

[Jesus89]

Setup

To use @deck.gl/carto as a custom library I have added in the bundle.ts the following line:

globalThis.CartoLayerLibrary = {CartoLayer: CartoUtils.CartoLayer};

I think it could make sense to include this, so we can use the bundle from unpkg instead of generating a new one.

Test 1

Importing pydeck_carto registers and makes available the CartoLayer by editing pdk.settings.custom_libraries. This is an implicit action to make it easy for the user, but it can be an explicit function if required.

In the example, I'm using the type_ workaround to bypass the type param collision. Also use pdk.types.String(...) to make explicit pydeck Strings. You can see it works like any other pydeck Layer.

Test 2

This example is using a CartoLayer class, which provides a new constructor, so we can change the params and use type (or better map_type). Check that type and connection are converted internally to pydeck Strings.

[Jesus89]

I will start a draft of pydeck_carto module.

Any suggestion or recommendation will be welcome!

[alasarr]

good job man!

[alasarr]

At #6880 @Jesus89 has included the module of CARTO for Pydeck inside of the bindings folder, we need to make a final decision on this. I prefer to keep it as it's and encourage more companies to do the same (also because CARTO module for deck is there). For example ESRI could do the same. Said that I don't have a strong opinion and I'm also ok to move this to another repo.

I understand that @ibgreen prefers an external repo, but before start moving I'd like to hear your thoughts on this @Pessimistress and @ajduberstein.

[ibgreen]

Separating "github repository" and "published modules":

So far I have insisted that CARTO-related pydeck functionality is published as a separate python module, and not included in the main pydeck module, just as we do for the JavaScript modules.

We already publish many different modules out of the single deck.gl monorepo, and my expectation is that can be done without moving CARTO code to a new repository.

---

So far we haven't discussed splitting CARTO code into a separate repo, though on reflection it strikes me now that it may be worth a discussion.

After all, the code in the deck.gl repository is supposed to be fully under open governance (not just open source). However with the growth of the CARTO module it seems to no longer just be a thin pass-through binding to a commercial service but a substantial and growing set of functionality developed and controlled by one company rather than the community.

It is not an urgent issue to solve, but at some point this can affect how large companies view the deck.gl repository. E.g. it is either fully open governance, or it is not...

[alasarr]

Thanks a lot @ibgreen for the clarification I better understand your point now.

Our idea is to publish it as an external package, we're trying to replicate in python what we already have for Javascript/NPM.

The CARTO module, I agree it's growing lately but just because CARTO is a large platform. The module is mainly an integration with the platform, and the vision is to include only the code required for that integration and move outside everything that can be useful for other users (for example QuadkeyLayer). That's true we've there some styling functionality that I feel we should move outside, but that's part of another discussion :).

Regarding governance, that's not necessarily bad for large companies, I think they'll understand that a commercial module will be managed by the company that provides/supports it. That could be a good thing for new companies as they will see their product can be integrated in deck, and be a good citizen inside an open-source environment.