Enable 3rd party vector tile sources

[wilhelmberg]

@yhahn do you have time to review/test/merge, or tell me, whom else I should ask?
@springmeyer FYI

It's a hack, any suggestions for a clean solution are welcome

Current State

• about to release next Mapbox Studio version
• with the adoption of vector tiles 3rd party vector tile services will be more common and the need to style them with MBS, too.
• MBS does not sent Accept-Encoding:gzip, because mapbox servers always send gzipped vector tiles
• tile* modules expect gzipped vector tiles
• 3rd party servers compress their response, depending on the presence of Accept-Encoding
  • e.g.: wikimedia: Accept-encoding not sent by MapBox Studio mapbox-studio-classic#1268 (comment)
• as MBS sends requests without Accept-Encoding:gzip, servers respond with uncompressed content
• this uncompressed response breaks tile* modules
• just adding Accept-Encoding:gzip to requestjs header options, makes the 3rd party vector tiles work, but breaks other stuff, as uncompressed content is expected.
• adding gzip:true to requestjs options makes the other stuff work, because requestjs decompresses on the fly, but breaks the 3rd party vector tiles as they are also decompressed before being passed to the tile* modules.

Obvious problem

If 3rd party tile services use a different extension than .pbf or .mvt

Hacky Solution

• always send Accept-Encoding:gzip header to get gzipped responses
• let requestjs decompress (or not) on a pre request basis (gzip:[true|false]). currently I'm checking, if
  • there is .pbf or .mvt in the url => do nothing: gzip:false
  • if not => uncompress: gzip:true

Manual tests

I tried

• Mapbox Styles: streets, satellite, comic, ...
• custom styles with custom source hosted by Mapbox
• custom styles with 3rd party tile services: Mapzen and Wikimedia

Didn't ru into any problems

[wilhelmberg]

Ooops, tests seem to need some care after that change.

[wilhelmberg]

All green again:
https://travis-ci.org/mapbox/node-tilejson/builds/77412588
https://ci.appveyor.com/project/Mapbox/node-tilejson/build/1.0.80

[yhahn]

@BergWerkGIS I'm merging #39 and rolling a new release based on request first. I've read your PR and have some mulling/soul searching to do...

[pnorman]

This is more to follow the HTTP specification than to meet the needs of third-party sources.

[springmeyer]

I'm working on getting out a new Mapbox Studio Classic release (motivated by having a stable base before moving Mapbox Studio Classic to producing v2 vector tiles per mapbox/vector-tile-spec#39).

• I've noticed that Mapbox Studio Classic is still using the request-mbs branch in discussion here: https://github.com/mapbox/mapbox-studio-classic/blob/7b21fa460f0dedc377a47051916dfefd83313726/package.json#L27
• We don't have tests for this hack in place so I'm worried it may regress.
• Also node-tilejson releases have continued along so the request-mbs branch is now out of date.

My plan:

• Create a new branch that applies the hack/fix from @BergWerkGIS to latest master
• Point Mapbox Studio Classic at that new branch
• Release new Mapbox Studio Classic and hope that third party tile services work
• In the meantime I've emailed @nyurik to get details on the wikimedia endpoint so that we can eventually add tests to Mapbox Studio Classic to ensure that this functionality does not regress in Mapbox Studio Classic.

/cc @yhahn @BergWerkGIS

[springmeyer]

@yhahn - as part of the v2 spec push team @mapbox/mapnik is hitting a lot of tilelive modules. If you think it might be a good idea to get tilelive modules updated to support non-compressed tiles, let me know and we can push on that.

[springmeyer]

I'm working on getting out a new Mapbox Studio Classic release (motivated by having a stable base before moving Mapbox Studio Classic to producing v2 vector tiles per mapbox/vector-tile-spec#39).

This never happened since it was not a priority. So I think this PR can be closed.