Wmts Probe

[jochemthart1]

This PR adds a probe for Mapbox Vector tile service based on issue #379.

Design:

This probe contains 2 probes. One that requests a single layer and another that requests all layers.

KVP/REST
Because WMTS can be requested through KVP and REST, there are a few pieces of code in place to determine whether to request through KVP or REST.

In expand_params() it is determined if the base_url can be requested through KVP, REST or both. These are given as user input options with KVP being default if available.

In wmts.py the REQUEST_TEMPLATE is a dict object that features both a KVP and REST url template. The request template that is used is chosen based on the user input

Issues with REST requests
Because owslib WebMapTileService does not support url's that only provide REST requests, I made sure to check in get_metadata()-wmts.py and WmtsGetCaps(OwsGetCaps)-owsgetcaps.py if KVP request works and to add "/1.0.0/WMTSCapabilities.xml" if not before requesting metadata.
Example for this issue would be this endpoint that can only be accessed through REST:

• Capabilities url: ".../wmts/1.0.0/WMTSCapabilities.xml"
• User input url: ".../wmts"
• Normal get_metadata() request: ".../wmts?service=WMTS&version=1.0.0&request=GetCapabilities" (this will fail)
• Request should be: ".../wmts/1.0.0/WMTSCapabilities.xml"

after_request is used to restore self._resource.url to the original user input url.

I realize this solution is not really neat, so I'm open to other suggestions.

Requesting center tile
For GetTile requests, the center tile index is calculated using toprightcorner coordinates, scale and matrixwidth/height. Most WMTS services should have some data at the center tile.

Issue in probe.py
In probe, an adjustment has been made to getting self._parameters. copy.deepcopy() is used from plugin.py. This is because previously, the parameters would be overwritten in the following piece of code:

for param in request_parms:
    if param_defs[param]['type'] == 'stringlist':
        request_parms[param] = ','.join(request_parms[param])

Instead of adjusting the request_params variable in probe.py, the original self._parameters was changed.
This problem probably never became apparent, because other probes only send 1 request per layer, but for WMTS, there are multiple requests for each layer and this meant that more and more "," would be added to the request string giving url's similar to this:
.../wmts?service=WMTS&version=1.0.0&request=GetTile&layer=l,,,a,,,y,,,e,,,r&etc...

[justb4]

Almost there: to complete add a test for WmtsGetTileAll. Maybe use a PDOK WMTS URL for variety. Thanks for patience!

[jochemthart1]

I specifically left out WMTS GetTileAll because many resources have so many layers that it might take 10-20 minutes. However, I did find a pdok resource that doesn't take too long, but total testing time is still 16 minutes now.

[justb4]

I specifically left out WMTS GetTileAll because many resources have so many layers that it might take 10-20 minutes.
However, I did find a pdok resource that doesn't take too long, but total testing time is still 16 minutes now.

That is indeed quite long. Is that because GetTileAll goes through all Layers, Zoomlevels and (image) Formats?
We had a similar problem with WMS. Also that many of the formats advertised simply did not work or their conversion took too long. If those Probes take too long, than that is straining for the WMTS (and GHC) anyway. I see no easy solution, unless like an option to have a single image format with GetTileAll as parameter option.

[jochemthart1]

I specifically left out WMTS GetTileAll because many resources have so many layers that it might take 10-20 minutes.
However, I did find a pdok resource that doesn't take too long, but total testing time is still 16 minutes now.

That is indeed quite long. Is that because GetTileAll goes through all Layers, Zoomlevels and (image) Formats? We had a similar problem with WMS. Also that many of the formats advertised simply did not work or their conversion took too long. If those Probes take too long, than that is straining for the WMTS (and GHC) anyway. I see no easy solution, unless like an option to have a single image format with GetTileAll as parameter option.

GetTileAll goes through all layers, each tilematrixset (crs), and each zoomlevel, sampling the center tile. It takes the image format from user input.

pdok WMTS
The above PDOK WMTS resource for example has 4 layers, 3 tilematrixsets with 15, 12 and 20 zoomlevels respectively, which comes down to 188 requests in total.

Changing GetTileAll to randomly sample a zoomlevel (12 checks for this url) or timematrixset (~60 checks for this url) might be a solution. Changing name to GetTileSample would be in order then.

[justb4]

That (GetTileSample) is a good suggestion. My suggestion: instead of going through all MatrixSets (zoomlevels), take at least two. Maybe the lowest and highest levels. In my tiling-setups (and I think PDOK and well-setup tiling as well) for example levels 1-13 (RD) are cached and 14+ always requested from source (and never cached) or "blown-up" from level 13 (MapProxy option). So that is a good test if the cache works but also if requesting from the source (WMS) works. A random level could miss out the source request.

[jochemthart1]

Ready for review
I have made the proposed changes and it is now possible to choose to sample or to check all of the tilematrixsets (projections) and tilematrices (zoom levels). This is possible for both GetTile and GetTileAll. I have not changed the name for GetTileAll as this probe still checks all layers opposed to GetTile which asks the user to select one layer.

Tilecol and tilerow indices were set on the center tile within the bounding box and could not be changed. I adjusted this to make the center coordinates of the bounding box appear as a default in the user parameters. A user can change these values to other WGS84 coordinates if they desire to request tiles based on the tile indices of these coordinates

Tests are also adjusted to the new probe parameters. GetTile is tested with REST and KVP and a sample of tilematrixsets and all tilematrices. GetTileAll is tested with KVP and a sample of tilematrixsets and tilematrices.

[justb4]

Seeing this just now. Checked the PR list regularly but it still said "Changes requested" and got no notification.
The changes from jan 25 are after our meeting right?

[justb4]

Let's just merge and test with some diverse WMTSs. It is hard to oversee all code-scenarios with REST, KVP etc. Any problems will be local to the Probe is expected.

[justb4]

Thanks for contribution and patience @jochemthart1 !

[justb4]

All ok, let's go!