Can tilequeue regenerate tiles based on osmosis diffs?

[ambientlight]

How do you guys regenerate the tiles based on OSM updates?

I have found that I can pre-generate tiles with tilequeue, however I haven't found any info on how do you guys refresh them. I only found that config specifies expired-location, but it didn't give me any further clues.

1. What do you use to invalidate/delete outdated tiles? It doesn't seem to be a tilequeue's job since I haven't found any .yamls that specify when certain zoom-level tiles become outdated?

2. Can I feed osmosis generated .osm.gz to tilequeue so it will look on query config, do a bunch of postgres queries and figure out what tilesets need to be recursively updated?

If there are no such mechanisms, writing tile refresh scripts is not a big deal, but doing tile updates based on osmosis diffs is bit more complex. I technically can wait until osmosis applies .osm.gz, parse the .osm.gz xmls, query-config .yamls and osmosis styles and then it is technically possible to understand what features corresponsd to this osm object, so I can then recursively regenerate the tileset within this quad subtree. What do you recommend here?

[nvkelso]

Rob or Matt can provide more details, but we use a different process described here: - https://github.com/tilezen/chef-minutely_mapnik

…

On Wed, Apr 12, 2017 at 9:38 PM, Taras Vozniuk ***@***.***> wrote: How do you guys regenerate the tiles based on OSM updates? I have found that I can pre-generate tiles with tilequeue, however I haven't found any info on how do you guys refresh them. I only found that config specifies expired-location, but it didn't give me any further clues. 1. What do you use to invalidate/delete outdated tiles? I doesn't seem to be a tilequeue's job since I haven't found any .yamls that specify when certain zoom-level tiles become outdated? 2. Can I feed osmosis generated .osm.gz to tilequeue so it will look on query config, do a bunch of postgres queries and figure out what tilesets need to be recursively updated? If there are no such mechanisms, writing tile refresh scripts is not a big deal, but doing tile updates based on osmosis diffs is bit more complex. I technically can wait until osmosis applies .osm.gz, parse the .osm.gz xmls, query-config .yamls and osmosis styles and then it is technically possible to understand what features corresponsd to this osm object, so I can then recursively regenerate the tileset within this quad subtree. What do you recommend here? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#186>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA0EO1f3MKZi6G5X8r9sFXrxgdINWt5kks5rvaaqgaJpZM4M8QA3> .

[ambientlight]

awesome, thanks!

[nvkelso]

I'm going to close this issue for now, but if you have followup questions please reopen it here.

[ambientlight]

chef-Minutely_mapnik seems like a helper to run the osmosis and consume it into postgres.

It doesn't look like it calls any other tool to update the tiles based on the osmosis diff. Or you have some mechanism in tilequeue that will find what has changes in postgres and regenerate only the needed tiles?

[zerebubuth]

Yes, we run osm2pgsql with the --expire-tiles 16-16 to get the list of updated tiles at zoom 16. This is fed into the tilequeue intersect command which generates all the lower zoom tiles which include those zoom 16 tiles. It takes all of those and finds which are also part of our "Tiles of Interest" (TOI) set. This is a set of tiles which have been requested via tileserver or we think are important for other reasons - there's a bunch of ongoing work in this area to make the TOI more flexible and adaptable, e.g: #176.

The results of the TOI intersection are put into an SQS queue and get re-rendered eventually on our tilequeue cluster.

If you wanted to run this without our TOI steps, then I think it's possible to feed the output of osm2pgsql's --expire-tiles 0-16 step straight into a "file" queue for tilequeue, which just reads tile coordinates from a file and renders them.

[ambientlight]

@zerebubuth: Thanks a lot for a flow explanation. It's great to know it is available out-of-the box.

One questions regarding TOI. So TOI are these tiles which are often requested + extra tilesets manually configured in tilequeue/config.yaml. So primarily for the sake of saving computation power/bandwidth tilequeue will not rerender the tiles outside TOI -> only delete them. Is my thinking correct here?

[zerebubuth]

Yes, that's almost it. The current behaviour is that we take tiles which are often requested and extra tiles that we think are important to make the initial TOI. Then, as the system is running, the tileserver adds any tiles that it serves into the TOI. As you point out, this saves the effort of rendering and updating tiles which have never been viewed.

However, this leads to a TOI which always grows bigger and bigger, and effort might be spent rendering updates for tiles which were only viewed once and have not been viewed since. Controlling that is the idea behind #176, so that tiles which are not viewed often will be removed from the TOI (and deleted from S3), and we'll save any further effort needed to keep them up-to-date.

Depending on how you want to use the tiles, using a TOI might or might not be the right workflow. When the size of the TOI set is small this can be a very effective technique to keep the tiles up-to-date. Unfortunately, when the TOI gets into the millions or tens of millions, re-rendering updates in the TOI takes up quite a lot of computer time!

[ambientlight]

@zerebubuth I got it, thanks a lot. Great, #176 has been merged, we can play around with it! Thanks!