Architecture
This is a description of the Tilezen architecture as it was running at Mapzen in production. Of note is that this is meant to capture the various systems as they were running before rawr tiles and global builds.
The system that emerged reflects the choices and trade-offs that were made given our goals. Typically, the variables that affect most of the decisions are acceptable latencies, both typical and worst case, data freshness, and cost (database, compute, and storage). We wanted to serve tiles that were usually just a few hours out of date, with most of the service being reasonably fast, while it being acceptable for a small percentage of tiles to be slow to render.
The way we chose to balance the goals was to create a concept called the TOI, or tiles of interest list. This represented a set of tiles that should be fast with the most up-to-date data, and is the list of tiles that were "most requested" in a particular time window. This is the set of tiles that was pre-generated by tilequeue and stored to S3 with great (fast, <100 ms) latency. Requests for tiles outside of this set would be served live with poor (slow, ~450 ms) latency.
Requests are satisfied by first trying a cache, then if the tile is pre-generated on S3, and finally it is rendered on-demand if necessary. fastly would run this logic (in a custom VCL script), and the tapalactl service would return a 404 if the tile didn't exist, which would signal to fastly that the request should be directed to tileserver to generate the tile on-demand.
NOTE: Additional logic would route requests based on custom vector tile layers or terrain tiles. The newer tapalcatl-py enables "serverless" but doesn't yet support all the original's features.
osm2pgql applies OpenStreetMap planet file diffs to the database, and it would generate a tile expiry list. tilequeue (tilequeue intersect command) would read this list, and the tiles that were in the TOI (tiles of interest list) would get enqueued onto Amazon's SQS for processing. A number of instances running tilequeue process would read tiles from SQS, generate them and store them onto Amazon's S3.
NOTE: For initial tilequeue seed runs the process is essentially the same as diff processing but with an initial TOI set. The sample config has more details. A high level recommendation is all zoom 0-10 tiles because that's just a million tiles, and then bounding boxes based on countries (say all USA from zoom 10 to zoom 14) or metro extracts (say San Francisco Bay Area all the way down to zoom 16). Mapbox also published a list of top 50k most popular tiles.
To manage the TOI (tiles of interest list), a "gardener" process (tilequeue prune-tiles-of-interest command and config), would run periodically (say daily) to add and remove tiles from the TOI based on a rolling window of requests (usually 2 months average service usage). New tiles would get enqueued for processing by tilequeue, and tiles to be removed would get deleted from S3.
Low zoom tile content doesn't change much so keep it around longer, and prefer to update "max zoom" 15 and 16 tiles at a faster cadence to create a virtuous cycle of edit in OSM.org, go-live in vector tiles, review, and iterate. Zoom 17+ are excessive but popular for generic low-value uses so keep them around longest in cache.
ttls:
"0-10":
ttl: 12h
max-age: 43200
grace: 13h
"11-12":
ttl: 8h
max-age: 43200
grace: 9h
"13-14":
ttl: 4h
max-age: 43200
grace: 5h
"15-16":
ttl: 2h
max-age: 43200
grace: 3h
"17-20":
ttl: 1w
max-age: 604800
grace: 2w
default:
ttl: 4h
max-age: 43200
grace: 5h