Skip to content

Commit

Permalink
feat(download): Allow downloading combined files from any mirror
Browse files Browse the repository at this point in the history
Background: Over at Geocode Earth we've now set up our own mirror of WOF
data that sits a bit closer to our servers. It lets us kick off new
builds, etc a bit faster than downloading from the _public_ WOF server
we host on Bunny CDN.

When trying that out today, I noticed some logic from ~2 years ago was
causing the download script to download not only the combined,
planet-level files, but every country file as well.

I think this logic dates back from when we were transitioning off
`dist.whosonfirst.org`, which as I recall didn't have combined files
(they had recently moved from one giant Github repo to the per-country
repository setup that exists today).

As far as I can tell this logic isn't needed any more, it's fair to
assume any WOF mirror would follow the same directory structure as the
primary mirror that we host, meaning it would have both combined and
country-level files. That's what this PR now allows for.
  • Loading branch information
orangejulius committed Sep 23, 2021
1 parent 7b4a00b commit fa43ca3
Showing 1 changed file with 1 addition and 2 deletions.
3 changes: 1 addition & 2 deletions utils/download_sqlite_all.js
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,7 @@ function download(callback) {
const countries = getCountriesToDownload();
return (e) => {
if (countries.length === 0) {
// This is specific to geocode earth, it will select global sqlites
return COMBINED_REGEX.test(e.name_compressed) || wofDataHost !== DATA_GEOCODE_EARTH_URL;
return COMBINED_REGEX.test(e.name_compressed);
}
// This will download sqlites with the selected country code
return COUNTRY_REGEX.test(e.name_compressed) && countries.some(c => e.name_compressed.indexOf(`-${c}-latest`) >= 0);
Expand Down

0 comments on commit fa43ca3

Please sign in to comment.