Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support multiple input files #701

Merged
merged 1 commit into from
Apr 2, 2024
Merged

Conversation

cldellow
Copy link
Contributor

@cldellow cldellow commented Mar 29, 2024

Fixes #696, a regression introduced in the v3.0 work:

  • fix bug in thread locals for generate_stream and generate_output
    • if the thread local was read on the main thread, we would previously fail to reset it for the subsequent PBF file
  • use the old NodeStore/WayStore when multiple input files are present
    • the optimization for Sort.Type_then_ID assumes IDs are monotonically increasing. This is not true when there are multiple PBFs, as the subsequent PBF restarts its numbering.
  • don't use lazy geometries when multiple input files present
    • LeasedStore assumed that there would be at most 1 PBF being read. This might be fixable by multiplying threadNum by the number of PBFs to be read in the initializers here:
      pointStores(threadNum),
      linestringStores(threadNum),
      multilinestringStores(threadNum),
      multipolygonStores(threadNum),
  • compute the bbox from the union of all the pbfs
    • I think this may have been a pre-existing bug, made more noticeable by changes in v3 that relied more heavily on the bbox being correct

For testing, I built a pmtiles with monaco + liechtenstein and verified they both showed up.

I think we could probably support lazy geometries, but have left it alone for now.

Fixes systemed#696, a regression introduced in the v3.0 work:

- use the old NodeStore/WayStore when multiple input files
- don't use lazy geometries when multiple input files
- compute the bbox from the union of all the pbfs

I suspect we could probably restore lazy geometries for multiple input
files, but I haven't dug in to see what would be required.

For testing, I built a pmtiles with monaco + liechtenstein and verified
they both showed up.
@systemed
Copy link
Owner

systemed commented Apr 2, 2024

Thanks! Tested with Oxfordshire+Worcestershire+Gloucestershire and works well. (I'd had a brief look previously and spotted the thread local issue but not the others.)

@systemed systemed merged commit 0190fba into systemed:master Apr 2, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants