-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: replace logic in OsmPbfLoader from osmium to QuackOSM #405
Merged
Merged
Changes from 33 commits
Commits
Show all changes
163 commits
Select commit
Hold shift + click to select a range
d0fbd70
chore: add duckdb dependency
RaczeQ a45754a
feat: add first working pipeline
RaczeQ 238682b
feat: add geoarrow-python dependency
RaczeQ fef8fbf
feat: modify tests and add geoparquet functionality
RaczeQ 784ae19
fix(pre-commit.ci): auto fixes from pre-commit.com hooks
pre-commit-ci[bot] ea2e138
chore: remove osm pbf files clipping
RaczeQ 3b14eac
Merge branch 'duckdb_osm_native_reader' of https://github.com/kraina-…
RaczeQ 991c37b
chore: add geoarrow-python dependency
RaczeQ 2560fda
fix: modify tests
RaczeQ eaab0ac
chore: remove prints
RaczeQ a5960fa
chore: add duckdb connection closing
RaczeQ c5cb5cc
feat: scale up PbfFileHandler for big files
RaczeQ 73462e2
chore: apply refurb suggestions
RaczeQ c043099
Merge branch 'main' into duckdb_osm_native_reader
RaczeQ 5d3e471
fix: change sql query for ways intersections
RaczeQ f41aa78
Merge branch 'duckdb_osm_native_reader' of https://github.com/kraina-…
RaczeQ 6083e1d
chore: add debugging message for tests
RaczeQ 0a89035
chore: add debugging message for tests
RaczeQ dab044f
fix: change empty required ways case
RaczeQ 806b2da
chore: remove debugging message from tests
RaczeQ b12cf6f
chore: add in-sql features grouping
RaczeQ 026f257
chore: modify docstrings
RaczeQ b5f8eeb
chore: add automatic directories removal
RaczeQ ab841eb
chore: modify docstrings
RaczeQ d9e09b8
chore: remove comments from SQL
RaczeQ 9ccae64
chore: add changelog entry
RaczeQ 15421a9
fix: change optional imports and directory removal
RaczeQ a97b8ec
chore: change default download source from protomaps to geofabrik
RaczeQ 1c3ba65
chore: add option to load data to geoparquet only
RaczeQ 3e4523b
chore: change default download source from protomaps to geofabrik
RaczeQ 7c3f3f6
Merge branch 'main' into duckdb_osm_native_reader
RaczeQ 7f1812d
fix(pre-commit.ci): auto fixes from pre-commit.com hooks
pre-commit-ci[bot] f5652cd
Merge branch 'main' into duckdb_osm_native_reader
RaczeQ f63d327
chore: lock geoarrow-pyarrow commit ref
RaczeQ 12bf153
refactor: simplify the osm pbf loader code
RaczeQ 41c7078
feat: add more options to merge OsmTagsFilters
RaczeQ 0e352e6
chore: removed PbfFileClipper
RaczeQ 0f8b666
Merge branch 'main' into duckdb_osm_native_reader
RaczeQ 6589068
chore: add explode_tags parameter
RaczeQ 5cc6372
Merge branch 'main' into duckdb_osm_native_reader
RaczeQ 03bc05b
chore: add pyogrio for testing osm reader
RaczeQ d5f76f9
Merge branch 'duckdb_osm_native_reader' of https://github.com/kraina-…
RaczeQ 8f31b89
chore: update pyarrow and ruff versions
RaczeQ a7ec353
feat: add logic for explode_tags parameter
RaczeQ 7d0e28c
fix(pre-commit.ci): auto fixes from pre-commit.com hooks
pre-commit-ci[bot] baa7cdb
chore: add new parameter to tests
RaczeQ 4ad0575
Merge branch 'duckdb_osm_native_reader' of https://github.com/kraina-…
RaczeQ bd9094d
chore: lock pyarrow minimal version
RaczeQ 6c865e4
fix: change explode_tags value in tests
RaczeQ 4d2f654
chore: add osmconf.ini file to test files
RaczeQ ab1a6c4
Merge branch 'duckdb_osm_native_reader' of https://github.com/kraina-…
RaczeQ cb43c7c
feat: add metadata tags filtering
RaczeQ 1a1f7b9
chore: add pghstore for testing purposes
RaczeQ 9b88469
chore: change osmconf file
RaczeQ d6eb207
chore: change osmconf file
RaczeQ f7dc198
test: add gdal testing with geometries
RaczeQ b7a91ac
chore: paste way polygon features configs
RaczeQ ee9a9ba
feat: add dynamic osm way polygon filter generation
RaczeQ 243177c
chore: add first tests for gdal parity
RaczeQ cdb3922
chore: apply refurb changes
RaczeQ e553cd7
chore: apply refurb changes
RaczeQ 1270de5
chore: change pghstore source
RaczeQ bddba85
chore: remove pghstore dependency
RaczeQ 7e183a0
chore: change gdal loading logic
RaczeQ ce40eae
chore: change tests logic with ogr2ogr
RaczeQ b14dfc9
chore: add option to check all geometries during single test
RaczeQ 7e238bb
chore: add GDAL installation step
RaczeQ 209c8d8
chore: change GDAL installation step
RaczeQ effe3f5
chore: change GDAL installation step
RaczeQ dd110de
chore: change GDAL installation step
RaczeQ f0da70a
chore: change GDAL installation step
RaczeQ 13ed2e9
chore: change GDAL installation step
RaczeQ e0f67a1
chore: add GDAL installation step
RaczeQ 20e2432
chore: change GDAL installation step
RaczeQ f2ad317
chore: change GDAL installation step
RaczeQ 871bfe3
chore: change GDAL installation step
RaczeQ cac7241
chore: change GDAL installation step
RaczeQ e718fe1
chore: change GDAL installation step
RaczeQ 0ce1658
chore: add GDAL version checking
RaczeQ 8f6a332
chore: add GDAL version checking
RaczeQ 8089980
chore: add GDAL version checking
RaczeQ 684fe90
chore: add GDAL version checking
RaczeQ 935541d
chore: add GDAL version checking
RaczeQ f1bdef6
chore: add GDAL version checking
RaczeQ a95ed2d
chore: add GDAL version checking
RaczeQ df08a05
chore: add GDAL version checking
RaczeQ a0931c5
chore: add ogr2ogr to path
RaczeQ 08a6f3a
chore: add ogr2ogr to path
RaczeQ c69f35a
chore: add ogr2ogr to path
RaczeQ cdc72bb
chore: add ogr2ogr to path
RaczeQ 05f43a8
chore: add ogr2ogr to path
RaczeQ 74d3aac
chore: add ogr2ogr to path
RaczeQ be64c54
chore: add ogr2ogr to path
RaczeQ 87d36f3
chore: add ogr2ogr to path
RaczeQ d3c6d23
chore: add ogr2ogr to path
RaczeQ 3768260
fix: change ogr2ogr execution on windows
RaczeQ 830bff7
chore: add skipping if ogr2ogr is not found
RaczeQ 8399560
chore: remove prints and echos
RaczeQ 6301f3e
fix: change job config
RaczeQ a22ece4
fix: change retry command
RaczeQ 8a5d878
fix: change timeout config
RaczeQ 7b913d2
fix: add required config value
RaczeQ 83d2207
chore: change timeout length
RaczeQ f1351bd
chore: revert retry for gdal installation
RaczeQ 768eceb
chore: change tests
RaczeQ 1930d60
chore: fix duckdb way geometries
RaczeQ f9aa607
chore: modify gdal parity test
RaczeQ 0ef257d
fix: skip invalid relations from evaluation
RaczeQ 770a393
chore: start changing relations parsing
RaczeQ 9da8e92
chore: save todo note
RaczeQ f3e0408
fix: change geometries difference logic
RaczeQ ae5fc1c
chore: cut test examples
RaczeQ be47403
fix: change pbf_reader logic
RaczeQ 9005ece
chore: speed up sql left join
RaczeQ d960202
feat: add logic to parse relations without outer geometries
RaczeQ f03f112
feat: update remove_interiors function
RaczeQ bd755b2
chore: modify geometry checking flow
RaczeQ 6e08f60
chore: add new logic to geometry checking logic
RaczeQ 8fdcce0
chore: extract comparation logic to another function
RaczeQ aab80c8
chore: remove tqdm progress bar
RaczeQ 752cf80
fix: change comapration logic
RaczeQ 79bc092
chore: break relations geometry building into steps
RaczeQ a70b2c8
chore: add more in-depth tags comparation
RaczeQ 3ed2558
chore; modify hstore parsing
RaczeQ ab18fbb
chore: add check for valid geometries in relations
RaczeQ 0d5b57a
chore: remove same_number_of_points condition
RaczeQ fcd3a65
chore: vectorize pandas operations for tags
RaczeQ 871690c
chore: merge branch 'duckdb_osm_native_reader' of https://github.com/…
RaczeQ 20d4a6d
fix: change pandas bool condition
RaczeQ 6363e96
fix: change tags comparison
RaczeQ acea5c4
fix: change tags comparison
RaczeQ 6c3eb0d
chore: vectorize geometry comparison
RaczeQ debaedb
chore: change few examples for gdal parity test
RaczeQ cf9d3b9
chore: upload monaco pbf file
RaczeQ 8a022cd
chore: change test values for a new monaco extract
RaczeQ 6560eee
chore: change monaco osmpbfloader tests
RaczeQ 0eea4fe
chore: change tests and add new ignore_cache flag
RaczeQ 5980a04
chore: remove testing notebooks
RaczeQ e3c68ab
chore: add geometry fixing for polygons in relations
RaczeQ 2830a81
Merge branch 'main' into duckdb_osm_native_reader
RaczeQ c1666c4
feat: add quackosm dependency
RaczeQ 3d5df3d
Merge branch 'duckdb_osm_native_reader' of https://github.com/kraina-…
RaczeQ 0dff419
feat: replace PbfFileReader with QuackOSM implementation
RaczeQ 244ec4b
chore: remove osmconf.ini
RaczeQ 28c1801
chore: merge branch 'main' of https://github.com/kraina-ai/srai into …
RaczeQ 6914b92
chore: remove gdal installation
RaczeQ 9a5fe3b
chore: change osm pbf loader example notebook
RaczeQ fbf8704
chore: remove osm way polygon config
RaczeQ c47a7ad
chore: change docstring
RaczeQ 8797ef6
chore: change optional imports
RaczeQ 30ffde5
chore: bumped QuackOSM version
RaczeQ 2836d2c
refactor: removed pbf related classes from srai
RaczeQ fbaf865
chore: bumped quackosm version and moved osmnx to main dependencies
RaczeQ e048c61
chore: change default download source for pbf files
RaczeQ 5d2d983
fix: change osm pbf loader example
RaczeQ e98d4f0
chore: modified changelog entries
RaczeQ 973e434
fix: add new error from osmnx
RaczeQ 87db89e
refactor: add OsmExtractSource typing
RaczeQ 35fb94c
chore: add geoparquet related tests
RaczeQ 649dfec
chore: change filters for test with geoparquet osm pbf
RaczeQ 0966687
chore: bump quackosm version
RaczeQ af5f50b
Merge branch 'main' into duckdb_osm_native_reader
RaczeQ 98aa6b5
fix: typo in changelog
RaczeQ File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't get the need for duckdb examples. I understand the part with loading data into geoparquet, but not the next things. I'm not saying we should drop them completely, just wondering if we should expect our users to use raw DuckDB and have an example of that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wanted to emphasize the possibility of using
srai
as the tool for just parsing the*.osm.pbf
file togeoparquet
in big scale, since it's a big use case in the current cloud computing. Do you think we should add a new example notebook just for that, focused solely on thePbfFileHandler
?