Repository navigation
inherent limit on OSM ID bit range? #297
Description
Describe the bug
This might be rather curious or anecdotal, but maybe someone else stumbles over this too:
I created a custom OSM file from proprietary data (no mingling with actual OSM data). Most objects have their own IDs already which I just use as OSM ID (weird range from 1 - 48ish bits). Some objects don't and I have to assign an ID. Since OSM IDs are technically valid up to 64bit and this custom dataset doesn't seem to have IDs > 50ish bits, I went up to 62bit for the "synthetic" IDs.
When importing them into ES it seemed to be ok according to the logs:
log
info: [openstreetmap] Creating read stream for: /data/openstreetmap/berlin-charging.eclever.pbf
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/ASM.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/GUM.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/MNP.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/PRI.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/USA.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/USA.override.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pe
berlin-charging.eclever.pbf.zip
lias-wof-admin-lookup/src/data/VIR.tsv
debug: [wof-admin-lookup] [postalCityMap] loaded /code/pelias/openstreetmap/node_modules/pelias-wof-admin-lookup/src/data/NLD.tsv
info: [wof-pip-service:master] starting with layers neighbourhood,borough,locality,localadmin,county,macrocounty,macroregion,region,dependency,country,empire,continent,marinearea,ocean
info: [wof-pip-service:master] borough worker loaded 12 features in 0.031 seconds
info: [wof-pip-service:master] locality worker loaded 1 features in 0.02 seconds
info: [wof-pip-service:master] county worker loaded 0 features in 0.014 seconds
info: [wof-pip-service:master] localadmin worker loaded 1 features in 0.023 seconds
info: [wof-pip-service:master] macrocounty worker loaded 0 features in 0.011 seconds
info: [wof-pip-service:master] macroregion worker loaded 0 features in 0.01 seconds
info: [wof-pip-service:master] neighbourhood worker loaded 103 features in 0.106 seconds
info: [wof-pip-service:master] region worker loaded 1 features in 0.024 seconds
info: [wof-pip-service:master] dependency worker loaded 0 features in 0.011 seconds
info: [wof-pip-service:master] empire worker loaded 0 features in 0.009 seconds
info: [wof-pip-service:master] continent worker loaded 0 features in 0.009 seconds
info: [wof-pip-service:master] marinearea worker loaded 0 features in 0.01 seconds
info: [wof-pip-service:master] ocean worker loaded 0 features in 0.01 seconds
info: [wof-pip-service:master] country worker loaded 1 features in 0.072 seconds
info: [wof-pip-service:master] PIP Service Loading Completed!!!
info: [wof-admin-lookup] Shutting down admin lookup service
info: [admin-lookup:worker] neighbourhood worker process exiting, stats: {"calls":3,"hits":3,"misses":0}
info: [admin-lookup:worker] ocean worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] macrocounty worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] country worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] borough worker process exiting, stats: {"calls":3,"hits":3,"misses":0}
info: [admin-lookup:worker] localadmin worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] dependency worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] region worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] empire worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] continent worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] marinearea worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] locality worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] county worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [admin-lookup:worker] macroregion worker process exiting, stats: {"calls":0,"hits":0,"misses":0}
info: [dbclient-openstreetmap] paused=false, transient=1, current_length=0
info: [dbclient-openstreetmap] paused=false, transient=1, current_length=0
info: [dbclient-openstreetmap] paused=false, transient=0, current_length=0, indexed=3, batch_ok=1, batch_retries=0, failed_records=0, venue=3, persec=0.3
info: [dbclient-openstreetmap] paused=false, transient=0, current_length=0, indexed=3, batch_ok=1, batch_retries=0, failed_records=0, venue=3, persec=0.3
But then I couldn't find them with /search, e.g. /v1/search?categories=transport&focus.point.lat=52.469763&focus.point.lon=13.440254&text=bla1 and the pelias elastic stats says:
{
"key" : "openstreetmap",
"doc_count" : 1,
"layers" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "venue",
"doc_count" : 1
}
]
}
}
I guess it's some ES field definition (no idea about ES :D)?
Steps to Reproduce
Load the attached minimal PBF file:
pelias elastic start && pelias elastic wait && pelias elastic create
pelias import osm
pelias elastic stats
Expected behavior
Either the importer complaining about "invalid" entries or a successful import. Like I said, curious but pretty niche. And it did work with up to 50bit range (1e15).
Environment (please complete the following information):
- linux
- docker version 20.10.17 (including compose)
- ES v7.16.1
Activity
michaelkirk commentedon Jul 21, 2023
Just a wild guess, but maybe related to javascript's MAX_SAFE_INTEGER=2^53-1