Support Geospatial Data Types (GEO)

[ddupg]

Description

The storage and analysis of geospatial data represent a significant demand, with a robust ecosystem already evolving around these needs. Projects like Iceberg and Parquet have introduced native support for geospatial types. Given this trend, should we consider adding native support for geospatial types in Lance as well?

I'm thinking there may be 4 main categories of implementation:

• Geospatial types: discovered repo geoarrow-rs, which is a Rust implementation of the GeoArrow specification. Perhaps we can draw inspiration from it to implement our own solution, or directly rely on it.
• Geospatial expressions: geoarrow-rs integrates with DataFusion , which aligns well with Lance’s selection and serves as a valuable reference.
• Geospatial index: Lance’s point lookup performance is a major strength. Perhaps we should also consider implementing geospatial type indexing to accelerate queries.
• Engine Integration: the integration of Spark, Daft engines may be necessary.

I created a simple demo based on geoarrow-rs to store geospatial data in LanceDB, then perform SQL queries and spatial computations on Lance Table.

[jackye1995]

I created a simple demo based on geoarrow-rs to store geospatial data in LanceDB, then perform SQL queries and spatial computations on Lance Table.

Wow this is super nice! I think it makes sense to directly integrate with geoarrow-rs (I am actually surprised it is not a part of the official Arrow project. Is there an equivalent one under official Arrow?). Even if we integrate with it, it does not prevent us from adding additional components in Lance like (1) easier API to define Geo types, (2) adding Geo index types, (3) developing better Geo encoding to store the data type (if that makes sense)

@westonpace @wjones127 what do you think?

[jackye1995]

@ddupg regrading this thread, I think we should probably add a user guide about using the geo data types. what do you think?

[kylebarron]

I think geospatial is niche enough that core Arrow maintainers don't want to accept responsibility for geospatial Arrow support. There will be minimal support in parquet (under a feature flag) for the new geospatial logical type: apache/arrow-rs#8222

If there becomes a larger community around geo + rust + arrow, we could think about putting geoarrow-rs and related projects under open governance somewhere.

[kylebarron]

Also, feel free to create an issue/discussion in geoarrow-rs if you have feedback on API/documentation/etc

[jackye1995]

we could think about putting geoarrow-rs and related projects under open governance somewhere.

Could you elaborate more on that? I thought geoarrow-rs is basically that open governance project of geo types in Arrow as an extension?

[kylebarron]

Good question!

Arrow is an Apache project under Apache open governance. The Arrow format defines a metadata location for user-defined extension types. It defines some Canonical Extension Types like JSON, UUID, and tensors, that are managed as part of the Arrow spec.

But as of now, geospatial support is considered a "community extension type". GeoArrow as a specification has really just been a collaboration between @paleolimbot, @jorisvandenbossche and myself. There's no "official" open governance yet for that specification; we aren't a part of Apache officially.

Then in terms of implementations, the three of us have push access to repositories under the geoarrow organization, but there's no official open governance process for the repos. I.e. there's no voting thread for making a new release.

Dewey has been the primary author of geoarrow-c and I've been the primary author of geoarrow-rs, though lately there's been a few other budding contributors to geoarrow-rs.

Does that answer your question?

[jackye1995]

Geospatial index: Lance’s point lookup performance is a major strength. Perhaps we should also consider implementing geospatial type indexing to accelerate queries.

I think this is probably the main focus we should put in Lance, agree that we should leverage the random access strength to see if we can reuse the current indexes, or develop additional Geo-specific indexes that can accelerate the queries. @ddupg Are there any potential options you are considering for the Geo index?

[ddupg]

I think this is probably the main focus we should put in Lance

Agree with that.

if we can reuse the current indexes

I'm guessing that currently available indexes, like B-Tree, may not be valuable for geospatial data. Developing additional geo-index would be the appropriate way.

Are there any potential options you are considering for the Geo index?

I found a couple of repos that might be valuable for us, but I haven't looked into them any further yet

• geo-index, an RTree and KDTree implementation, which uses an immutable indexing implementation that results in faster access and less space consumption.
• rstar, a flexible, n-dimensional r*-tree implementation for the Rust ecosystem, suitable for use as a spatial index.

[jackye1995]

The geo-index repo provides a list of other projects, I will just put here for easy reference:

rstar: a dynamic RTree implementation.
kdtree: a dynamic KDTree implementation.
kdbush: a port of kdbush but does not strive for FFI-compatibility.
static_aabb2d_index: a port of flatbush but does not strive for FFI-compatibility.
static-bushes: a port of flatbush and kdbush but does not strive for FFI-compatibility.

[jackye1995]

Just from my very shallow knowledge of the geo indexes, it seems like the immutable indexes could work well with Lance, since our indexes are mostly copy-on-write. Even if we train an index with new data, we load the existing index into memory and then do the training and write back. Having a dynamic index structure is not a high priority. If being immutable makes it faster, it seems pretty ideal.

[jaystarshot]

@ddupg @jackye1995 @westonpace I spent some time using https://github.com/georust/rstar and supported GEO index in lance in a small POC

Commit

Sample ‎test_geoarrow_geo_index.py)

🌍 Testing GeoArrow Point Geo Index in Lance
==================================================
✅ Cleaned up existing dataset: test_geoarrow_geo

🔵 Step 1: Creating GeoArrow Point data
✅ Created GeoArrow Point data
📊 Table schema:
id: int64
city: string
start_location: extension<geoarrow.point<PointType>>
population: int64
📍 Point column type: extension<geoarrow.point<PointType>>
📍 Point column metadata: None

🔵 Step 2: Writing to Lance dataset
✅ Successfully wrote GeoArrow data to Lance dataset
📊 Dataset has 3 rows
📊 Dataset schema:
id: int64
city: string
start_location: extension<geoarrow.point<PointType>>
population: int64

🔵 Step 3: Creating geo index on GeoArrow Point column
🌍 is_suitable_for_geo_index: field.name = start_location
🌍 is_suitable_for_geo_index: field.logical_type = LogicalType("struct")
🌍 is_suitable_for_geo_index: field.metadata = {"ARROW:extension:name": "geoarrow.point", "ARROW:extension:metadata": "{}"}
🌍 is_suitable_for_geo_index: found extension: geoarrow.point
🌍 is_suitable_for_geo_index: GeoArrow Point detected!
🌍 Starting GEO INDEX TRAINING
🌍 Processing batch with 3 rows
🌍 Extracted 3 bounding boxes
🌍 Building R-tree with 3 entries
🌍 Saving R-tree entries using binary serialization
🌍 R-tree data saved to IndexStore successfully using binary serialization
🌍 R-tree data saved to IndexStore successfully using binary serialization
🌍 GEO INDEX TRAINING COMPLETE - Index saved successfully
✅ Successfully created geo index
🌍 Loading RTreeIndex from IndexStore (GeoIndex trait)
🌍 Loading RTreeIndex from IndexStore
🌍 Loaded 3 R-tree entries using binary deserialization
📊 Available indexes:
  - {'name': 'start_location_idx', 'type': 'Geo', 'uuid': '6d14bd90-aa11-40dc-a0c5-0a2f98dfa83e', 'fields': ['start_location'], 'version': 1, 'fragment_ids': {0}}

🔵 Step 4: Testing st_intersects spatial query

📋 Running EXPLAIN ANALYZE...
🌍 GeoQueryParser::new created for index: start_location_idx
🌍 GeoQueryParser::visit_scalar_function: func = st_intersects, args.len() = 2
🌍   Index: start_location_idx, Column: start_location
🌍   Column: start_location, Index: start_location_idx
🌍   Data type: Struct([Field { name: "x", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "y", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }])
🌍   Arg[0]: Column(Column { relation: None, name: "start_location" })
🌍   Arg[1]: Literal(Utf8("BBOX(-125,30,-115,45)"), None)
🌍   Processing ST_Intersects
🌍   Extracting bounding box from arg[1]
🌍     extract_bounding_box_from_expr: Literal(Utf8("BBOX(-125,30,-115,45)"), None)
🌍     Found literal value: Utf8("BBOX(-125,30,-115,45)")
🌍       Parsing encoded BBOX coordinates: -125,30,-115,45
🌍       ✅ Parsed encoded BBOX: BoundingBox { min_x: -125.0, min_y: 30.0, max_x: -115.0, max_y: 45.0 }
🌍     ✅ Extracted bounding box from encoded string: BoundingBox { min_x: -125.0, min_y: 30.0, max_x: -115.0, max_y: 45.0 }
🌍   ✅ Created ST_Intersects query: Intersects(BoundingBox { min_x: -125.0, min_y: 30.0, max_x: -115.0, max_y: 45.0 })
🌍 GEO INDEX QUERY: Intersects(BoundingBox { min_x: -125.0, min_y: 30.0, max_x: -115.0, max_y: 45.0 })
🌍 R-tree size: 3 entries
🌍 INTERSECTS query: bbox=BoundingBox { min_x: -125.0, min_y: 30.0, max_x: -115.0, max_y: 45.0 }
🌍 Query AABB: AABB { lower: [-125.0, 30.0], upper: [-115.0, 45.0] }
🌍 Found 2 intersecting entries:
🌍   Entry 0: row_id=1, bbox=BoundingBox { min_x: -118.2437, min_y: 34.0522, max_x: -118.2437, max_y: 34.0522 }
🌍   Entry 1: row_id=0, bbox=BoundingBox { min_x: -122.4194, min_y: 37.7749, max_x: -122.4194, max_y: 37.7749 }
🌍 INTERSECTS result: 2 matching row IDs: [1, 0]
🔍 EXPLAIN ANALYZE Result:
Schema: plan_type: string not null
plan: string not null
Rows: 1
📊 plan_type: Plan with Metrics
📊 plan: LanceRead: uri=Users/jay.narale/work/Uber/lance/python/test_geoarrow_geo/data, projection=[id, city, population], num_fragments=1, range_before=None, range_after=None, row_id=false, row_addr=false, full_filter=st_intersects(start_location, Utf8("BBOX(-125,30,-115,45)")), refine_filter=--, metrics=[output_rows=2, elapsed_compute=591.499µs, bytes_read=72, fragments_scanned=1, iops=4, ranges_scanned=1, requests=4, rows_scanned=2, task_wait_time=591.249µs]
  RepartitionExec: partitioning=RoundRobinBatch(10), input_partitions=1, metrics=[fetch_time=194.333µs, repartition_time=1ns, send_time=6.967µs]
    ScalarIndexQuery: query=[ST_Intersects(start_location, BBOX(-125, 30, -115, 45))]@start_location_idx, metrics=[output_rows=2, elapsed_compute=185.917µs, index_comparisons=0, indices_loaded=0, output_batches=1, parts_loaded=0, search_time=117.125µs, ser_time=46.708µs]

⚠️  Geo index was not detected in execution plan
📋 Full plan: Plan with Metrics

📋 Running actual query...
🌍 GeoQueryParser::new created for index: start_location_idx
🌍 GeoQueryParser::visit_scalar_function: func = st_intersects, args.len() = 2
🌍   Index: start_location_idx, Column: start_location
🌍   Column: start_location, Index: start_location_idx
🌍   Data type: Struct([Field { name: "x", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "y", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }])
🌍   Arg[0]: Column(Column { relation: None, name: "start_location" })
🌍   Arg[1]: Literal(Utf8("BBOX(-125,30,-115,45)"), None)
🌍   Processing ST_Intersects
🌍   Extracting bounding box from arg[1]
🌍     extract_bounding_box_from_expr: Literal(Utf8("BBOX(-125,30,-115,45)"), None)
🌍     Found literal value: Utf8("BBOX(-125,30,-115,45)")
🌍       Parsing encoded BBOX coordinates: -125,30,-115,45
🌍       ✅ Parsed encoded BBOX: BoundingBox { min_x: -125.0, min_y: 30.0, max_x: -115.0, max_y: 45.0 }
🌍     ✅ Extracted bounding box from encoded string: BoundingBox { min_x: -125.0, min_y: 30.0, max_x: -115.0, max_y: 45.0 }
🌍   ✅ Created ST_Intersects query: Intersects(BoundingBox { min_x: -125.0, min_y: 30.0, max_x: -115.0, max_y: 45.0 })
🌍 GEO INDEX QUERY: Intersects(BoundingBox { min_x: -125.0, min_y: 30.0, max_x: -115.0, max_y: 45.0 })
🌍 R-tree size: 3 entries
🌍 INTERSECTS query: bbox=BoundingBox { min_x: -125.0, min_y: 30.0, max_x: -115.0, max_y: 45.0 }
🌍 Query AABB: AABB { lower: [-125.0, 30.0], upper: [-115.0, 45.0] }
🌍 Found 2 intersecting entries:
🌍   Entry 0: row_id=1, bbox=BoundingBox { min_x: -118.2437, min_y: 34.0522, max_x: -118.2437, max_y: 34.0522 }
🌍   Entry 1: row_id=0, bbox=BoundingBox { min_x: -122.4194, min_y: 37.7749, max_x: -122.4194, max_y: 37.7749 }
🌍 INTERSECTS result: 2 matching row IDs: [1, 0]
✅ Query Results:
📊 Schema: id: int64
city: string
population: int64
📊 Number of rows: 2
📍 Row 0: {'id': 1, 'city': 'San Francisco', 'population': 883305}
📍 Row 1: {'id': 2, 'city': 'Los Angeles', 'population': 3898747}

✅ Found 2 cities with st_intersects: ['San Francisco', 'Los Angeles']

🔵 Step 5: Verifying index files
📂 Index directories: 1
📂 Files in test_geoarrow_geo/_indices/6d14bd90-aa11-40dc-a0c5-0a2f98dfa83e:
  - page_lookup.lance (290 bytes)
  - page_data.lance (629 bytes)

🎉 Test completed!
==================================================

Main aspects,

1. Add a new index type and support loading and storing to disk
2. Support SpatialQuery (new type of queries) which can query the geo index
3. Add Functions st_within and bbox so that user queries can be translated and intercepted into the SpatialQuery on the index.

New to lance so any comments are appreciated.
Thanks!

[do-me]

Great work, super excited to see this merged in LanceDB at some point!
Is there an open PR or something you could reference too? Your links point to the official LanceDB repo where I get this confusing message:
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

[jackye1995]

@jaystarshot this is amazing! I did some brief review on the shared commit, as @do-me suggested above, I also think it would be great if you can put up an actual PR so that we can all more properly review the details! Would you be up for that?

[jaystarshot]

Thanks! Yes that commit was a part of my fork. I created a draft PR #4632

[jackye1995]

Sounds good, I also moved the comments there, let's get this in! 🚀 🚀

[jaystarshot]

Thanks

[hpvd]

maybe to learn from:

this is the blogpost to
Geoparquet 2.0: Regular Parquet files utilizing native GEOMETRY and GEOGRAPHY data types
https://cloudnativegeo.org/blog/2025/02/geoparquet-2.0-going-native/

this is the first draft attempt to geoparquet 2.0
opengeospatial/geoparquet@58b433a

and this is the last final spec 1.1
https://geoparquet.org/releases/v1.1.0/

and nice write up with comparison table and FAQs about "parquet with geometry type" is not "geoparquet"
https://rednegra.net/blog/20250925-parquet-with-geometry-type-is-not-geoparquet/

[jackye1995]

thanks for sharing!

As the two standards are orthogonal, you can combine them, with only one caveat: the columns must be encoded as WKB.

@ddupg @jaystarshot just for my personal understanding, we are storing the GeoArrow with native encoding, rather than using KWB right?

[jaystarshot]

for points geo arrow stores them in an arrow struct{x,y} with extra arrow extension metadata. Its not a native encoding. The extra arrow extension metadata allow arrow readers etc to read them with the extension metadata

[kylebarron]

The arrow struct is what GeoParquet refers to as the "native" encoding