docs: add delta lake documentation

[martin-g]

238: To review by AI

---

Note

Adds a new SedonaDB + Delta Lake example (read/write, WKT conversion, spatial querying) and reorganizes docs nav under an Examples section with a Delta Lake page.

• Docs:
  • New example docs/delta-lake.md (+ companion notebook docs/delta-lake.ipynb) showing:
    • Writing SedonaDB DataFrame to Delta Lake (geometry to WKT).
    • Reading Delta table via DeltaTable -> PyArrow -> SedonaDB.
    • Converting geometry_wkt back to geometry and running spatial filters.
• Site navigation (mkdocs.yml):
  • Add Examples section and link new Delta Lake page; regroup existing example pages under it; keep Contributors Guide under SedonaDB.

^{Written by Cursor Bugbot for commit 9db183e. This will update automatically on new commits. Configure here.}

[coderabbitai]

Walkthrough

Introduces SedonaDB Delta Lake integration documentation through a Jupyter notebook and markdown guide demonstrating end-to-end workflows including installation, reading GeoParquet data, writing to Delta Lake tables with geometry-to-WKT conversion, and spatial filtering. Reorganizes mkdocs.yml navigation by creating an Examples subsection under SedonaDB API.

Changes

Cohort / File(s)	Summary
Delta Lake Documentation docs/delta-lake.ipynb, docs/delta-lake.md	Introduces Jupyter notebook and markdown guide demonstrating SedonaDB-Delta Lake workflows including reading GeoParquet, writing/reading Delta tables, geometry-WKT conversion, and spatial queries.
Documentation Navigation mkdocs.yml	Reorganizes navigation structure by creating Examples subsection under SedonaDB API, moving GeoPandas interop, Overture Examples, CRS Examples, Delta Lake, Parquet, and SQL documentation entries. Reorders Contributors Guide position.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch pr-238-2025-10-28-20-19-06

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

docs/delta-lake.md (1)

71-94: Consider using fenced code blocks for output displays.

The output displays are formatted as indented code blocks (4 spaces). For better consistency with the fenced code blocks used for Python examples, consider wrapping these outputs in fenced code blocks with a language identifier (e.g., text or no identifier).

Example for lines 71-94:

-    ┌─────────────────────────────┬───────────────┬────────────────────────────────────────────────────┐
-    │             name            ┆   continent   ┆                    geometry_wkt                    │
+```text
+┌─────────────────────────────────┬───────────────┬────────────────────────────────────────────────────┐
+│             name            ┆   continent   ┆                    geometry_wkt                    │
+...
+└─────────────────────────────┴───────────────┴────────────────────────────────────────────────────┘
+```

This applies to output blocks at lines 71-94, 114-137, 150-153, and 179-202.

Also applies to: 114-137, 150-153, 179-202

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1c5731f and 9db183e.

📒 Files selected for processing (3)

• docs/delta-lake.ipynb (1 hunks)
• docs/delta-lake.md (1 hunks)
• mkdocs.yml (1 hunks)

🧰 Additional context used

🪛 markdownlint-cli2 (0.18.1)

docs/delta-lake.md

71-71: Code block style
Expected: fenced; Actual: indented

(MD046, code-block-style)

---

114-114: Code block style
Expected: fenced; Actual: indented

(MD046, code-block-style)

---

150-150: Code block style
Expected: fenced; Actual: indented

(MD046, code-block-style)

---

179-179: Code block style
Expected: fenced; Actual: indented

(MD046, code-block-style)

🪛 Ruff (0.14.2)

docs/delta-lake.ipynb

17-17: Probable insecure usage of temporary file or directory: "/tmp/delta_with_wkt"

(S108)

---

38-38: Found useless expression. Either assign it to a variable or remove it.

(B018)

🔇 Additional comments (3)

mkdocs.yml (1)

48-55: LGTM! Navigation structure improvement.

The reorganization of documentation under the new "Examples" subsection improves discoverability and logical grouping. The Delta Lake documentation is appropriately placed alongside other integration examples.

docs/delta-lake.ipynb (2)

61-73: LGTM! Temporary directory usage is appropriate for examples.

The use of /tmp/delta_with_wkt is flagged by static analysis (S108) as potentially insecure, but this is acceptable for documentation and example code. For production use, users should implement proper temporary directory handling with appropriate permissions.

---

216-218: LGTM! Standard Jupyter notebook display pattern.

The res.schema expression is flagged by Ruff (B018) as useless, but this is standard Jupyter notebook practice. The final expression in a cell is automatically displayed as output, which is the intended behavior here to show users the DataFrame schema.

[martin-augment]

71-94: Consider using fenced code blocks for output displays.

The output displays are formatted as indented code blocks (4 spaces). For better consistency with the fenced code blocks used for Python examples, consider wrapping these outputs in fenced code blocks with a language identifier (e.g., text or no identifier).

value:valid-but-wont-fix; category:documentation; feedback:The current way renders very well (see at https://github.com/MrPowers/sedona-db/blob/9db183ecb6d57031c8cc34d8ab8aa83d641d8884/docs/delta-lake.md). Wrapping it in text code snippet does not change it. Using Markdown tables (https://www.codecademy.com/resources/docs/markdown/tables) also does not look better.

[martin-augment]

augment review

[martin-augment]

cursor review

[cursor]

✅ Bugbot reviewed your changes and found no bugs!

---

[augmentcode]

Review completed. 2 suggestions posted.

[augmentcode]

The sentence mentions “cells in this notebook,” but this page is a Markdown doc; this reads like a copy from the notebook and could confuse readers. Consider rephrasing to refer to the example/page rather than a notebook.

🤖 React with 👍 or 👎 to let us know if the comment was useful.

[martin-augment]

value:useful; category:documentation; feedback:The AI reviewer is correct! This is a copy/paste error from the IPython notebook. The text should be updated to :.. to run the code snippets in this document`.

[augmentcode]

The description says “countries in the western half of South America,” but the query (as shown in the output) includes Panama (North America), since only a spatial predicate is applied. This seems inconsistent with the text and may mislead readers.

🤖 React with 👍 or 👎 to let us know if the comment was useful.

[martin-augment]

value:useful; category:bug; feedback:The AI reviewer is correct that the used polygon region returns too much. It should be edited to return only countries from South America, west part.

[claude]

Code Review - PR #6: Add Delta Lake Documentation

Thank you for this contribution! This is a valuable addition that demonstrates SedonaDB's interoperability with Delta Lake. Below is my detailed review.

---

✅ Strengths

1. Clear Documentation Structure: The documentation follows a logical flow (read → convert → write → read back → query), making it easy for users to understand the Delta Lake integration workflow.

2. Consistent with Existing Style: The documentation matches the style of other examples in the repository (e.g., working-with-parquet-files.md, overture-examples.md) with proper Apache license headers.

3. Good Navigation Organization: Moving examples into a dedicated "Examples" section in mkdocs.yml improves discoverability and site structure.

4. Practical Use Case: The WKT conversion approach is a practical workaround for Delta Lake's lack of native geometry support.

---

📝 Code Quality & Best Practices

Positive:

• The code examples are functional and demonstrate the complete workflow
• Proper use of SedonaDB APIs (to_view(), sql(), to_pandas())
• Good use of markdown sections to break up content

Suggestions for Improvement:

1. Hardcoded Paths (docs/delta-lake.ipynb:70, docs/delta-lake.md:70)

   table_path = "/tmp/delta_with_wkt"

   • Using /tmp/ may not work on Windows systems
   • Consider using tempfile.mkdtemp() or documenting as user-configurable
   • Example: table_path = "delta_with_wkt" # or provide your own path

2. View Overwrite Behavior (docs/delta-lake.ipynb:66, docs/delta-lake.md:51)

   countries.to_view("countries", True)

   • The True parameter for overwriting should be explained with a comment
   • Suggested: countries.to_view("countries", replace=True) # Replace if exists

3. Inconsistent View Overwrite Usage

   • Line 66: countries.to_view("countries", True) - with overwrite
   • Line 99: df.to_view("my_table", True) - with overwrite
   • Consider explaining why overwrite is needed (for notebook re-execution)

---

🐛 Potential Issues

1. Missing Error Handling: No error handling for network issues when downloading the GeoParquet file or Delta Lake operations. Consider adding a note about potential failures.

2. Schema Output in Markdown (docs/delta-lake.md:123-126)

   res.schema

   The output is shown but without markdown formatting around it. Consider using a code block for better rendering.

3. Dependency Management: The notebook requires pip install deltalake but doesn't mention version constraints. Consider adding version information if there are compatibility requirements.

---

⚡ Performance Considerations

1. Repeated WKT Conversions (docs/delta-lake.md:138-148)

   ST_GeomFromWKT(geometry_wkt)  # Called twice in the same query

   • The query converts WKT to geometry twice for the same column
   • Suggested optimization:

   WITH geom_data AS (
     SELECT name, continent, ST_GeomFromWKT(geometry_wkt) as geom
     FROM my_table
   )
   SELECT * FROM geom_data
   WHERE ST_Intersects(
     geom,
     ST_GeomFromWKT('POLYGON((-81 12, -58 12, -58 -55, -81 -55, -81 12))')
   )

2. View Caching: The documentation could mention that views are in-memory references, so users understand the performance implications for large Delta tables.

---

🔒 Security Concerns

No significant security issues identified. The examples use:

• Public data sources (safe)
• Local file operations (standard)
• No credential handling or external connections

---

🧪 Test Coverage

Observation: This PR only adds documentation (no code changes), so traditional test coverage doesn't apply.

Recommendations:

1. Consider running the notebook as part of CI to ensure examples remain functional
2. The example could benefit from a section showing error handling for common issues
3. Add a note about testing Delta Lake operations in production environments

---

📋 Specific Suggestions

1. Add Installation Section: Include a note about installing both packages:

   pip install "apache-sedona[db]" deltalake

2. Add Performance Note: Mention that WKT conversion is lossy for certain precision scenarios and has performance overhead for large geometries.

3. Add Versioning Note: Document which Delta Lake and SedonaDB versions were tested.

4. Add Troubleshooting Section: Common issues like:

   • Write permission errors
   • Delta Lake version conflicts
   • Large geometry WKT limitations

5. mkdocs.yml Change: The reorganization is good, but consider if "Working with SQL in SedonaDB" should remain under Examples or move back to a top-level guide (as it's more of a guide than an example).

---

🎯 Minor Nitpicks

1. Notebook Cell Numbers: Execution counts are visible in the notebook (16-22). Consider clearing outputs before committing for cleaner version control.

2. Consistency: Some sections use "Now" to start paragraphs, while others don't. Consider standardizing the narrative style.

3. Code Comments: The notebook could benefit from inline comments explaining the SedonaDB-specific functions for users new to the library.

---

✨ Overall Assessment

Rating: Approve with Minor Suggestions

This is a solid documentation contribution that will be valuable for users integrating SedonaDB with Delta Lake. The code is functional and the documentation is clear. The suggestions above are mostly minor improvements and optimizations that can be addressed in follow-up work or before merging.

Priority Fixes:

1. Address the hardcoded /tmp/ path for cross-platform compatibility
2. Optimize the WKT conversion performance in the final query example
3. Add a combined installation command at the beginning

Nice-to-Have:

• Error handling examples
• Version compatibility notes
• Troubleshooting section

Great work on this addition! 🚀

[github-actions]

Issues

• python/sedonadb/src/lib.rs:22 & python/sedonadb/python/sedonadb/context.py:123 – All of the Python UDF plumbing (mod udf, sedona_scalar_udf, SedonaContext.register_udf, etc.) is removed, along with the sedonadb.udf module and its tests. That breaks existing public API for registering UDFs from Python clients and will regress current users. Please restore the UDF bridge or move the functionality to its new home before merging.
• c/sedona-geos/src/register.rs:33 & c/sedona-geos/src/lib.rs:21 – The GEOS implementations for ST_IsSimple and ST_SimplifyPreserveTopology have been dropped (their modules and registrations are gone). These SQL functions will disappear from the engine after this merge. If that’s unintended, the removal needs to be reverted; if it’s intentional, please add replacements and update docs/tests accordingly.
• c/sedona-geos/src/register.rs:60 – st_crosses and st_overlaps are appended to the registry a second time, creating duplicate entries in the returned vector. That looks like an editing error; please remove the duplicates.

[martin-augment]

Issues

• python/sedonadb/src/lib.rs:22 & python/sedonadb/python/sedonadb/context.py:123 – All of the Python UDF plumbing (mod udf, sedona_scalar_udf, SedonaContext.register_udf, etc.) is removed, along with the sedonadb.udf module and its tests. That breaks existing public API for registering UDFs from Python clients and will regress current users. Please restore the UDF bridge or move the functionality to its new home before merging.
• c/sedona-geos/src/register.rs:33 & c/sedona-geos/src/lib.rs:21 – The GEOS implementations for ST_IsSimple and ST_SimplifyPreserveTopology have been dropped (their modules and registrations are gone). These SQL functions will disappear from the engine after this merge. If that’s unintended, the removal needs to be reverted; if it’s intentional, please add replacements and update docs/tests accordingly.
• c/sedona-geos/src/register.rs:60 – st_crosses and st_overlaps are appended to the registry a second time, creating duplicate entries in the returned vector. That looks like an editing error; please remove the duplicates.

value:annoying; category:bug; feedback:The Codex AI reviewer has hallucinations. Those files are not changed in this Pull Request

[martin-augment]

• Hardcoded Paths (docs/delta-lake.ipynb:70, docs/delta-lake.md:70)

  table_path = "/tmp/delta_with_wkt"

  • Using /tmp/ may not work on Windows systems
  • Consider using tempfile.mkdtemp() or documenting as user-configurable
  • Example: table_path = "delta_with_wkt" # or provide your own path

value:valid-but-wont-fix; category:documentation; feedback: The Claude AI reviewer is correct that the file path is hardcoded and may not work well for Windows users for example, but this is just a simple howto document explaining how to use Deltalake with SedonaDB. It is clear enough that the path is just an example and the user may provide another value.

[martin-augment]

View Overwrite Behavior (docs/delta-lake.ipynb:66, docs/delta-lake.md:51)

countries.to_view("countries", True)

• The True parameter for overwriting should be explained with a comment
• Suggested: countries.to_view("countries", replace=True) # Replace if exists

value:good-to-have; category:documentation; feedback: The Claude AI reviewer is correct using a boolean parameter that is not named is usually confusing and the reader will need to consult the method documentation to understand what this parameter does. Both adding "replace=" and trailing comment would make it more clear!