How do I append to a vortex file from python using io.write?

[pavanramkumar]

Just head about vortex today! My goal is to benchmark vortex for 10B+ records as an alternative to parquet and lance for ML applications. Looking to sequentially or concurrently write to a vortex file from an ETL pipeline. Currently seems like I need to have the entire dataset as an vortex array in memory to write to disk?

[a10y]

Hey @pavanramkumar for the Python SDK you're correct, that's how it currently works. We should add incremental writes to PyVortex.

From Rust, you are able to async stream writes into Vortex files though, checkout this example from our CLI which streams Arrow recordbatch out of a Parquet file directly into a Vortex file: https://github.com/vortex-data/vortex/blob/develop/vortex-tui/src/convert.rs

[pavanramkumar]

Thanks! As a non-rust dev, it seems like I can either pre-convert to parquet and then stream into vortex via CLI, or wait for a pyVortex feature?

My current datasets are all in lance. Can I hack rust to get arrow batches out of lance and write to vortex?

Would love to help make it easier with my non existent rust skills

[gatesn]

@claude can you fix PyVortex API to allow it to take an Arrow FFIArrayStream in the write API.

[gatesn]

Let's give this a go 😅

[claude]

Claude finished @gatesn's task —— View job • claude/issue-4160-20250807-2038 • Create PR ➔

---

Adding Arrow FFI streaming support to PyVortex Write API ✅

Successfully implemented Arrow FFI streaming support for the PyVortex write API to enable streaming writes without loading entire datasets into memory.

Completed Tasks ✅

• Analyze current PyVortex write API structure
• Examine Rust streaming example in vortex-tui/src/convert.rs
• Research Arrow FFI integration patterns in the codebase
• Implement FFIArrayStream support in Python bindings
• Run cargo clippy and formatting checks
• Test the implementation

Implementation Summary

The PyVortex write() API now accepts:

• PyArrow Tables: vx.io.write(pa.table(data), "file.vortex")
• PyArrow RecordBatchReaders: vx.io.write(reader, "file.vortex")
• Existing Vortex objects: Arrays and ArrayIterators (unchanged)

Key Technical Changes

File: vortex-python/src/io.rs

• Extended PyIntoArrayIterator::extract_bound() to detect PyArrow objects
• Added try_arrow_stream_to_iterator() function using ArrowArrayStreamReader::from_pyarrow_bound()
• Streams Arrow RecordBatches directly to ArrayIteratorAdapter without memory buffering
• Updated documentation with streaming examples

Benefits for Users

✅ Stream 10B+ records directly without memory constraints
✅ Zero-copy Arrow FFI integration via existing infrastructure
✅ Backwards compatible - existing code continues to work
✅ ETL pipeline friendly - perfect for data processing workflows

Usage Examples

import pyarrow as pa
import vortex as vx

# Stream a PyArrow Table
table = pa.table({'id': range(1000000), 'value': range(1000000)})
vx.io.write(table, "large_dataset.vortex")  # No memory bloat!

# Stream from RecordBatchReader
reader = pa.RecordBatchReader.from_batches(schema, batches)
vx.io.write(reader, "streamed.vortex")  # Process batches incrementally

This implementation leverages the existing Arrow FFI infrastructure (ArrowArrayStreamReader) already used elsewhere in the codebase, ensuring consistency and reliability.

---