feat[python]: support full read/write from object storage by a10y · Pull Request #6022 · vortex-data/vortex

a10y · 2026-01-19T02:37:42Z

This adds the ability to write in PyVortex directly to object storage, along with more ergonomic configuration of object store credentials for the major PyVortex read/write paths: vx.open, vx.io.read, and vx.io.write, as well as VortexWriteOptions.write.

This is using the https://docs.rs/pyo3-object_store/latest/pyo3_object_store/ crate, which is what runs the obstore Python library.

Notably, we can't reuse obstore and all of its types directly. This is b/c of the way it works, you're required to actually link in pyo3-object_store and then expose it as a new module within your Python bundle.

There are some other examples of this pattern:

Sadly there's no nicely distributed types package, so I need to copy all of the pyi files here with their original MIT license.

I've added a Python unit test to demonstrate using the new object store builders with simple local file system object store

Resolves #5673

codecov · 2026-01-19T02:56:36Z

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 82.93%. Comparing base (12ba988) to head (cc7a06b).
⚠️ Report is 11 commits behind head on develop.

Files with missing lines	Patch %	Lines
vortex-io/src/write.rs	0.00%	1 Missing ⚠️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codspeed-hq · 2026-01-22T15:59:00Z

CodSpeed Performance Report

Merging this PR will degrade performance by 31.79%

_{Comparing aduffy/pyvortex-obstore (7785aca) with develop (68130ce)¹}

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

Summary

⚡ 3 improved benchmarks
❌ 7 regressed benchmarks
✅ 1151 untouched benchmarks
🆕 18 new benchmarks
⏩ 1323 skipped benchmarks²

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
🆕	WallTime	`10M_90pct[10000000]`	N/A	371.9 µs	N/A
🆕	WallTime	`10M_90pct[10000000]`	N/A	365 µs	N/A
🆕	WallTime	`1M_10pct[100000]`	N/A	45.9 µs	N/A
🆕	WallTime	`10M_90pct[10000000]`	N/A	198.2 µs	N/A
🆕	WallTime	`1M_90pct[1000000]`	N/A	57.1 µs	N/A
🆕	WallTime	`10M_50pct[5000000]`	N/A	281.5 µs	N/A
🆕	WallTime	`1M_50pct[500000]`	N/A	49.5 µs	N/A
🆕	WallTime	`1M_10pct[100000]`	N/A	45.8 µs	N/A
🆕	WallTime	`10M_50pct[5000000]`	N/A	304.9 µs	N/A
🆕	WallTime	`10M_10pct[1000000]`	N/A	219.2 µs	N/A
🆕	WallTime	`1M_50pct[500000]`	N/A	22.4 µs	N/A
🆕	WallTime	`1M_10pct[100000]`	N/A	25.6 µs	N/A
🆕	WallTime	`1M_50pct[500000]`	N/A	50 µs	N/A
🆕	WallTime	`10M_10pct[1000000]`	N/A	133.7 µs	N/A
🆕	WallTime	`10M_10pct[1000000]`	N/A	219.3 µs	N/A
🆕	WallTime	`1M_90pct[1000000]`	N/A	62.7 µs	N/A
🆕	WallTime	`10M_50pct[5000000]`	N/A	157.9 µs	N/A
🆕	WallTime	`1M_90pct[1000000]`	N/A	28.9 µs	N/A
❌	Simulation	`canonical_into_non_nullable[(10000, 1, 0.0)]`	25.7 µs	28.8 µs	-10.83%
❌	Simulation	`canonical_into_non_nullable[(10000, 10, 0.0)]`	195.5 µs	286.6 µs	-31.79%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

No successful run was found on develop (ed75f70) during the generation of this report, so 68130ce was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
1323 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a10y · 2026-01-28T15:18:46Z

Just pushed a commit with a modified version of the ObjectStoreRegistry that resolves configurations out of the environment in a more consistent way.

This is related to apache/arrow-rs-object-store#529 which we stumbled upon last week

a10y · 2026-01-28T15:30:33Z

vortex-python/src/object_store/registry.rs

+            // NOTE(aduffy): object_store doesn't let us downcast stores, the only way to verify
+            //  that a configuration was added was to validate that it ends up in the Debug
+            //  output :/
+            let mut debug_str = String::new();
+            write!(&mut debug_str, "{store:?}").unwrap();
+
+            assert!(debug_str.contains("us-east-3"));


this is pretty sad but i don't have a better idea

AdamGS · 2026-01-28T17:15:56Z

docs/api/python/store.rst

@@ -0,0 +1,17 @@
+Object Store support


this file renders in a a weird way (see image). I would expect the config classes to appear here at least.

Yea I'm so bad with Python/Sphinx tooling. I might just try and get Claude to clean this up in a FLUP if that's ok

AdamGS

Looks great to me, aside from the docs rendering thing (do we also want to put some example there too?)

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

This adds the ability to write in PyVortex directly to object storage, along with more ergonomic configuration of object store credentials for the major PyVortex read/write paths: `vx.open`, `vx.io.read`, and `vx.io.write`, as well as `VortexWriteOptions.write`. This is using the https://docs.rs/pyo3-object_store/latest/pyo3_object_store/ crate, which is what runs the [obstore](https://developmentseed.org/obstore/latest/) Python library. Notably, we can't reuse `obstore` and all of its types directly. This is b/c of the way it works, you're required to actually link in pyo3-object_store and then expose it as a new module **within your Python bundle**. There are some other examples of this pattern: * [nutpie](https://github.com/pymc-devs/nutpie/blob/adb83e24235773d7ff97f86e1858ee9d31db0936/src/wrapper.rs#L1624-L1625) * [geoarrow](https://github.com/geoarrow/geoarrow-rs/blob/main/python/geoarrow-io/src/lib.rs#L46-L48) Sadly there's no nicely distributed types package, so I need to copy all of the pyi files here with their original MIT license. I've added a Python unit test to demonstrate using the new object store builders with simple local file system object store Resolves #5673 --------- Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a10y added the changelog/feature A new feature label Jan 19, 2026

a10y requested review from danking and gatesn January 19, 2026 02:48

a10y force-pushed the aduffy/pyvortex-obstore branch from 64382ba to 77ac62a Compare January 22, 2026 15:53

a10y added the feature A feature request label Jan 22, 2026

feat[python]: support full read/write from object storage

190d169

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a10y force-pushed the aduffy/pyvortex-obstore branch from 35ed5b7 to 190d169 Compare January 22, 2026 17:22

fixup

ac3aff7

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a10y force-pushed the aduffy/pyvortex-obstore branch from b4e95cb to ac3aff7 Compare January 22, 2026 18:31

a10y added 4 commits January 22, 2026 14:15

fix more

70d3743

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

this?

2d94025

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

add SPDX headers

30e5b32

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

one more

c5e7a33

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a10y commented Jan 28, 2026

View reviewed changes

AdamGS reviewed Jan 28, 2026

View reviewed changes

AdamGS approved these changes Jan 28, 2026

View reviewed changes

add ObjectStoreRegistry that normalizes env vars for resolve

7785aca

Signed-off-by: Andrew Duffy <andrew@a10y.dev>

a10y force-pushed the aduffy/pyvortex-obstore branch from 26a73d3 to 7785aca Compare January 29, 2026 15:19

a10y enabled auto-merge (squash) January 29, 2026 15:20

a10y merged commit 0636d8d into develop Jan 29, 2026
44 of 46 checks passed

a10y deleted the aduffy/pyvortex-obstore branch January 29, 2026 15:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat[python]: support full read/write from object storage#6022

feat[python]: support full read/write from object storage#6022
a10y merged 7 commits intodevelopfrom
aduffy/pyvortex-obstore

a10y commented Jan 19, 2026 •

edited

Loading

Uh oh!

codecov bot commented Jan 19, 2026

Uh oh!

codspeed-hq bot commented Jan 22, 2026 •

edited

Loading

Uh oh!

a10y commented Jan 28, 2026

Uh oh!

a10y Jan 28, 2026

Uh oh!

AdamGS Jan 28, 2026

Uh oh!

a10y Jan 29, 2026

Uh oh!

AdamGS left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

a10y commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 19, 2026

Codecov Report

Uh oh!

codspeed-hq bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging this PR will degrade performance by 31.79%

Summary

Performance Changes

Footnotes

Uh oh!

a10y commented Jan 28, 2026

Uh oh!

a10y Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

AdamGS Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

a10y Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

AdamGS left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

a10y commented Jan 19, 2026 •

edited

Loading

codspeed-hq bot commented Jan 22, 2026 •

edited

Loading

AdamGS left a comment •

edited

Loading