Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement end-user-facing Python lib wrapping ObjectStore #240

Merged
merged 19 commits into from
Oct 17, 2024

Conversation

kylebarron
Copy link
Owner

  • Maybe it's better to have methods on the classes instead of free functions, because it's easier to accidentally mix and match stores from multiple python packages with free functions, whereas you can't accidentally do that with class methods

This works!

from io import BytesIO

import arro3.io
import boto3
import object_store_rs

session = boto3.Session()

store = object_store_rs.store.S3Store.from_session(session, "ookla-open-data")
path = "parquet/performance/type=mobile/year=2019/quarter=1/2019-01-01_performance_mobile_tiles.parquet"
out = object_store_rs.get(store, path)

reader = arro3.io.read_parquet(BytesIO(out))
table = reader.read_all()

@kylebarron
Copy link
Owner Author

  • Maybe it's better to have methods on the classes instead of free functions, because it's easier to accidentally mix and match stores from multiple python packages with free functions, whereas you can't accidentally do that with class methods

This is true, but the issue is that in contrast to pyo3-arrow, in pyo3-object_store we're expecting the dependents to reexport the classes exported from pyo3-object_store. Therefore any Python APIs there become part of the public API, I want the public Python API of pyo3-object_store to only have constructors, because that makes the public API surface of pyo3-object_store much smaller.

So I think for now we'll have free functions from a python-facing object store package.

@kylebarron
Copy link
Owner Author

Todo:

  • Make a separation between put and put_file so that you can allow PutOptions on the former. This provides a nice PutMode enum that some people will probably want, but which PutMultipartOpts doesn't seem to have.

@kylebarron kylebarron changed the title sketch of python object-store api Implement end-user-facing Python lib wrapping ObjectStore Oct 16, 2024
@kylebarron kylebarron enabled auto-merge (squash) October 17, 2024 14:02
@kylebarron kylebarron merged commit 246c135 into main Oct 17, 2024
5 checks passed
@kylebarron kylebarron deleted the kyle/object-store-python branch October 17, 2024 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant