Question: R2 example

[rgreinho]

Would it be possible to add an example to connect to R2 in the documentation please?

I keep getting errors while trying to list the objects in my bucket, so obviously I am doing something wrong, but I cannot figure out what. An example would probably help other people as well 😃

Here is my code:

from __future__ import annotations

import asyncio
import os
from typing import TYPE_CHECKING

import obstore
from obstore.store import (
    S3Store,
)


if TYPE_CHECKING:
    from obstore.store import (
        S3Store,
    )


async def r2_test():
    bucket = os.getenv("BUCKET")
    r2_account_id = os.getenv("R2_ACCOUNT_ID")
    r2_access_key_id = os.getenv("R2_ACCESS_KEY_ID")
    r2_secret_access_key = os.getenv("R2_SECRET_ACCESS_KEY")
    r2_region = os.getenv("R2_REGION")

    r2_store = S3Store(
        bucket=bucket,
        client_options={"connect_timeout": "1h"},
        config={
            "access_key_id": r2_access_key_id,
            "secret_access_key": r2_secret_access_key,
            "endpoint": f"https://{r2_account_id}.r2.cloudflarestorage.com",
            "region": r2_region,
        },
    )
    list_stream = obstore.list(r2_store, "/")
    for batch in list_stream:
        for meta in batch:
            print(f"Name: {meta['path']}, size: {meta['size']}")


if __name__ == "__main__":
    asyncio.run(r2_test())

And here is the error message I get:

obstore.exceptions.GenericError: Generic S3 error: Error performing list request: Error performing GET https://xxxxxxxxxxxxxxxxxxxxx.r2.cloudflarestorage.com/xxxxxxxxxxxxxxxxxxxxxxx?list-type=2 in 218.336667ms - Server returned non-2xx status code: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><Message>X-Amz-Security-Token</Message></Error>

[kylebarron]

I don't think you want to pass in region. I think that's only for AWS regions and that causes obstore to generate AWS credentials with a different signing mechanism.

I have a public R2 bucket I created a long time ago but rarely think about now. Looks like steps are:

1. Create API token with read or read/write access to one or more buckets
2. On the general settings of a bucket, take note of the S3 API URL. In my case it's https://f0b62eebfbdde1133378bfe3958325f6.r2.cloudflarestorage.com/kylebarron-public
3. Pass to S3Store.from_url

from obstore.store import S3Store

access_key_id = ...
secret_access_key = ...
store = S3Store.from_url(
    "https://f0b62eebfbdde1133378bfe3958325f6.r2.cloudflarestorage.com/kylebarron-public",
    access_key_id=access_key_id,
    secret_access_key=secret_access_key,
)
store.list_with_delimiter()

Or you can construct manually with endpoint

from obstore.store import S3Store

access_key_id = ...
secret_access_key = ...
bucket = "kylebarron-public"
endpoint = "https://f0b62eebfbdde1133378bfe3958325f6.r2.cloudflarestorage.com"
store = S3Store(
    bucket,
    access_key_id=access_key_id,
    secret_access_key=secret_access_key,
    endpoint=endpoint,
)
store.list_with_delimiter()

[rgreinho]

It worked like a charm! Thank you very much ❤️