Add OCI as an object storage backend for StreamingData

[dkennetzoracle]

🚀 Feature

I'd like to add OCI Object Storage as an object storage backend.

Motivation

I primarily work in Oracle Cloud Infrastructure, and I'd like to leverage lit data StreamingDataset in OCI.

Pitch

I will contribute an OCI_STORAGE_AVAILABLE and OCIDownloader with test coverage.

Alternatives

Additional context

[github-actions]

Hi! thanks for your contribution!, great first issue!

[bhimrazy]

Hi @dkennetzoracle,

Thank you for creating this issue!

OCI Object Storage is S3-compatible, so you may be able to connect to it using existing S3 methods. Here’s some documentation that could help:

• Using the S3-Compatible API with Oracle Object Storage

• Example of Configuring AWS SDK for Python

  For example:

  import boto3
    
  s3 = boto3.resource(
      's3',
      aws_access_key_id="your-access-key-id",
      aws_secret_access_key="your-secret-access-key",
      region_name="us-phoenix-1",  # Replace with your region
      endpoint_url="https://<your-namespace>.compat.objectstorage.us-phoenix-1.oraclecloud.com"
  )
    
  # Print out the bucket names
  for bucket in s3.buckets.all():
      print(bucket.name)

Let us know if this approach works for your implementation.

cc: @tchaton

[dkennetzoracle]

@bhimrazy - thanks for the reply - I know how to use boto3, it is just inconvenient because I am not using aws. I'd prefer to authenticate with the provider I am using, and also use their object storage clients. We can close this. Thanks!

[bhimrazy]

Thanks for your input, @dkennetzoracle! We totally understand your preference, and happy to share that we'll soon be adding support for fsspec, which supports a variety of storage systems, including OCIFS. This will allow you to reference your buckets with oci:// instead of s3:// directly.

We truly appreciate your interest!
If you're open to it, we would be grateful for any contributions to the fsspec integration.

[dkennetzoracle]

@bhimrazy if it's in your backlog, I can take it on as I have cycles outside of work. Would be fun to work on and the project is great. Thanks!

[bhimrazy]

@dkennetzoracle, feel free to get started.
Here’s the PR you can refer to: PR Link.

It was already merged but had to be reverted due to some issues in the multi-node setting.
Also, feel free to join the Lightning AI community on Discord: Discord Invite.
Happy to chat more over there!

[dkennetzoracle]

@bhimrazy you helped me today with my litdata question! I'm dk