botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidd

[robmarkcole]

🐛 Bug

I've recently started getting

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidd

I think the issue is boto3 not using the correct profile. Note we use sso and I do see Loading cached SSO token for eda-sso but apparently this is not used

To Reproduce

Use any streaming dataset

Expected behavior

No issues until recently, not sure if a bucket permission might have changes (devops say no)

Full trace

Loading cached SSO token for eda-sso
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
Traceback (most recent call last):
  File "/teamspace/studios/this_studio/hawkeye/lightning_ai/cli.py", line 163, in <module>
    run(obj={})
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/teamspace/studios/this_studio/hawkeye/lightning_ai/cli.py", line 141, in train
    run_trainer(lightning_cli.model, lightning_cli.trainer, lightning_cli.datamodule)
  File "/teamspace/studios/this_studio/hawkeye/lightning_ai/cli.py", line 118, in run_trainer
    datamodule.setup("fit")
  File "/teamspace/studios/this_studio/hawkeye/common/datamodules/streaming.py", line 88, in setup
    self.train_dataset = self.create_dataset(subset="train")
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/teamspace/studios/this_studio/hawkeye/common/datamodules/streaming.py", line 114, in create_dataset
    return self.dataset_class(**dataset_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/teamspace/studios/this_studio/hawkeye/common/datasets/streaming.py", line 228, in __init__
    super().__init__(*args, transforms=transforms, band_indices=band_indices, **kwargs)
  File "/teamspace/studios/this_studio/hawkeye/common/datasets/streaming.py", line 46, in __init__
    super().__init__(*args, **kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/streaming/dataset.py", line 123, in __init__
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/streaming/dataset.py", line 123, in __init__
    self.subsampled_files, self.region_of_interest = subsample_streaming_dataset(
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/utilities/dataset_utilities.py", line 45, in subsample_streaming_dataset
    cache_path = _try_create_cache_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/utilities/dataset_utilities.py", line 220, in _try_create_cache_dir
    updated_at = _read_updated_at(resolved_input_dir, storage_options, index_path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/utilities/dataset_utilities.py", line 164, in _read_updated_at
    downloader.download_file(os.path.join(input_dir.url, _INDEX_FILENAME), temp_index_filepath)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/streaming/downloader.py", line 148, in download_file
    self._client.client.download_file(
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/boto3/s3/inject.py", line 192, in download_file
    return transfer.download_file(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/boto3/s3/transfer.py", line 406, in download_file
    future.result()
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/s3transfer/futures.py", line 264, in result
    raise self._exception
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/s3transfer/tasks.py", line 265, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/s3transfer/download.py", line 352, in _submit
    response = client.head_object(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

[robmarkcole]

Grok suggests it is necessary to path through the profile to use, but I don't see this supported:

        "storage_options": {
            "profile_name": "my-profile",
            "region_name": "us-east-1"
        }
    }

[deependujha]

Hi @robmarkcole, is there any issue with providing storage_options to StreamingDataset

pip install s5cmd

ds = StreamingDataset("..", storage_options= {
            "profile_name": "my-profile",
            "region_name": "us-east-1"
        }
)

If you don't want to s5cmd, you can also try setting up env variables.

export AWS_PROFILE=my-profile
export AWS_DEFAULT_REGION=us-east-1

[robmarkcole]

It is not supported, test script:

from litdata.streaming.dataset import StreamingDataset

dataset = StreamingDataset(
    input_dir="s3://my/datasets/v3/",
    storage_options= {
            "profile_name": "my-profile",
            "region_name": "us-east-1",
    }
)

# Try accessing the dataset
try:
    print("Dataset length:", len(dataset))
    print("First item:", dataset[0])
except Exception as e:
    print("Error:", e)

Result

raceback (most recent call last):
  File "/teamspace/studios/this_studio/test_litdata.py", line 3, in <module>
    dataset = StreamingDataset(
              ^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/streaming/dataset.py", line 123, in __init__
    self.subsampled_files, self.region_of_interest = subsample_streaming_dataset(
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/utilities/dataset_utilities.py", line 45, in subsample_streaming_dataset
    cache_path = _try_create_cache_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/utilities/dataset_utilities.py", line 220, in _try_create_cache_dir
    updated_at = _read_updated_at(resolved_input_dir, storage_options, index_path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/utilities/dataset_utilities.py", line 164, in _read_updated_at
    downloader.download_file(os.path.join(input_dir.url, _INDEX_FILENAME), temp_index_filepath)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/streaming/downloader.py", line 148, in download_file
    self._client.client.download_file(
    ^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/streaming/client.py", line 64, in client
    self._create_client()
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/streaming/client.py", line 42, in _create_client
    self._client = session.client(
                   ^^^^^^^^^^^^^^^
TypeError: Session.client() got an unexpected keyword argument 'profile_name'

[robmarkcole]

Tried

⚡ ~ pip install s5cmd
Collecting s5cmd
  Downloading s5cmd-0.2.0-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.7 kB)
Downloading s5cmd-0.2.0-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 114.8 MB/s eta 0:00:00
Installing collected packages: s5cmd
Successfully installed s5cmd-0.2.0
⚡ ~ python test_litdata.py
Traceback (most recent call last):
  File "/teamspace/studios/this_studio/test_litdata.py", line 3, in <module>
    dataset = StreamingDataset(
              ^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/streaming/dataset.py", line 123, in __init__
    self.subsampled_files, self.region_of_interest = subsample_streaming_dataset(
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/utilities/dataset_utilities.py", line 45, in subsample_streaming_dataset
    cache_path = _try_create_cache_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/utilities/dataset_utilities.py", line 220, in _try_create_cache_dir
    updated_at = _read_updated_at(resolved_input_dir, storage_options, index_path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/utilities/dataset_utilities.py", line 164, in _read_updated_at
    downloader.download_file(os.path.join(input_dir.url, _INDEX_FILENAME), temp_index_filepath)
  File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.11/site-packages/litdata/streaming/downloader.py", line 140, in download_file
    raise RuntimeError(error_message)
RuntimeError: Failed to execute command `s5cmd cp s3://earthdaily-pathfinders-lightningai/datasets/venus_vessels/v3/index.json /tmp/tmpfqmmjzss/index.json` (exit code: 1). This might be due to an incorrect file path, insufficient permissions, or network issues. To resolve this issue, you can either:
- Pass `storage_options` with the necessary credentials and endpoint.
- Example:
  storage_options = {
      "AWS_ACCESS_KEY_ID": "your-key",
      "AWS_SECRET_ACCESS_KEY": "your-secret",
      "S3_ENDPOINT_URL": "https://s3.example.com" (Optional if using AWS)
  }
- or disable `s5cmd` by setting `DISABLE_S5CMD=1` if `storage_options` do not work.
For further debugging, please check the command output below:
ERROR "cp s3://earthdaily-pathfinders-lightningai/datasets/venus_vessels/v3/index.json /tmp/tmpfqmmjzss/index.json": SSOProviderInvalidToken: the SSO session has expired or is invalid caused by: open /teamspace/studios/this_studio/.aws/sso/cache/4a5d0e05092840bd4f00e3700576dc7afa33992f.json: no such file or directory

Note I had just refreshed the sso with aws sso login
Seems it is checking for the wrong file

⚡ ~ ls /teamspace/studios/this_studio/.aws/sso/cache/
183b0e13ecdf927fd4fbb1ea4dbbf39e89c98e3a.json  5807a5037f6ff675c00e4ca5b7c35e8977d71ca9.json

[deependujha]

Sorry to hear this, so env variable also doesn't work?

make sure to either remove s5cmd or set DISABLE_S5CMD=1

[deependujha]

Hi, Based on s5cmd readme. try updating storage_keys to match what s5cmd expects.
https://github.com/peak/s5cmd?tab=readme-ov-file#specifying-credentials

---

So, use:

storage_options= {
            "profile": "my-profile",
            "region_name": "us-east-1",
    }

Also may be for region https://github.com/peak/s5cmd?tab=readme-ov-file#region-detection

[robmarkcole]

I tried with the env var, different error:

RuntimeError: Failed to execute command `s5cmd cp s3://earthdaily-pathfinders-lightningai/datasets/venus_vessels/v3/index.json /tmp/tmp8t4l2fn4/index.json` (exit code: 1). This might be due to an incorrect file path, insufficient permissions, or network issues. To resolve this issue, you can either:
- Pass `storage_options` with the necessary credentials and endpoint. 
- Example:
  storage_options = {
      "AWS_ACCESS_KEY_ID": "your-key",
      "AWS_SECRET_ACCESS_KEY": "your-secret",
      "S3_ENDPOINT_URL": "https://s3.example.com" (Optional if using AWS)
  }
- or disable `s5cmd` by setting `DISABLE_S5CMD=1` if `storage_options` do not work.
For further debugging, please check the command output below:
ERROR "cp s3://earthdaily-pathfinders-lightningai/datasets/venus_vessels/v3/index.json /tmp/tmp8t4l2fn4/index.json": MissingEndpoint: 'Endpoint' configuration is required for this service

Not sure s5cmd even supports SSO:

• Support AWS SSO profiles peak/s5cmd#385
• Unable to use s5cmd when using aws sso peak/s5cmd#726

[deependujha]

sorry, working on fixing s3 client creation and storage_options.

[bhimrazy]

Hi @robmarkcole, please feel free to give it a try now.
We tested it with aws sso, and it seems to be working fine with the new session_options functionality (using boto3 under the hood).

We configured SSO using:

aws configure sso

And then passed just the session option with profile_name to the dataset:

session_options = {
    "profile_name": "your-profile-name",
}

Let us know how it goes!

cc: @deependujha

[robmarkcole]

@bhimrazy I'm afraid my script still fails with botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

from litdata.streaming.dataset import StreamingDataset

dataset = StreamingDataset(
    input_dir="s3://your-bucket-name/your/prefix/path/",
    session_options={
        "profile_name": "your-profile-name",
        "region_name": "your-region",
    }
)

try:
    print("Dataset length:", len(dataset))
    print("First item:", dataset[0])
except Exception as e:
    print("Error:", e)

The following is successful:

import boto3

# Create a session using a named AWS profile
session = boto3.Session(profile_name='your-profile-name')
s3_client = session.client('s3')

bucket = 'your-bucket-name'
key = 'your/prefix/path/_INDEX_FILENAME'  # Adjust if the index file path differs

# Try checking if a specific object exists
try:
    response = s3_client.head_object(Bucket=bucket, Key=key)
    print("HeadObject succeeded:", response)
except Exception as e:
    print("Error:", e)

# Try listing objects under a given prefix
try:
    response = s3_client.list_objects_v2(Bucket=bucket, Prefix='your/prefix/path/')
    print("ListObjects succeeded:", response)
except Exception as e:
    print("Error:", e)

I wonder if the sessions is set correctly, might be debuggable with print("Profile used:", session.profile_name)