S3: IRSA web identity ignored on EKS, falls back to the node instance role

[khustup2]

Version: 4.7.0 (also 4.7.2) · Linux aarch64, Python 3.12 · S3 storage

What happens

On EKS with IRSA, a pod that has AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE set is not authenticated as its ServiceAccount role. The credential chain falls through to the node instance role, and S3 is asked with that identity instead:

StorageAccessDenied: [S3] Access denied: ...
User: arn:aws:sts::123456789012:assumed-role/eks_node_groups/i-0xxxxxxxxxxxxxxxx
is not authorized to perform: s3:ListBucket on <BUCKET>

The pod's own role is allowed on that bucket. The node role is not, so the failure named an identity nobody configured.

Reproduce

On EKS, a ServiceAccount annotated for IRSA with a role that can read the bucket, and a node role that cannot:

import deeplake
ds = deeplake.open("s3://<BUCKET>/<prefix>/table")   # no creds passed

Expected: authenticated as the ServiceAccount role.
Actual: authenticated as the node instance role.

Why it matters

1. It fails to a broader principal, silently. Web identity is the narrow credential; the node role is usually broader and shared by every pod on that node. Where the node role happens to have access, the call succeeds under the wrong identity and no one notices — per-pod scoping is silently not in effect.
2. boto3, aws-sdk-go-v2 and aws-sdk-rust all resolve AssumeRoleWithWebIdentity in their default chain, so this is a deviation from what every other AWS client does.

Workaround

Resolve the web identity ourselves and pass keys explicitly:

creds = {"aws_access_key_id": ..., "aws_secret_access_key": ...,
         "aws_session_token": ..., "aws_region": "us-east-1"}
deeplake.open(url, creds=creds)

That works, but it means every caller on EKS has to reimplement a standard step of the AWS credential chain.

Ask

Add AssumeRoleWithWebIdentity to the default chain, ahead of the instance-metadata step. If it cannot be added, fail rather than fall through to IMDS when AWS_WEB_IDENTITY_TOKEN_FILE is present — falling back to a different principal than the environment asks for is worse than an error.