Cubestore: S3 refresh loop not detecting new AIM Role token from EKS IRSA file

[barbarosh]

Describe the bug

CubeStore fails to reliably refresh AWS S3 credentials when using AWS Web Identity credentials with a Kubernetes projected ServiceAccount token (for example, EKS IRSA).

CubeStore detects Web Identity token rotation by comparing the token file's modified timestamp (mtime):

let current_modified = std::fs::metadata(file)
    .ok()
    .and_then(|m| m.modified().ok());

if current_modified == last_modified {
    continue;
}

However, Kubernetes projected ServiceAccount tokens are managed and rotated by kubelet. Relying only on the file modification timestamp is not a reliable way to detect that the JWT token content has changed.

As a result, CubeStore may fail to detect a rotated ServiceAccount token and does not call Credentials::from_sts(...) with the new token. The previously issued temporary AWS credentials eventually expire, causing S3 requests to fail with expired token/credentials errors.

Additionally, last_modified is updated before the STS credential refresh succeeds:

last_modified = current_modified;
info!("Web identity token file changed, refreshing S3 credentials");

If the STS request fails after last_modified has been updated, CubeStore considers the token file already processed and does not retry the refresh unless the file modification timestamp changes again.

To Reproduce

1. Run CubeStore in Kubernetes/EKS using a ServiceAccount associated with an AWS IAM role via IRSA/Web Identity.

2. Configure CubeStore to use the projected ServiceAccount token:

CUBESTORE_AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
CUBESTORE_AWS_WEB_IDENTITY_ROLE_ARN=arn:aws:iam::<account-id>:role/<role-name>

3. Do not configure static AWS access and secret keys.

4. Configure CubeStore to use an S3 bucket.

5. Start CubeStore and verify that initial S3 access works successfully.

6. Keep CubeStore running long enough for the Kubernetes projected ServiceAccount token and AWS STS credentials to rotate/expire.

7. Observe that CubeStore may continue using expired AWS credentials or fail to refresh credentials successfully.

8. S3 operations start failing with an expired token/credentials error.

Restarting the CubeStore pod restores S3 access because CubeStore reads the current projected ServiceAccount token and obtains new STS credentials during startup.

Expected behavior

CubeStore should reliably refresh AWS credentials when using Kubernetes/EKS Web Identity authentication.

The refresh logic should not rely exclusively on the projected token file's mtime.

CubeStore should periodically read the token file and detect changes in the JWT content, for example by comparing the token content or a hash of the token.

The token should only be marked as successfully processed after Credentials::from_sts(...) succeeds.

If the STS credential refresh fails, CubeStore should retry the refresh on the next polling interval instead of waiting for another token file modification.

Expected behavior:

1. Kubernetes rotates the projected ServiceAccount token.
2. CubeStore detects that the JWT content has changed.
3. CubeStore calls AssumeRoleWithWebIdentity using the new JWT.
4. New temporary AWS credentials are obtained.
5. The S3 client/bucket credentials are replaced.
6. S3 access continues without interruption.

Version:

1.6.69

Additional context

The issue occurs when running CubeStore on Kubernetes/EKS with AWS IRSA and a projected ServiceAccount token.

The relevant credential refresh logic uses file metadata to detect token rotation:

let mut last_modified = token_file
    .as_ref()
    .and_then(|f| std::fs::metadata(f).ok()?.modified().ok());

and later:

let current_modified = std::fs::metadata(file)
    .ok()
    .and_then(|m| m.modified().ok());

if current_modified == last_modified {
    continue;
}

last_modified = current_modified;

A more reliable implementation would periodically read the projected token and compare its content or hash:

let jwt = std::fs::read_to_string(file)?;

if last_token.as_deref() == Some(jwt.as_str()) {
    continue;
}

let credentials = Credentials::from_sts(
    arn,
    "cubestore",
    &jwt,
)?;

// Mark the token as processed only after STS succeeds.
last_token = Some(jwt);

Alternatively, CubeStore could refresh temporary AWS credentials based on their expiration time with an appropriate safety margin.

Currently, restarting the CubeStore pod is a workaround because the current ServiceAccount token is read and fresh STS credentials are obtained during startup.