remotecache: Refactor S3 cache backend to use minio-go client

[sorenhansendk]

The MinIO SDK is designed to talk to any S3-compatible object storage, and the project actively maintains interoperability across providers. This directly addresses the signature/header edge cases we’ve hit with S3-compatible endpoints (e.g., checksum and Accept-Encoding interactions)

Why

• Better cross-provider compatibility: MinIO’s Go client targets the S3 API surface used by “S3-compatible” stores, which reduces surprises when talking to non-AWS endpoints.
• Avoids recent checksum/signing pitfalls: AWS SDK v2 introduced default response/integrity checksum behaviors that can interact poorly with some S3-compatible services, contributing to SignatureDoesNotMatch failures. Using minio-go sidesteps those defaults and focuses on portable S3 semantics.
• GCS S3 interoperability: We rely on HMAC/S3-style access against Google Cloud Storage; their interoperability path is documented, and minio-go is frequently used with such endpoints

What changes

• Implementation detail only: all object operations (HEAD/GET/PUT/COPY) are handled by minio-go
• No changes to caller-facing config/flags (bucket/prefix/path-style/endpoint)

---

Closes: #3749 (minio-go works with GCS via HMAC)
Closes: #5804 (minio-go works with OpenStack S3)
Closes: #4487 (minio-go works with Alibaba Cloud)

[tonistiigi]

cc @bpaquet

[crazy-max]

Rebased and update to minio 7.0.97 removing a module importing a lot: minio/minio-go@v7.0.95...v7.0.97#diff-ca70b2f0fe15f1fa83771dd5b232679a5db6bd062df5cbd081a5cb14c00d87c8L14

[tonistiigi]

@bpaquet I think we would like to get this in, unless you have some arguments against it.

[tonistiigi]

cc @azr as well

[bpaquet]

I had a quick look, seems the client only support static creds. IMHO we should keep a way to use any kind of creds.
With a buildx scenario, the creds can be "resolved" by buildx, but without buildx, using minio-go seems to limit to static creds usage.

Standard creds are

• Ec2 Instance profile
• Web identify token (inside kube)
• Temporary creds through SSO

From a personal point of view, I think the standard is s3 :)

[azr]

Question: why not make a minio remotecache ? and sort of slowly deprecate the s3 one, to give people options & time ? It would also make it easy to fix/circumvent any issues and still allow to avoid these pitfalls described up there.

Other than that, I don't think I can come up with any strong objection with this.

[bpaquet]

+1, seems better to do a minio remote cache exporter and mutualize the code

[sorenhansendk]

Thanks for the review and for flagging the credential concerns, @bpaquet 🙏

I had a quick look, seems the client only support static creds. IMHO we should keep a way to use any kind of creds. With a buildx scenario, the creds can be "resolved" by buildx, but without buildx, using minio-go seems to limit to static creds usage._

You’re right that the current diff effectively narrows us to static keys. I’m proposing to wire minio-go’s credential chain so we keep parity with common setups (env, shared file, and IAM/IRSA/ECS/IMDS), while preserving explicit static keys as today.

Tiny, backward-compatible change:

func resolveCredentials(config Config) *credentials.Credentials {
    if config.AccessKeyID != "" || config.SecretAccessKey != "" || config.SessionToken != "" {
        return credentials.NewStaticV4(config.AccessKeyID, config.SecretAccessKey, config.SessionToken)
    }

    return credentials.NewChainCredentials([]credentials.Provider{
        &credentials.EnvAWS{},             // AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN
        &credentials.EnvMinio{},           // MINIO_ACCESS_KEY / MINIO_SECRET_KEY / MINIO_SESSION_TOKEN
        &credentials.FileAWSCredentials{}, // ~/.aws/credentials or AWS_SHARED_CREDENTIALS_FILE (+ AWS_PROFILE)
        &credentials.IAM{Client: &http.Client{}}, // EC2 IMDS / ECS / IRSA (web identity)
    })
}

… and then use it when creating the client:

client, err := minio.New(parsedURL.Host, &minio.Options{
    Creds:        resolveCredentials(config),
    Secure:       parsedURL.Scheme == "https",
    Region:       config.Region,
    BucketLookup: bucketLookup,
})

+1, seems better to do a minio remote cache exporter and mutualize the code

Yes, that is of course also a option. Should I create a new pull request with the new cache provider?

[crazy-max]

+1, seems better to do a minio remote cache exporter and mutualize the code

Yes, that is of course also a option. Should I create a new pull request with the new cache provider?

Don't think we want another cache exporter that would increase the binary size with new set of dependencies if there is no real adding-value.

Tiny, backward-compatible change:

func resolveCredentials(config Config) *credentials.Credentials {
if config.AccessKeyID != "" || config.SecretAccessKey != "" || config.SessionToken != "" {
return credentials.NewStaticV4(config.AccessKeyID, config.SecretAccessKey, config.SessionToken)
}

return credentials.NewChainCredentials([]credentials.Provider{
    &credentials.EnvAWS{},             // AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN
    &credentials.EnvMinio{},           // MINIO_ACCESS_KEY / MINIO_SECRET_KEY / MINIO_SESSION_TOKEN
    &credentials.FileAWSCredentials{}, // ~/.aws/credentials or AWS_SHARED_CREDENTIALS_FILE (+ AWS_PROFILE)
    &credentials.IAM{Client: &http.Client{}}, // EC2 IMDS / ECS / IRSA (web identity)
})

}

… and then use it when creating the client:

client, err := minio.New(parsedURL.Host, &minio.Options{
Creds: resolveCredentials(config),
Secure: parsedURL.Scheme == "https",
Region: config.Region,
BucketLookup: bucketLookup,
})

Thanks @sorenhansendk, I think using a default set of credential providers looks good if config.AccessKeyID and config.SecretAccessKey are not set.

[crazy-max]

Pushed extra commit to support multiple credential providers

PTAL @bpaquet @azr @sorenhansendk

[bpaquet]

Even if we solve the credentials issue, what will happen if we have an issue specific to AWS which is not covered by the minio SDK?
I believe we have a way more user using a real S3 than users on mino.

On the snippet above, do you think it's working with Instance profile and WebIdentiy tokens?

[sorenhansendk]

Thanks for raising this, it’s a very fair concern.

On the credentials side: the MinIO Go client’s IAM provider already supports both EC2/ECS instance profiles and WebIdentity/IRSA (via AWS_WEB_IDENTITY_TOKEN_FILE + AWS_ROLE_ARN). The intention of the chain in this PR is to keep all of those flows working.

That way we keep support for:

• AWS environment
• ~/.aws/credentials
• Static keys (flags/env)
• Instance profiles
• WebIdentity/IRSA

Regarding AWS-specific issues: there is always a theoretical risk that MinIO’s client diverges from the AWS SDK in some edge case, but the MinIO Go client is actively maintained, heavily used directly against real S3, and has a strong focus on S3 API compatibility.

BuildKit’s S3 cache only relies on a small, well-trodden subset of S3 (basic object/bucket ops), so the practical risk is quite low, while the gain in interoperability with S3-compatible backends is significant. If we do hit an AWS-specific bug, we can reproduce and fix it upstream in minio-go and benefit the wider ecosystem.

[azr]

Don't think we want another cache exporter that would increase the binary size with new set of dependencies if there is no real adding-value.

In the use cases I've seen, binary size really doesn't matter as the image would be cached, plus bandwidth and storage are cheap, etc. and this wouldn't be such a big increase. Also, this would be temporary, until we remove the s3 lib. Moreover, isn't the s3 lib used elsewhere ?

The annoying cost would be a maintain cost, I think, and there I hear you, that's a bigger burden than it should.

The added value would be the stability here, I don't know that there is a bug but a few discrepancies were found already, etc.

[bpaquet]

Another point. Today we have parallel upload, but not parallel download (golang interface of the downloader require a file, while buildkit want a byte sequence) I started a PR sometime ago to implement parallel download, but never finished it. Does the minio client support parallel upload and download out of the box?

…

On Wed 19 Nov 2025 at 14:31, Søren Hansen ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In cache/remotecache/s3/s3.go <#6200 (comment)>: > + Creds: credentials.NewChainCredentials([]credentials.Provider{ + &credentials.Static{ + Value: credentials.Value{ + AccessKeyID: config.AccessKeyID, + SecretAccessKey: config.SecretAccessKey, + SessionToken: config.SessionToken, + SignerType: credentials.SignatureV4, + }, + }, + &credentials.EnvAWS{}, + &credentials.EnvMinio{}, + &credentials.FileAWSCredentials{}, + &credentials.IAM{}, + }), Thanks for raising this, it’s a very fair concern. On the credentials side: the MinIO Go client’s IAM provider already supports both EC2/ECS instance profiles and WebIdentity/IRSA (via AWS_WEB_IDENTITY_TOKEN_FILE + AWS_ROLE_ARN). The intention of the chain in this PR is to keep all of those flows working. That way we keep support for: - AWS environment - ~/.aws/credentials - Static keys (flags/env) - Instance profiles - WebIdentity/IRSA Regarding AWS-specific issues: there is always a theoretical risk that MinIO’s client diverges from the AWS SDK in some edge case, but the MinIO Go client is actively maintained, heavily used directly against real S3, and has a strong focus on S3 API compatibility. BuildKit’s S3 cache only relies on a small, well-trodden subset of S3 (basic object/bucket ops), so the practical risk is quite low, while the gain in interoperability with S3-compatible backends is significant. If we do hit an AWS-specific bug, we can reproduce and fix it upstream in minio-go and benefit the wider ecosystem. — Reply to this email directly, view it on GitHub <#6200 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACSJXLVN6D5JF3GCWE6J2L35RWKDAVCNFSM6AAAAACF4MT3UKVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZTIOBSG42DCMBVHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>