Skip to content

Conversation

hssyoo
Copy link
Contributor

@hssyoo hssyoo commented Oct 2, 2025

Currently, boto3 requires runtime environments in select optimized instance types to use the CRT transfer client (unless users directly interface with the s3transfer library). This commit adds a new crt mode to TransferConfig's preferred_transfer_client parameter. When crt mode is used, boto3 will enable CRT transfers in all environments as long as the minimum required version of CRT is installed.

@hssyoo hssyoo requested a review from jonathan343 October 2, 2025 18:03
is_optimized_instance
and pref_transfer_client == constants.AUTO_RESOLVE_TRANSFER_CLIENT
):
) or pref_transfer_client == constants.CRT_TRANSFER_CLIENT:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do the top level configuration parameters on download_obj, download_file, upload_obj, and upload_file do when this is turned on?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I'm just getting back to this work. It's the same behavior as using the auto value on an optimized instance:

  • Filename, Bucket, Key, ExtraArgs, and Callback work as expected (same as core/native transfer manager).
  • Values in Config (ie boto3.s3.transfer.TransferConfig) are ignored.

Ideally this would be rolled out with support for transfer manager config parameters. But this at least gets us "feature parity" in environments that aren't optimized instances. ie It's additive without being a regression (in itself).

I think whether this could expose us to potential future regressions depends on what we envision transfer manager config parameter support looks like:

  1. Same interface/usage as the native transfer manager. It uses the values in the boto3.s3.transfer.TransferConfig object passed to the Config arg.
  2. Some other interface, eg a new field in boto3.s3.transfer.TransferConfig. Something like crt_config. It doesn't really matter exactly what it would look like, just that it would be a different interface.

The way I'm thinking about this is that we wouldn't want to pursue to 1st option because that could break users:

  • I'm on an optimized instance with the auto preferred client configured
  • I pass a transfer config that's currently doing nothing
  • I upgrade and all of a sudden my transfer config starts doing something

So while not exposing configs along with this update isn't ideal, it at least gives users an option to use CRT transfer manager with CRT-determined default values. I'm also open to rolling out the 2 together instead of in separate releases. I just figured it would be faster to let users enable CRT everywhere if we roll out incrementally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants