Does PynamoDB support Boto3's various retry modes?

[ryancausey]

I'm looking at using PynamoDB over direct use of Boto3 to talk to DynamoDB. However, one of the features of Boto3 I've been using is its retry modes. I didn't find any similar retry mode configuration options in the PynamoDB connection API docs. It's not clear to me where botocore ends and boto3 begins, and whether PynamoDB supports this functionality based on using botocore.

[ikonst]

I haven't been using the library for a while (my current work is in the golang ecosystem) but IIRC #1079 has switches PynamoDB back to use more of botocore under the hood, which should leverage botocore's retry behavior.

I suspect boto3's retry settings are controlling the botocore implementation, so we should respect the retry settings. Do you mind giving it a try?

Maybe there's a reasonable integration test we can add, e.g. if we fashion a simple proxying server with https://docs.python.org/3/library/http.server.html, so we could inject errors.

[ryancausey]

I found where I believe PynamoDB configures this retry behavior in the Connection class when instantiating the botocore client:

PynamoDB/pynamodb/connection/base.py

Line 407 in f0bc917

retries={

This looks like it overrides whatever may be set in a AWS configuration file. I guess it would be a feature request to allow setting this retry configuration dict somewhere in code (I don't know where is best) or allowing botocore to read it from an AWS configuration file? It appears that the retry configuration in regular botocore can be set in two ways:

1. passing an explicit Config object to the client instantiation.
2. setting the retry configuration in the ~/.aws/config file.

The passing a Config object takes precedence, and will override whatever settings may be in the ~/.aws/config file.

[ikonst]

Oh yes, we definitely need to fix this. It makes no sense to hardcode a retry mode here.

[ryancausey]

I don't know how to make this a non-breaking change, as it looks like the default settings for retry attempts is 3: https://pynamodb.readthedocs.io/en/stable/settings.html#max-retry-attempts

This is the value that gets used in that retry config block. The only way to allow botocore to use the ~/.aws/config file values is to not specify the retry dict at all in the config, meaning we would have to remove the default value of max-retry-attempts in the settings in order to support that.

[ikonst]

I suppose we'd have to add a disable_legacy_retry_policy option, which we could remove at a future major version.