django-storages support for AWS_SESSION_TOKEN for the boto backend

[bxm156]

In AWS Lambda, you can assign IAM role to a function, which should describe the permissions allowed for the function.This is placed into the env, via 3 params:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_SESSION_TOKEN

Lambda is using STS, and all 3 must be supplied to the S3Connection object.

Normally, boto would handle this itself, but since we manually create the S3Connection, we need to handle this ourselves. This is achieved by passing security_token=<AWS_SESSION_TOKEN> into the S3Connection object.

Examples

Without the session token

zappa invoke prod "import boto; from boto.s3.connection import S3Connection; import os; c = S3Connection(os.getenv('AWS_ACCESS_KEY_ID'), os.getenv('AWS_SECRET_ACCESS_KEY')); rs = c.get_all_buckets(); print rs[7].get_key('index.html');" --raw

=>

Calling invoke for environment prod..
START RequestId: fc2d8da1-0965-11e7-b4fc-45f80254b4aa Version: $LATEST
S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>**REDACTED**</AWSAccessKeyId><RequestId>9B0A00B299B5BD02</RequestId><HostId>x+cb94gePg+XCM119UdKNktXAI26fqrru7Ih7rpVqvQVJ+K4Z/tTSSsYDsGXUQ03XP+E8BHhgZo=</HostId></Error>: S3ResponseError
Traceback (most recent call last):
  File "/var/task/handler.py", line 507, in lambda_handler
    return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 241, in lambda_handler
    return handler.handler(event, context)
  File "/var/task/handler.py", line 370, in handler
    exec(raw_command)
  File "<string>", line 1, in <module>
  File "/tmp/pip-build-LWM5xq/boto/boto/s3/connection.py", line 444, in get_all_buckets
S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>**REDACTED**</AWSAccessKeyId><RequestId>9B0A00B299B5BD02</RequestId><HostId>x+cb94gePg+XCM119UdKNktXAI26fqrru7Ih7rpVqvQVJ+K4Z/tTSSsYDsGXUQ03XP+E8BHhgZo=</HostId></Error>

END RequestId: fc2d8da1-0965-11e7-b4fc-45f80254b4aa
REPORT RequestId: fc2d8da1-0965-11e7-b4fc-45f80254b4aa  Duration: 358.01 ms   Billed Duration: 400 ms  Memory Size: 512 MB     Max Memory Used: 302 MB

And when using the session token...

 zappa invoke prod "import boto; from boto.s3.connection import S3Connection; import os;  c = S3Connection(os.getenv('AWS_ACCESS_KEY_ID'), os.getenv('AWS_SECRET_ACCESS_KEY'), security_token=os.getenv('AWS_SESSION_TOKEN')); rs = c.get_all_buckets(); print rs[7].get_key('index.html');" --raw

=>

Calling invoke for environment prod..
START RequestId: 84af8947-0966-11e7-9bcb-438f216ed0ae Version: $LATEST
<Key: queercon-web-assets,index.html>
END RequestId: 84af8947-0966-11e7-9bcb-438f216ed0ae
REPORT RequestId: 84af8947-0966-11e7-9bcb-438f216ed0ae  Duration: 590.61 ms   Billed Duration: 600 ms  Memory Size: 512 MB     Max Memory Used: 302 MB

[jschneier]

Fixed by #283.

[aarcro]

There is still an issue here. If self.access_key and self.secret_key are set from settings.py, then it can never pick up the security_token

[jschneier]

Have a look at #370 please.

[jlev]

This still appears to be a problem. I need to be able to set access_key, secret_key and session_token, in order to use S3 Multi-Factor authentication.

[msheiny]

Sooo this is not ideal fix for the issue @jlev (and myself) are experiencing... but i didn't feel like maintaining a fork and it appears that PRs are in a backlog to merge (totally understandable by the way! Maintaining OSS software can be all time-consuming).

So anyways I was able to work around this in the current logic with my backend class like so:

class MediaStorage(S3Boto3Storage):
    access_key = False
    secret_key = False

[thesunlover]

from storages.utils import lookup_env
from storages.backends.s3boto3 import S3Boto3Storage


class S3Boto3StorageForZappaAWSRole(S3Boto3Storage):
    """ 
    This is required, because AWS Role changes its credentials
    from 1 to 12 hours and if the Lambda instance has stored
    expired version of them we can't have a successful request
    until the Lambda Instance's live has expired
    """
    def _get_security_token(self):
        """
        Gets the security token to use when accessing S3. Get it from
        the environment variables.
        """
        return lookup_env(S3Boto3Storage.security_token_names)

    def _get_access_keys(self):
        """
        Gets the access keys to use when accessing S3. If none is
        provided in the settings then get them from the environment
        variables.
        """
        access_key = lookup_env(S3Boto3Storage.access_key_names)
        secret_key = lookup_env(S3Boto3Storage.secret_key_names)
        return access_key, secret_key