Add S3 trigger handlers to storage construct

[cromo]

Start from the Use-case

I want to run some code when a file is uploaded to an S3 bucket.

Currently, I'm trying to use the storage construct to create the bucket and attempting to attach a handler via Serverless's s3 event:

service: aws-python-sqs-worker-project
frameworkVersion: '3'

plugins:
  # Adds CDK constructs, see: https://github.com/getlift/lift
  - serverless-lift

package:
  patterns:
    - '!node_modules/**'

provider:
  name: aws
  runtime: python3.8

constructs:
  raw-messages:
    type: storage

functions:
  normalize:
    handler: src/normalize.lambda_handler
    events:
      - s3:
          bucket: ${construct:raw-messages.bucketName}
          event: s3:ObjectCreated:*

But this produces an error:

% npm run sls -- deploy --stage dev

> aws-python-sqs-worker-project@1.0.0 sls
> sls "deploy" "--stage" "dev"


Deploying aws-python-sqs-worker-project to stage dev (us-east-1)

✖ Stack aws-python-sqs-worker-project-dev failed to deploy (1s)
Environment: darwin, node 16.8.0, framework 3.19.0 (local), plugin 6.2.2, SDK 4.3.2
Credentials: Local, environment variables
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
When specifying "s3.bucket" with CloudFormation intrinsic function, you need to specify that the bucket is existing via "existing: true" property.

The bucket doesn't exist though - it's being created as part of this deployment, so the error doesn't make sense. At the very least, I'd like to know a reasonable workaround to this problem.

Even better, I would like to be able to specify trigger handlers in the construct itself, similar to how the queue construct works.

Example Config

Proposed config:

constructs:
  bucket:
    type: storage
    triggers:
      - event: s3:ObjectCreated:*
        handler: handler.trigger

Making triggers an array allows different handlers to be attached to different events.

Implementation Idea

No response

[fredericbarthelet]

Thanks for opening this issue @cromo :)

You can indeed fix your problem by adding existing: true within your event configuration as such :

functions:
  normalize:
    handler: src/normalize.lambda_handler
    events:
      - s3:
          bucket: ${construct:raw-messages.bucketName}
          event: s3:ObjectCreated:*
          existing: true

While the name might not be well chosen, this instruct Serverless framework not to handle bucket creation. This is indeed shipped by Lift and therefore no the scope of the framework anymore. One more adapted naming could be provisionedElsewhere.

Specifying triggers on the storage construct itself seems like a good idea indeed. Would you like to detail the proposed implementation strategy first and then eventually produce a PR for such feature ?

[cromo]

Ah, I was under the impression that Lift somehow extended Serverless's existing CloudFormation deploy structure; I wasn't aware that it actually deploys separately. Knowing that makes the need to use existing: true make sense.

Unfortunately, I don't have the bandwidth at work or in my personal life to carry through on such a feature right now, though it does sound interesting. At the very least, I'd like to see the explanation of why existing: true is needed in the storage construct docs, as it had no hints on how to attach triggers at all. Maybe in a month or two I'd be able to contribute more than that.

[throberto]

Currently I need an architecture where any object created in storage triggers a queue in SQS, which consequently triggers a lambda function.

It looks more or less like this:

constructs:

  my-storage:
    type: storage
    extensions:
       bucket:
         Properties:
           NotificationConfiguration:
             QueueConfigurations:
               - Event: s3:ObjectCreated:*
               ...
  my-queue:
    type: queue
    ...

It would be interesting if the storage constructor itself could already have the triggers natively ;)