2. Configure AWS Lambda

Configure AWS Lambda

Create a Lambda function to trigger CI/CD pipeline that builds a DB image.

Example AWS Lambda implementation (lambda_function.py):

import os
import requests

def lambda_handler(event, context):
    url = 'https://gitlab.com/api/v4/projects/123/trigger/pipeline'
    payload = {
        'variables[DOCKERIZER_AWS_S3_REGION]': event['Records'][0]['awsRegion'],
        'variables[DOCKERIZER_AWS_S3_BUCKET]': event['Records'][0]['s3']['bucket']['name'],
        'variables[DOCKERIZER_AWS_S3_OBJECT_KEY]': event['Records'][0]['s3']['object']['key'],
        'token': os.environ['GITLAB_PIPELINE_TRIGGER_TOKEN'],
        'ref': 'main'
    }
    headers = {}
    res = requests.post(url, data=payload, headers=headers)

Here 123 is the ID of the GitLab project that acts as a database builder. See GitLab pipeline to build DB images.

Environment variable GITLAB_PIPELINE_TRIGGER_TOKEN allows to trigger the pipeline externally. It must be secure. See GitLab (or other CI/CD) documentation to learn how to trigger pipelines via the API.

Other variables - DOCKERIZER_AWS_S3_REGION, DOCKERIZER_AWS_S3_BUCKET, DOCKERIZER_AWS_S3_OBJECT_KEY - are used to pass information about the dump to the GitLab pipeline.

Example shell script to build Lambda:

#!/bin/bash
# https://docs.aws.amazon.com/lambda/latest/dg/python-package.html
set -e

rm -rf ./lambda_package/*
cp ./lambda_function.py ./lambda_package/
cd ./lambda_package/
pip install --target ./package requests
cd ./package/
zip -r ../deployment-package.zip .
cd ..
zip deployment-package.zip lambda_function.py

Deploy the result archive to AWS Lambda.

Trigger configuration

Trigger configuration includes:

• Trigger source: S3
• Bucket: choose a bucket
• Event types: All object create events (s3:ObjectCreated:*)
• Suffix: .json. Thus, only the metadata file update (which happens after the dump upload) triggers the CI/CD pipeline.

Important! Be sure to add a new trigger after creating every new bucket.