As of 11/18/2016, AWS introduced a native mechanism for providing environment variables to lambda functions, which is also supported by CloudFormation. These features negate the gap that streambot existed to fill. You should not use streambot to configure new Lambda functions.
oh faithful streambot
supplying runtime config
to stateless functions
without fail you kept
functions wrapped with care, and safe
from context dot done
a misnomer, sure
your creator may chuckle
but things change, you know
now we bid adieu
to that one point of failure
the streambot table
event source mappings
service tokens, streambot envs
farewell our dear bot
no more consumers
deprecation awaits you
we'll see you never
by @emilymcafee
Streambot is a tool to help you manage AWS Lambda functions as persistent services via AWS CloudFormation templates. It provides assistance for common scenarios where existing CloudFormation support is still inadequate.
Lambda functions are inherently stateless, but a well-designed Twelve-factor app separates configuration from code. The existing Lambda API offers no way for you to configure your application differently across multiple deploys. Streambot provides helper functions to manage and load runtime configuration parameters separately from your code.
An EventSourceMapping connects a Kinesis or DynamoDB stream to a Lambda function, causing records in the stream to trigger Lambda invocations. Presently, there is no support for EventSourceMappings as standalone CloudFormation resources. Streambot provides a way to manage EventSourceMappings through a custom CloudFormation resource.
In order to manage a Lambda function via CloudFormation, the code behind your function must be bundled into a .zip
file and uploaded to S3. If your function has dependencies that include C++ addons precompiled via node-pre-gyp, Streambot will help you bundle your code with dependencies for the correct platform/architecture to run on AWS Lambda, regardless of your native OS.
To use Streambot, you must run a single CloudFormation stack in your AWS account using Streambot's primary template. To do this, clone and build Streambot:
$ git checkout https://github.com/mapbox/streambot
$ cd streambot
$ npm install
$ npm run-script build
# Template file is written to `./cloudformation/streambot.template`
# See ./docs/streambot.template.html for details of the resources that are created
Now you can start a new CloudFormation stack using the template. You may do this using the web console or using an automation tool such as cfn-config. Note: you must create this template in the us-east-1
region. If you don't, your Lambda functions will not know where to find their runtime configuration.
Once the Streambot stack has been created, you're ready to write your own Lambda-backed services. The streambot-example
folder provides an example of how a service might look. There are a number of concepts to grasp as you embark on this journey.
Streambot provides a wrapper function for you to put around your code. This wrapper makes sure that any runtime configuration for your function is loaded into the environment before your function runs. To use this streambot wrapper, you'll adjust your module's index.js
file to look like:
var streambot = require('streambot');
function myFunction(event, callback) {
// This is your custom code to process an event.
// fire the `callback` when finished, in familiar Node.js-style (err first,
// result after).
// Using the streambot wrapper means that `process.env` will be populated with
// your runtime configuration.
}
module.exports.streambot = streambot(myFunction);
If your function has dependencies that include C++ addons precompiled via node-pre-gyp, you can use Streambot's bin/bundle
script to bundle your code with dependencies for the correct platform/architecture to run on AWS Lambda, regardless of your native OS. Once you've bundled your function, you need to upload it to S3.
If your function doesn't have dependencies that rely on C++ addons, you can create a ZIP file yourself with any archiving tool and upload it to S3.
Remember where you upload the bundles, because you'll need to specify their location when you create the Lambda function.
Your CloudFormation template will need to create a Lambda function and also an IAM role defining that function's permissions. Using Streambot may mean adding a couple more resources, depending on how you wish to use it.
You'll have to make an IAM role, and it should completely cover any permissions that your function needs in order to do what it is supposed to do. Additionally, you need to make sure that your function has permission to read runtime configuration from the DynamoDB table created by Streambot's primary template.
{
"Effect": "Allow",
"Action": [
"dynamodb:GetItem"
],
"Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/streambot-env*"
}
}
If your Lambda function needs runtime configuration, you'll need to create a a custom CloudFormation resource backed by the StreambotEnv
Lambda function created as part of the primary Streambot template. This custom resource will manage a record in DynamoDB that your Lambda functions can read their runtime configuration from.
{
"Type": "Custom::StreambotEnv",
"Properties": {
"ServiceToken": "arn:aws:lambda:us-east-1:123456789012:function:StreambotEnvFunction",
"FunctionName": "my-lambda-function",
"EventBucket": "my-event-bucket",
"EventPrefix": "my-event-prefix"
}
}
Property | Description |
---|---|
ServiceToken | The ARN for the StreambotEnv function provided by Streambot's primary template. This is available as an output from the primary stack. |
FunctionName | The name of the Lambda function which you're providing configuration for. |
-any other- | All other properties passed to the custom resource will be provided as environment key-value pairs to your Lambda function at runtime. |
If your Lambda function intends to read from a Kinesis or DynamoDB stream, you'll need to create an EventSourceMapping between your stream and your Lambda function. There is not presently CloudFormation support for these resources. Instead, you can use another a custom CloudFormation resource, this one backed by the StreambotConnector
Lambda function created as part of the primary Streambot template.
{
"Type": "Custom::StreambotConnector",
"DependsOn": "StreambotExampleConfiguration",
"Properties": {
"ServiceToken": "arn:aws:lambda:us-east-1:123456789012:function:StreambotConnectorFunction",
"FunctionRegion": "us-east-1",
"FunctionName": "my-lambda-function",
"StreamArn": "arn:aws:kinesis:us-east-1:123456789012:stream/my-kinesis-stream",
"BatchSize": 1,
"StartingPosition": "LATEST"
}
}
Property | Description |
---|---|
ServiceToken | The ARN for the StreambotConnector function provided by Streambot's primary template. This is available as an output from the primary stack. |
FunctionName | The name of the Lambda function which you're connecting a stream to. |
FunctionRegion | The AWS region in which the Lambda function is located |
StreamArn | The ARN for the stream to feed the Lambda function |
BatchSize | [100] The maximum number of stream records to be processed per Lambda invocation |
StartingPosition | [TRIM_HORIZON] The stream iterator type, either TRIM_HORIZON or LATEST |
Enabled | [true] True or false |
The example service in the streambot-example
folder demonstrates a Lambda-backed service that uses Streambot. The purpose of the service is simply to read records from a Kinesis stream, and write them verbatim to files on S3. See ./docs/streambot-example.template.html
and ./docs/streambot-example.index.html
for detailed discussion of the example template and code.