Skip to content

Commit 1b1a3d3

Browse files
treedertreeder
committed
Merge pull request #53 from iron-io/docs
Docs
2 parents 520c324 + c1bf769 commit 1b1a3d3

File tree

11 files changed

+425
-266
lines changed

11 files changed

+425
-266
lines changed

‎CONTRIBUTING.md

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,16 @@ Contributing to Lambda
33

44
### Prerequisites
55

6-
This has only been tested on OSX and Linux.
6+
These workflows have been tested on Linux and Mac. You must have a:
77

8-
* Working Go 1.5 onwards installation.
9-
* Working [Docker](http://docker.com) installation.
10-
* GNU Make.
8+
- Working `make` command
9+
- Working [Go][go] >=1.5 installation,
10+
- Working [Glide][glide] installation,
11+
- Working [Docker][docker] installation.
12+
13+
[go]: http://golang.org
14+
[glide]: http://glide.sh
15+
[docker]: http://www.docker.com
1116

1217
To work on Java code, you'll need a working JDK and [Apache Maven](http://maven.apache.org).
1318
For node.js, any version of node >=0.10.0 will do.

‎docs/aws.md

Lines changed: 288 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,10 @@
1-
## Using the AWS SDK from Lambda functions.
1+
Interacting with AWS Services
2+
=============================
3+
4+
The node.js and Python stacks include SDKs to interact with other AWS services.
5+
For Java you will need to include any such SDK in the JAR file.
6+
7+
## Credentials
28

39
Running Lambda functions outside of AWS means that we cannot automatically get
410
access to other AWS resources based on Lambda subsuming the execution role
@@ -56,3 +62,284 @@ If you have an existing image with the same name registered with IronWorker,
5662
the environment variables will not simply be updated. You need to first delete
5763
the code from HUD and then publish the function again. This will unfortunately
5864
result in a new webhook URL for the function.
65+
66+
## Example: Using Lambda with IronWorker and Amazon Simple Notification Service
67+
68+
Lambda's premise of server-less computing requires a few infrastructural pieces
69+
other than just the Docker image. First there needs to be a platform that can
70+
run these Docker images on demand. Second, we need some way to invoke the
71+
Lambda function based on an external event.
72+
73+
In this example, we will look at how to use IronWorker and Amazon Simple
74+
Notification Service (SNS) to create a function that can search a given URL for
75+
a user-specified keyword. You can build upon this, coupled with some storage
76+
provider (like Amazon S3) to build a simple search engine.
77+
78+
The concepts introduced here can be used with any infrastructure that let's you
79+
start a Docker container on some event. It is not tied to IronWorker.
80+
81+
The code for this example is located [here](../examples/sns/sns.js).
82+
83+
### Setup
84+
85+
Make sure you have an [IronWorker](https://www.iron.io/platform/ironworker/)
86+
account. You can make one [here](https://www.iron.io/get-started/). You will
87+
need a [Docker Hub](https://hub.docker.com) account to publish the Lambda function.
88+
89+
Also set up an [AWS
90+
account](http://docs.aws.amazon.com/sns/latest/dg/SNSBeforeYouBegin.html) and [create a SNS topic](http://docs.aws.amazon.com/sns/latest/dg/CreateTopic.html). Call this topic `sns-example`. Carefully note the region the topic was created in. The region is found in the topic ARN.
91+
92+
You will also need credentials to use the AWS SDK. These credentials can be
93+
obtained from the IAM and are in the form of an Access Key and Secret. See the
94+
[AWS page](./aws.md) page for more information.
95+
96+
### Function outline
97+
98+
SNS can notify a variety of endpoints when a message is published to the topic.
99+
One of these is an HTTPS URL. IronWorker provides an HTTPS URL that runs an
100+
instance of the Docker image when the URL receives a POST.
101+
102+
In this example, we will manually publish messages to SNS, which will trigger
103+
the webhook, our Lambda function will fetch the URL passed in the message,
104+
search for the keyword in the response, and print out the count.
105+
106+
Here is the beginning of the function:
107+
108+
```js
109+
var http = require('http');
110+
var AWS = require('aws-sdk');
111+
AWS.config.region = 'us-west-1';
112+
113+
function searchString(context, text, key) {
114+
// Global and Ignore case flags.
115+
var regex = new RegExp(key, 'gi');
116+
117+
var results = [];
118+
var m;
119+
while ((m = regex.exec(text)) != null) {
120+
results.push(m);
121+
}
122+
123+
console.log("Found", results.length, "instances of", key);
124+
context.succeed();
125+
}
126+
127+
function searchBody(context, res, key) {
128+
if (res.statusCode === 200) {
129+
var body = "";
130+
res.on('data', function(chunk) { body += chunk.toString(); });
131+
res.on('end', function() { searchString(context, body, key); });
132+
} else {
133+
context.fail("Non-200 status code " + res.statusCode + " fetching '" + message.url + "'. Aborting.");
134+
}
135+
}
136+
```
137+
138+
Here we set up the various functions that will implement our Lambda function's
139+
logic. Set the AWS region to the region in the SNS topic ARN, otherwise our
140+
function will fail.
141+
142+
The `searchBody` function takes a node `http.ClientResponse` and gathers the
143+
body data, then calls `searchString` to perform the regular expression match.
144+
Finally each function invokes the `context.fail()` or `context.succeed()`
145+
functions as appropriate. This is important, otherwise our function won't
146+
terminate until it times out, even if execution was done.
147+
148+
#### Handling SNS event types.
149+
150+
SNS events send the payload as a JSON message to our webhook. These are passed
151+
on to the Lambda function in the handler's `event` parameter. Each SNS message
152+
contains a `Type` field. We are interested in two types - `Notification` and
153+
`SubscriptionConfirmation`. The former is used to deliver published messages.
154+
155+
Before SNS can start sending messages to the subscriber, the subscriber has to
156+
confirm the subscription. This is to prevent abuse. The
157+
`SubscriptionConfirmation` type is used for this. Our function will have to
158+
deal with both.
159+
160+
```js
161+
exports.handler = function(event, context) {
162+
if (event.Type == 'Notification') {
163+
// ...
164+
}
165+
else if (event.Type == 'SubscriptionConfirmation') {
166+
// ...
167+
} else {
168+
console.log("unknown event.Type", event.Type);
169+
context.fail();
170+
}
171+
};
172+
```
173+
174+
We can use the SDK to confirm the subscription.
175+
176+
```js
177+
var sns = new AWS.SNS();
178+
var params = {
179+
Token: event.Token,
180+
TopicArn: event.TopicArn,
181+
};
182+
sns.confirmSubscription(params, function(err, data) {
183+
if (err) {
184+
console.log(err, err.stack);
185+
context.fail(err);
186+
} else {
187+
console.log("Confirmed subscription", data);
188+
console.log("Ready to process events.");
189+
context.done();
190+
}
191+
});
192+
```
193+
194+
The `Token` is unique and has to be sent to SNS to indicate that we are a valid
195+
subscriber that the message was intended for. Once we confirm the subscription,
196+
this run of the Lambda function is done and we can stop (`context.done()`).
197+
SNS is now ready to run this Lambda function when we publish to the topic.
198+
199+
Finally we come to the event type we expect to receive most often --
200+
`Notification`. In this case, we try to grab the url and keyword from the
201+
message and run our earlier `searchBody()` function on it.
202+
203+
```js
204+
try {
205+
var message = JSON.parse(event.Message);
206+
if (typeof message.url == "string" && typeof message.keyword == "string") {
207+
http.get(message.url, function(res) { searchBody(context, res, message.keyword); })
208+
.on('error', function(e) {
209+
context.fail(e);
210+
});
211+
} else {
212+
context.fail("Invalid message " + event.Message);
213+
}
214+
} catch(e) {
215+
context.fail(e);
216+
}
217+
```
218+
219+
### Trying it out
220+
221+
With this function ready, we can Dockerize it and publish it to actually try it
222+
out with SNS.
223+
224+
```sh
225+
ironcli lambda create-function -function-name <Docker Hub username>/sns-example -runtime
226+
nodejs -handler sns.handler sns.js
227+
```
228+
229+
This will create a local docker image. The `publish-function` command will
230+
upload this to Docker Hub and register it with IronWorker.
231+
232+
FIXME(nikhil): AWS credentials bit.
233+
234+
To be able to use the AWS SDK, you'll also need to set two environment
235+
variables. The values must be your AWS credentials.
236+
237+
```sh
238+
AWS_ACCESS_KEY_ID=<access key>
239+
AWS_SECRET_ACCESS_KEY=<secret key>
240+
241+
ironcli publish-function -function-name <Docker Hub username>/sns-example:latest
242+
```
243+
244+
Visit the published function's code page in the [IronWorker control
245+
panel](https://hud.iron.io). You should see a cloaked field called "Webhook
246+
URL". Copy this URL.
247+
248+
In the AWS SNS control panel, visit the `sns-example` topic. Click "Create
249+
Subscription". Select the subscription type as HTTPS and paste the webhook URL.
250+
Once you save this, the IronWorker task should have been launched and then
251+
finished successfully with a "Confirmed subscription" message.
252+
253+
Now you can click the blue "Publish to topic" button on the AWS SNS control
254+
panel. Select the message format as JSON and the contents as (for example):
255+
256+
```js
257+
{
258+
"default": "{\"url\": \"http://www.econrates.com/reality/schul.html\", \"keyword\": \"blackbird\"}"
259+
}
260+
```
261+
262+
SNS will send the string in the `"default"` key to all subscribers. You should
263+
be able to see that the IronWorker task has been run again. If everything went
264+
well, it should have printed out a summary and exited successfully.
265+
266+
That's it, a simple, notification based, Lambda function.
267+
268+
## Example: Reading and writing to S3 Bucket
269+
270+
This example demonstrates modifying S3 buckets and using the included
271+
ImageMagick tools in a node.js function. Our function will fetch an image
272+
stored in a key specified by the event, resize it to a width of 1024px and save
273+
it to another key.
274+
275+
The code for this example is located [here](../examples/s3/example.js).
276+
277+
The event will look like:
278+
279+
```js
280+
{
281+
"bucket": "iron-lambda-demo-images",
282+
"srcKey": "waterfall.jpg",
283+
"dstKey": "waterfall-1024.jpg"
284+
}
285+
```
286+
287+
The setup, imports and SDK initialization.
288+
289+
```js
290+
var im = require('imagemagick');
291+
var fs = require('fs');
292+
var AWS = require('aws-sdk');
293+
294+
exports.run = function(event, context) {
295+
var bucketName = event['bucket']
296+
var srcImageKey = event['srcKey']
297+
var dstImageKey = event['dstKey']
298+
299+
var s3 = new AWS.S3();
300+
}
301+
```
302+
303+
First we retrieve the source and write it to a local file so ImageMagick can
304+
work with it.
305+
306+
```js
307+
s3.getObject({
308+
Bucket: bucketName,
309+
Key: srcImageKey
310+
}, function (err, data) {
311+
312+
if (err) throw err;
313+
314+
var fileSrc = '/tmp/image-src.dat';
315+
var fileDst = '/tmp/image-dst.dat'
316+
fs.writeFileSync(fileSrc, data.Body)
317+
318+
});
319+
```
320+
321+
The actual resizing involves using the identify function to get the current
322+
size (we only resize if the image is wider than 1024px), then doing the actual
323+
conversion to `fileDst`. Finally we upload to S3.
324+
325+
```js
326+
im.identify(fileSrc, function(err, features) {
327+
resizeIfRequired(err, features, fileSrc, fileDst, function(err, resized) {
328+
if (err) throw err;
329+
if (resized) {
330+
s3.putObject({
331+
Bucket:bucketName,
332+
Key: dstImageKey,
333+
Body: fs.createReadStream(fileDst),
334+
ContentType: 'image/jpeg',
335+
ACL: 'public-read',
336+
}, function (err, data) {
337+
if (err) throw err;
338+
context.done()
339+
});
340+
} else {
341+
context.done();
342+
}
343+
});
344+
});
345+
```

‎docs/docker.md

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Running the Lambda Docker images
2+
3+
Docker images created by running the `create-function` subcommand on a Lambda
4+
function are ready to execute. They do not need any command line arguments.
5+
6+
The `test-function` subcommand can pass the correct parameters to `docker run`
7+
to run those images with the payload and environment variables set up
8+
correctly. If you would like more control, like mounting volumes, or adding
9+
more environment variables this guide describes how to directly run these
10+
images using `docker run`.
11+
12+
## Payload
13+
14+
The `payload` argument is written to a random, opaque directory on the host.
15+
The file itself is called `payload.json`. This directory is mapped to the
16+
`/mnt` volume in the container, so that the payload is available in
17+
`/mnt/payload.json`. This is not REQUIRED, since the actual runtimes use the
18+
`PAYLOAD_FILE` environment variable to discover the payload location.
19+
20+
## Environment variables
21+
22+
The `TASK_ID` variable maps to the AWS Request ID. This should be set to
23+
something unique (a UUID, or an incrementing number).
24+
25+
`test-function` runs a container with 300MB memory allocated to it. This same
26+
information is available inside the container in the `TASK_MAXRAM` variable.
27+
This value can be a number in bytes, or a number suffixed by `b`, `k`, `m`, `g`
28+
for bytes, kilobytes, megabytes and gigabytes respectively. These are
29+
case-insensitive.
30+
31+
The following variables are set for AWS compatibility:
32+
* `AWS_LAMBDA_FUNCTION_NAME` - The name of the docker image.
33+
* `AWS_LAMBDA_FUNCTION_VERSION` - The default is `$LATEST`, but any string is
34+
allowed.
35+
* `AWS_ACCESS_KEY_ID` - Set this to the Access Key to allow the Lambda function
36+
to use AWS APIs.
37+
* `AWS_SECRET_ACCESS_KEY` - Set this to the Secret Key to allow the Lambda
38+
function to use AWS APIs.
39+
40+
## Running the container
41+
42+
The default `test-function` can then be approximated as the following `docker
43+
run` command:
44+
45+
```sh
46+
mkdir /tmp/payload_dir
47+
echo "<payload>" >> /tmp/payload_dir/my_payload.json
48+
docker run -v /tmp/payload_dir:/mnt \
49+
-m 1G \
50+
-e PAYLOAD_FILE=/mnt/my_payload.json \
51+
-e TASK_ID=$RANDOM \
52+
-e TASK_MAXRAM=1G \
53+
-e AWS_LAMBDA_FUNCTION_NAME=user/fancyfunction \
54+
-e AWS_LAMBDA_FUNCTION_VERSION=1.0 \
55+
-e AWS_ACCESS_KEY_ID=<access key> \
56+
-e AWS_SECRET_ACCESS_KEY=<secret key> \
57+
--rm -it
58+
user/fancyfunction
59+
```

‎docs/environment.md

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -154,7 +154,17 @@ function, we have not tried it out.
154154

155155
[predef]: http://docs.aws.amazon.com/lambda/latest/dg/java-handler-using-predefined-interfaces.html
156156

157-
### Context object
157+
### Logging
158+
159+
The [log4j and LambdaLogger
160+
styles](http://docs.aws.amazon.com/lambda/latest/dg/java-logging.html) that log
161+
to CloudWatch are not supported.
158162

159-
TODO
163+
### Context object
160164

165+
* `context.getFunctionName()` returns a String of the form of a docker image,
166+
for example `iron/test-function`.
167+
* `context.getFunctionVersion()` is always the string `"$LATEST"`.
168+
* `context.getAwsRequestId()` reflects the environment variable `TASK_ID` which is
169+
set to the task ID on IronWorker. If TASK_ID is empty, a new UUID is used.
170+
* `getInvokedFunctionArn()`, `getLogGroupName()`, `getLogStreamName()`, `getIdentity()`, `getClientContext()`, `getLogger()` return `null`.

0 commit comments

Comments
 (0)