Init from local repo

.eslintignore

@@ -0,0 +1 @@
+node_modules

.gitignore

@@ -0,0 +1 @@
+node_modules

.npmignore

@@ -0,0 +1,5 @@
+base
+examples
+nodejs
+nodejs4.3
+python2.7

LICENSE

@@ -0,0 +1,19 @@
+Copyright 2016 Michael Hart
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
+of the Software, and to permit persons to whom the Software is furnished to do
+so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,153 @@
+docker-lambda
+-------------
+
+A sandboxed local environment that replicates the live [AWS Lambda](https://aws.amazon.com/lambda/)
+environment almost identically – including installed software and libraries,
+file structure and permissions, environment variables, context objects and
+behaviors – even the user and running process are the same.
+
+You can use it for testing your functions in the same strict Lambda environment,
+knowing that they'll exhibit the same behavior when deployed live. You can
+also use it to compile native dependencies knowing that you're linking to the
+same library versions that exist on AWS Lambda and then deploy using
+the [AWS CLI](https://aws.amazon.com/cli/).
+
+This project consists of a set of Docker images for each of the supported Lambda runtimes
+(Node.js 0.10 and 4.3, Python 2.7\* and Java 8\*) – as well as build
+images that include packages like gcc-c++, git, zip and the aws-cli for
+compiling and deploying.
+
+There's also an npm module to make it convenient to invoke from Node.js
+
+\* NB: Python 2.7 and Java 8 test runners are not yet complete, but both
+languages are installed in the images so can be manually tested
+
+Prerequisites
+-------------
+
+You'll need [Docker](https://www.docker.com) installed
+
+Example
+-------
+
+You can perform actions with the current directory using the `-v` arg with
+`docker run` – logging goes to stderr and the callback result goes to stdout:
+
+```console
+# Test an index.handler function from the current directory on Node.js v4.3
+docker run -v "$PWD":/var/task lambci/lambda
+
+# If using a function other than index.handler, with a custom event
+docker run -v "$PWD":/var/task lambci/lambda index.myHandler '{"some": "event"}'
+
+# Use the original Node.js v0.10 runtime
+docker run -v "$PWD":/var/task lambci/lambda:nodejs
+
+# To compile native deps in node_modules (runs `npm rebuild`)
+docker run -v "$PWD":/var/task lambci/lambda:build
+
+# Run custom commands on the build container
+docker run lambci/lambda:build java -version
+
+# To run an interactive session on the build container
+docker run -it lambci/lambda:build bash
+```
+
+Using the Node.js module (`npm install docker-lambda`) – for example in tests:
+
+```js
+var dockerLambda = require('docker-lambda')
+
+// Spawns synchronously, uses current dir – will throw if it fails
+var lambdaCallbackResult = dockerLambda({event: {some: 'event'}})
+
+// Manually specify directory and custom args
+lambdaCallbackResult = dockerLambda({taskDir: __dirname, dockerArgs: ['-m', '1.5G']})
+```
+
+Create your own Docker image for finer control:
+
+```dockerfile
+FROM lambci/lambda:build
+
+ENV AWS_DEFAULT_REGION us-east-1
+
+ADD . .
+
+RUN npm install
+
+CMD cat .lambdaignore | xargs zip -9qyr lambda.zip . -x && \
+  aws lambda update-function-code --function-name mylambda --zip-file fileb://lambda.zip
+
+# docker build -t mylambda .
+# docker run -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY mylambda
+```
+
+
+Questions
+---------
+
+* *When should I use this?*
+
+  When you want fast local reproducibility. When you don't want to spin up an
+  Amazon Linux EC2 instance (indeed, network aside, this is closer to the real
+  Lambda environment because there are a number of different files, permissions
+  and libraries on a default Amazon Linux instance). When you don't want to
+  invoke a live Lambda just to test your Lambda package – you can do it locally
+  from your dev machine or run tests on your CI system (assuming it has Docker
+  support!)
+
+
+* *Wut, how?*
+
+  By tarring the full filesystem in Lambda, uploading that to S3, and then
+  piping into Docker to create a new image from scratch – then creating
+  mock modules that will be required/included in place of the actual native
+  modules that communicate with the real Lambda coordinating services. Only the
+  native modules are mocked out – the actual parent JS/PY runner files are left
+  alone, so their behaviors don't need to be replicated (like the
+  overriding of `console.log`, and custom defined properties like
+  `callbackWaitsForEmptyEventLoop`)
+
+* *What's missing from the images?*
+
+  Hard to tell – anything that's not readable – so at least `/root/*` –
+  but probably a little more than that – hopefully nothing important, after all,
+  it's not readable by Lambda, so how could it be!
+
+* *Is it really necessary to replicate exactly to this degree?*
+
+  Not for many scenarios – some compiled Linux binaries work out of the box
+  and a CentOS Docker image can compile some binaries that work on Lambda too,
+  for example – but for testing it's great to be able to reliably verify
+  permissions issues, library linking issues, etc.
+
+* *What's this got to do with LambCI?*
+
+  Technically nothing – it's just been incredibly useful during the building
+  and testing of LambCI.
+
+Documentation
+------------
+
+TODO
+
+lambci/lambda
+  - uses ENTRYPOINT, override with `--entrypoint`
+lambci/lambda:build
+  - uses CMD
+
+  'AWS_LAMBDA_FUNCTION_NAME',
+  'AWS_LAMBDA_FUNCTION_VERSION',
+  'AWS_LAMBDA_FUNCTION_MEMORY_SIZE',
+  'AWS_LAMBDA_FUNCTION_TIMEOUT',
+  'AWS_LAMBDA_FUNCTION_HANDLER',
+  'AWS_LAMBDA_EVENT_BODY',
+
+  'AWS_REGION',
+  'AWS_DEFAULT_REGION',
+  'AWS_ACCOUNT_ID',
+  'AWS_ACCESS_KEY_ID',
+  'AWS_SECRET_ACCESS_KEY',
+  'AWS_SESSION_TOKEN',
+

base/Dockerfile

@@ -0,0 +1,17 @@
+FROM lambci/lambda-base
+
+ENV PATH=/usr/local/lib64/node-v4.3.x/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
+    LD_LIBRARY_PATH=/usr/local/lib64/node-v4.3.x/lib:/lib64:/usr/lib64:/var/runtime:/var/task:/var/task/lib
+
+WORKDIR /var/task
+
+ADD yum.conf /etc/yum.conf
+
+# A couple of packages are either missing critical-ish files, or didn't make it into the tar
+# Reinstalling filesystem might not succeed fully, but continue anyway
+RUN yum reinstall -y filesystem; \
+  yum reinstall -y shadow-utils && \
+  yum install -y aws-cli zip git vim docker gcc-c++ clang openssl-devel cmake autoconf automake libtool && \
+  rm -rf /var/cache/yum /var/lib/rpm/__db.* && \
+  > /var/log/yum.log
+

base/create-base.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+IMAGE_NAME=lambci/lambda-base
+
+curl http://lambci.s3.amazonaws.com/fs/nodejs4.3.tgz | gzip -d | docker import - $IMAGE_NAME
+
+curl http://lambci.s3.amazonaws.com/fs/nodejs.tgz -o ../nodejs/run/nodejs.tgz
+cp ../nodejs/run/nodejs.tgz ../nodejs/build/
+
+curl http://lambci.s3.amazonaws.com/fs/python2.7.tgz -o ../python2.7/run/python2.7.tgz
+cp ../python2.7/run/python2.7.tgz ../python2.7/build/
+
+echo "Sandbox user is: $(docker run $IMAGE_NAME stat -c '%U' /tmp)"
+

base/create-build.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+IMAGE_NAME=lambci/lambda-base:build
+
+docker build $BUILD_ARG -t ${IMAGE_NAME} .
+

base/dump-nodejs.js

@@ -0,0 +1,64 @@
+var fs = require('fs')
+var spawn = require('child_process').spawn
+var AWS = require('aws-sdk')
+var s3 = new AWS.S3()
+
+exports.handler = function(event, context) {
+  var filename = 'nodejs.tgz'
+  var cmd = 'tar -cvpzf /tmp/' + filename + ' --numeric-owner --ignore-failed-read /var/runtime'
+
+  var child = spawn('sh', ['-c', event.cmd || cmd])
+  child.stdout.setEncoding('utf8')
+  child.stderr.setEncoding('utf8')
+  child.stdout.on('data', console.log.bind(console))
+  child.stderr.on('data', console.error.bind(console))
+  child.on('error', context.done.bind(context))
+
+  child.on('close', function() {
+    if (event.cmd) return context.done()
+
+    console.log('Zipping done! Uploading...')
+
+    s3.upload({
+      Bucket: 'lambci',
+      Key: 'fs/' + filename,
+      Body: fs.createReadStream('/tmp/' + filename),
+      ACL: 'public-read',
+    }, function(err, data) {
+      if (err) return context.done(err)
+
+      console.log('Uploading done!')
+
+      console.log(process.execPath)
+      console.log(process.execArgv)
+      console.log(process.argv)
+      console.log(process.cwd())
+      console.log(__filename)
+      console.log(process.env)
+
+      context.done(null, data)
+    })
+  })
+}
+
+// /usr/bin/node
+// [ '--max-old-space-size=1229', '--max-new-space-size=153', '--max-executable-size=153' ]
+// [ 'node', '/var/runtime/node_modules/.bin/awslambda' ]
+// /var/task
+// /var/task/index.js
+// {
+// PATH: '/usr/local/bin:/usr/bin/:/bin',
+// LAMBDA_TASK_ROOT: '/var/task',
+// LAMBDA_RUNTIME_DIR: '/var/runtime',
+// AWS_REGION: 'us-east-1',
+// AWS_DEFAULT_REGION: 'us-east-1',
+// AWS_LAMBDA_LOG_GROUP_NAME: '/aws/lambda/dump-nodejs',
+// AWS_LAMBDA_LOG_STREAM_NAME: '2016/05/18/[$LATEST]85da517...0ec8b49e',
+// AWS_LAMBDA_FUNCTION_NAME: 'dump-nodejs', AWS_LAMBDA_FUNCTION_MEMORY_SIZE: '1536',
+// AWS_LAMBDA_FUNCTION_VERSION: '$LATEST',
+// LD_LIBRARY_PATH: '/lib64:/usr/lib64:/var/runtime:/var/task:/var/task/lib',
+// NODE_PATH: '/var/runtime:/var/task:/var/runtime/node_modules',
+// AWS_ACCESS_KEY_ID: 'ASIA...C37A',
+// AWS_SECRET_ACCESS_KEY: 'JZvD...BDZ4L',
+// AWS_SESSION_TOKEN: 'FQoDYXdzEMb//////////...0oog7bzuQU='
+// }

base/dump-nodejs43.js

@@ -0,0 +1,67 @@
+var fs = require('fs')
+var spawn = require('child_process').spawn
+var AWS = require('aws-sdk')
+var s3 = new AWS.S3()
+
+exports.handler = function(event, context, cb) {
+  var filename = 'nodejs4.3.tgz'
+  var cmd = 'tar -cvpzf /tmp/' + filename + ' -C / ' +
+    '--exclude=/proc --exclude=/sys --exclude=/tmp/* --exclude=/var/task/* ' +
+    '--numeric-owner --ignore-failed-read /'
+
+  var child = spawn('sh', ['-c', event.cmd || cmd])
+  child.stdout.setEncoding('utf8')
+  child.stderr.setEncoding('utf8')
+  child.stdout.on('data', console.log.bind(console))
+  child.stderr.on('data', console.error.bind(console))
+  child.on('error', cb)
+
+  child.on('close', function() {
+    if (event.cmd) return cb()
+
+    console.log('Zipping done! Uploading...')
+
+    s3.upload({
+      Bucket: 'lambci',
+      Key: 'fs/' + filename,
+      Body: fs.createReadStream('/tmp/' + filename),
+      ACL: 'public-read',
+    }, function(err, data) {
+      if (err) return cb(err)
+
+      console.log('Uploading done!')
+
+      console.log(process.execPath)
+      console.log(process.execArgv)
+      console.log(process.argv)
+      console.log(process.cwd())
+      console.log(__filename)
+      console.log(process.env)
+
+      cb(null, data)
+    })
+  })
+}
+
+// /usr/local/lib64/node-v4.3.x/bin/node
+// [ '--max-old-space-size=1229', '--max-semi-space-size=76', '--max-executable-size=153' ]
+// [ '/usr/local/lib64/node-v4.3.x/bin/node', '/var/runtime/node_modules/awslambda/index.js' ]
+// /var/task
+// /var/task/index.js
+// {
+// PATH: '/usr/local/lib64/node-v4.3.x/bin:/usr/local/bin:/usr/bin/:/bin',
+// LD_LIBRARY_PATH: '/usr/local/lib64/node-v4.3.x/lib:/lib64:/usr/lib64:/var/runtime:/var/task:/var/task/lib',
+// NODE_PATH: '/var/runtime:/var/task:/var/runtime/node_modules',
+// LAMBDA_TASK_ROOT: '/var/task',
+// LAMBDA_RUNTIME_DIR: '/var/runtime',
+// AWS_REGION: 'us-east-1',
+// AWS_DEFAULT_REGION: 'us-east-1',
+// AWS_LAMBDA_LOG_GROUP_NAME: '/aws/lambda/dump-nodejs43',
+// AWS_LAMBDA_LOG_STREAM_NAME: '2016/05/18/[$LATEST]c079a84d433534434534ef0ddc99d00f',
+// AWS_LAMBDA_FUNCTION_NAME: 'dump-nodejs43',
+// AWS_LAMBDA_FUNCTION_MEMORY_SIZE: '1536',
+// AWS_LAMBDA_FUNCTION_VERSION: '$LATEST',
+// AWS_ACCESS_KEY_ID: 'ASIA...C37A',
+// AWS_SECRET_ACCESS_KEY: 'JZvD...BDZ4L',
+// AWS_SESSION_TOKEN: 'FQoDYXdzEMb//////////...0oog7bzuQU='
+// }