VPC Security, Docker Daemon and Papertrail Changes

VPC Security Enhancements

The Rack VPC Security Group no longer exposes the SSH port to the world (0.0.0.0/0). It now accepts traffic only from inside the Rack VPC, which is sufficient because of the convox instances ssh websocket tunneling.

Thanks @beedub for the security improvement!

For more info see PR #716: Change ssh port sg to use vpc cidr, not 0.0.0.0.

Docker Daemon Changes

This week we discovered that Docker 1.11 was surprisingly showing up in production environments, and causing convox run to errantly exit non-zero in some cases.

The new update no longer applies updates on boot for anything but critical system security, effctively pinning Docker back to 1.09.

We will update Docker to 1.11 along with new ECS AMIs in the near future.

This update also changes the Docker Daemon limit for number of open files from 1024 to 1024000.

@awsmsrc found a great suggestion for a higher limit by @pahud on Medium as a much better setting for modern web applications.

For more information see PR #704 Up ulimit for dockerd and PR #728 dont upgrade all packages.

Papertrail Deprecation

In anticipation of removing Kinesis to save us all money this update blocks convox service create papertrail.

Now you should use convox services create syslog --url tcp+tls://logs1.papertrailapp.com:12345 to forward logs from CloudWatch Logs to Papertrail via secure syslog. See the Syslog Docs for more information.

For more information see PR #719 Syslog bugfixes and Papertrail deprecation.

Scale Down to Delete Unneded ELBs

@mwarkentin reported a couple bugs in the new feature that removes unneeded ELBs. You can now run convox scale redis --count=-1 to remove ELBs, even if they are linked to another process. You can also still convox run redis bash even if the process type is not running.

For more information see PR #721 and #726

CPU Metrics; ECR in Ireland; Scale Down To Remove Unneeded ELBs

Rack CPU Utilization

convox instances was incorrectly calculating 0% CPU Utilization based off the ECS Instances CPU reservation information. We now pull the proper CPU Utilization Average value directly from CloudWatch Metrics:

$ convox instances
ID          AGENT  STATUS  STARTED     PS  CPU    MEM  
i-3a0de2bc  on     active  4 days ago  3   1.27%  7.21%
i-ff44b863  on     active  5 days ago  2   1.66%  6.41%
i-d1ec644b  on     active  3 days ago  0   0.29%  0.00%

The result of expertly wrapping these low level AWS services is a Convox REST API that is really easy to work with for monitoring and automation:

$ convox api get /instances
[
  {
    "agent": true,
    "cpu": 0.0123,
    "id": "i-3a0de2bc",
    "memory": 0.07212622088655146,
    "private-ip": "10.0.2.7",
    "processes": 3,
    "public-ip": "54.83.186.197",
    "started": "2016-05-27T21:31:16Z",
    "status": "active"
  },
  {
    "agent": true,
    "cpu": 0.016819999999999998,
    "id": "i-ff44b863",
    "memory": 0.0641121963436013,
    "private-ip": "10.0.1.50",
    "processes": 2,
    "public-ip": "54.210.12.112",
    "started": "2016-05-27T08:51:08Z",
    "status": "active"
  },
  {
    "agent": true,
    "cpu": 0.0028399999999999996,
    "id": "i-d1ec644b",
    "memory": 0,
    "private-ip": "10.0.3.191",
    "processes": 0,
    "public-ip": "52.91.82.58",
    "started": "2016-05-28T20:39:19Z",
    "status": "active"
  }
]

EC2 Container Registry In Ireland (eu-west-1)

AWS released its highly scalable, high performance Docker image registry service in Ireland a few weeks ago. Convox users in eu-west-1 will start using it after the next convox rack update and two app deploys.

We expect this to be faster and more reliable than the legacy private registry running in your rack. One more region to go (Tokyo ap-northeast-1) before we can remove the legacy private registry for good!

Scale Down to Delete Unneeded ELBs

If you scale a web process down to 0, convox retains the ELB so you will keep the same hostname if and when you scale a web process back up to 1 or more.

This has an unfortunate side effect of costing some of us $18/mo for an ELB we may not be using. Now you can scale a web process down to -1 to de-provision the ELB and save money:

$ convox scale web --count=-1
NAME  DESIRED  RUNNING  MEMORY
web   -1       0        256 

$ convox apps info
Name       httpd
Status     running
Release    RDJQJEUAPTP
Processes  web
Endpoints  :80 (web)

At any time you can scale back up to 0 or more to create a new ELB. The hostname will be different but your service will be back online in minutes.

Local Development and Networking improvements

Bidirectional sync

Convox is thrilled to announce 2-way sync support in convox start. One-way, host-to-container sync has been present for quite a while, but with this release, file changes on the container will also be synced back to the host. This is extremely powerful for development workflows in frameworks like Rails that generate lots of code.

By default, any file or directory that appears in an ADD or COPY directive in your Dockerfile will be synced.

Run convox update to get the newest CLI including this feature.

#640

Thanks to @mwarkentin for design input.

Race condition fix

This release includes a fix for https://github.com/convox/rack/issues/679 where convox start could fail on containers with links if the container IP was attempted to be fetched before hostnames propagated.

Thanks to @sotte for the bug report.

VPC CIDR improvements

Convox created services (such as PostgreSQL and Redis) now automatically respect the VPC CIDR settings on the Rack in which they're created.

#690

Dev / Prod / Teardown Improvements

Dev Environment Supports More Project Types

@mattmanning is working hard on improving the convox development environment.

In this release, convox start has more smarts for projects with no Dockerfile / docker-compose.yml, and projects with a Procfile. Use convox update to update the CLI for these enhancements.

Thanks @prognostikos for contributions.

Rack Instance Count Validation

A rack requires at least 2 instances for:

• Running the rack API in a redundant fashion
• Spare capacity for a rolling deploy of a web process
• Service uptime or fast recovery when an instance dies.

So if you try running a rack with a single instance, strange side effects happen. Therefore we now prevent the ability to install or scale down a rack to a single instance.

Thanks to @markpundsack for a bug report, and @kmonkeyjam for a bug report and a patch!

Uninstall Robustness

Based on lots of feedback @nzoschke made convox uninstall much more robust. It now:

• Requires an explicit rack name and region preventing accidental rack deletion
• Deletes all services and apps automatically
• Retries deleting every service, app and rack to work around transient errors
• Empties and deletes S3 buckets

We hope that this makes Convox even easier to experiment with.

CLI Improvements; Deployment Parameters

CLI Improvements

convox init

Work continues on convox init improvements with .dockerignore generation. (#657)

convox start

You'll now receive a friendlier error from convox start if you don't have Docker running. (#660)

You can now pass a --shift argument to convox start to shift all of its port allocations by a certain amount. This can help with running more than one application at the same time. (#662)

$ convox start
$ curl https://localhost:443/

$ convox start --shift 10000
$ curl https://localhost:10443/

Thanks to @mwarkentin for help talking through the design in #484.

Thanks to @dbeard for fixing the COPY/ADD parsing during code synchronization. (#629)

The proxy in convox start has been updated to work with Docker for Mac. The .protocol, .proxy, and .secure labels for ports are now handled correctly. (#648, convox/proxy#3)

Deployment Parameters

You can now configure the ECS deploymentConfiguration using two new app parameters, DeploymentMinimum and DeploymentMaximum to tweak the behavior of rolling deployments. (#641)

$ convox apps params set DeploymentMinimum=50 DeploymentMaximum=150

Other Bugfixes

• Credentials in Convox services are now URL encoded rather than Base 64 encoded. (#651)

SSL; Uninstall Improvements ; Bugfixes

Free SSL Certs Everywhere

All Convox regions -- us-east-1, us-west-2, eu-west-1, ap-northeast-1 -- now can take advantage of the AWS Certificate Manager for free SSL certs:

$ convox certs generate foo.example.org
Requesting certificate... OK, acm-01234567890

Thank you @rcaught for enabling to secure all our services in #645.

Improved API Validations

It is now much more clear to users what syslog protocols are supported:

$ convox services create syslog --url http://example.com:11234
ERROR: Invalid url scheme `http`. Allowed schemes are `udp`, `tcp`, `tcp+tls`.

It is also more clear that the you can not name an app the same as the rack:

$ convox apps create convox
Creating app convox... ERROR: application name cannot match rack name (convox). Please choose a different name for your app.

Thanks to @prognostikos for #632 and #646.

Uninstall Improvements

@keshavab contributed some really nice improvements to make the uninstall IAM prompts more clear and robust:

$ convox uninstall --stack-name=staging --region=us-east-1
     ___    ___     ___   __  __    ___   __  _
    / ___\ / __ \ /  _  \/\ \/\ \  / __ \/\ \/ \
   /\ \__//\ \_\ \/\ \/\ \ \ \_/ |/\ \_\ \/>  </
   \ \____\ \____/\ \_\ \_\ \___/ \ \____//\_/\_\
    \/____/\/___/  \/_/\/_/\/__/   \/___/ \//\/_/


This installer needs AWS credentials to install/uninstall the Convox platform into
your AWS account. These credentials will only be used to communicate between this
installer running on your computer and the AWS API.

AWS Access Key ID: 
AWS Secret Access Key: 

@nzoschke also restored the Uninstall doc in https://github.com/convox/site/pull/151.

Keep the feedback coming on ways we can improve the uninstall experience. You can expect more improvements here in the future.

Improved init system and Rails defaults, better CLI messages

convox init improvements

This release includes internal improvements for convox init, the command that inspects your application and generates Docker assets.

Also included are improved default Dockerfile and docker-compose.yml files for Rails apps that use updated Ruby and Rails Docker images provided by Convox.

Better CLI messages

You will now see a more descriptive "app is still being created" message from the CLI if you try to run convox env on an app that's still being created.

This Rack release is associated with CLI release 20160514180126. To update your CLI, run convox update.

A more helpful error message is now displayed if you try to run a convox run command against a nonexistent process.

$ convox ps
ID            NAME  RELEASE      SIZE  STARTED     COMMAND
1bce410b52b6  web   RNRSDLEBTIP  256   6 days ago  /bin/web

$ convox run foo ls
ERROR: Unknown process name: foo

Thanks to Miguel Moll for his open-source contributions to this release. If you're interested in getting involved, check out our open issues and join #dev on the Convox Public Slack.

Updating

To get these updates run convox update && convox rack update. If you're not already using Convox, get started at console.convox.com.

CloudFlare Origin Certs, CLI help and bash autocompletion

CloudFlare Origin Certs

The free SSL cert trend continues! With #599, CloudFlare origin certs are now properly parsed and uploaded to IAM:

$ convox certs create mixable.net.pem mixable.net.key 
Uploading certificate... OK, cert-1462483771

$ convox certs
ID                         DOMAIN                                        EXPIRES          
cert-1462483771            CloudFlare Origin Certificate                 15 years from now

Thanks to @beedub for the issue.

CLI Autocomplete

The convox command now has Bash autocompletion. There are a few ways to enable this, per the codegangsta/cli docs.

To set it up on OS X with Homebrew, first source the bash completion initializer in your bash profile, then add a autocomplete helper for the convox command:

$ cat <<EOF >~/.bash_profile
if [ -f $(brew --prefix)/etc/bash_completion ]; then
  . $(brew --prefix)/etc/bash_completion
fi
EOF

$ curl -o $(brew --prefix)/etc/bash_completion.d/convox https://raw.githubusercontent.com/codegangsta/cli/master/autocomplete/bash_autocomplete

Then open a new tab, and try convox or convox builds followed by the key:

$ convox
api         builds      env         help        instances   proxy       racks       run         ssl         uninstall   
apps        certs       exec        init        login       ps          registries  scale       start       update      
build       deploy      h           install     logs        rack        releases    services    switch    

$ convox builds
copy    create  delete  h       help    info 

Thanks to @mwarkentin for the nudge on Slack.

CLI Help

All CLI commands should consistently honor the -h or --help flag now.

$ convox builds -h
convox builds: manage an app's builds

Usage:
  convox builds <command> [args...]

Subcommands: (convox builds help <subcommand>)
  create    create a new build
  copy      copy a build to an app
  info      print output for a build
  delete    Archive a build and its artifacts
  help, h   

Options:
  --app, -a             App name. Inferred from current directory if not specified.
  --help, -h            show help
  --generate-bash-completion    

$ convox builds create --help
convox builds create: create a new build

Usage:
  convox builds create 

Options:
   --app, -a                App name. Inferred from current directory if not specified.
   --no-cache               pull fresh image dependencies
   --incremental            use incremental build
   --file, -f "docker-compose.yml"  path to an alternate docker compose manifest file
   --description            description of the build

Thanks to @beedub for the issues.

Amazon Linux 2016.03

On April 5th, Amazon sent this announcement:

We recently released a new Amazon ECS-optimized AMI that includes Amazon Linux 2016.03 and addresses an issue that causes the agent to stop accepting incoming requests.

This update starts using the amzn-ami-2016.03.a-amazon-ecs-optimized AMIs that are listed on the Launching an Amazon ECS Container Instance docs

Generally Convox is fast to follow AMI updates. However the upgrade from Amazon Linux 2015.09 to 2016.03 had system changes that caused some problems with the Convox UserData around restarting the Docker daemon to pick up custom settings. We now configure the instance with CloudInit which offers the ability to write custom Docker settings to the disk earlier in the boot process so they are there when the AMI does the first Docker start.

This is all for your information only. This is precisely the type of work we think Convox should sort out so you don't have to.

Thanks a ton to @bobzoller for contributing the original patch and @mwarkentin for lots of testing.

SwapSize Parameter

Swap is now offered on a dedicated 5GB EBS volume. Advanced users can tune the amount of swap with the SwapSize Parameter.

Prior to this swap was a file on the root partition. In some cases this contributed to instance failures when the root partition is 100% full.

Most applications should not use swap, but we have seen some large Rails and Python apps benefit greatly from having swap available. To enable swap for an application, run convox env set SWAP=1.

Note that ECS does not not yet support swap (GitHub issue) but Convox does!

InstanceRunCommand Parameter

Convox offers an experimental parameter, InstanceRunCommand, that enables you to customize your cluster without burning custom AMIs.

You can use this to install custom packages on every instance:

$ convox rack params set InstanceRunCommand="yum install -y htop"

Run a Docker container on every instance:

$ convox rack params set InstanceRunCommand="docker run -d --rm logstash logstash -e 'input { stdin { } } output { stdout { } }'"

This can be extended to even setup complex monitoring tools like DataDog on every instance:

$ convox rack params set InstanceBootCommand="\"echo '**** START BOOTSTRAP ***' ; curl -H 'Authorization: token <token>' -H 'Accept: application/vnd.github.v3.raw' -o /tmp/bootstrap.sh -L https://raw.githubusercontent.com/user/repo/master/amazon-linux-ecs-bootstrap.sh ; DATADOG_API_KEY=<key> bash /tmp/bootstrap.sh\""

You do need to be careful about string and shell encoding on with this parameter.

Thanks to @mwarkentin for motivation and testing.

ACM Region Check

This update improves error report on convox certs create in regions that do not have AWS Cert Manager available.

Convox Service Proxy

This update offers a proxy utility to help securely access your Convox services like Postgres from your laptop:

$ convox services proxy postgres-1833
proxying 0.0.0.0:5432 to staging-postgres-1833.jrfuyrifhdjk.us-east-1.rds.amazonaws.com:5432

$ psql -h localhost
Password: 

Minor bugfixes, Linking

Bugfixes

This release includes 2 bugfixes;

• Handle environment section in map format #578
• fix check for proxy locally #583

Linking

This release introduces a change in the way environment variable names are generated using container linking (https://convox.com/docs/linking/). Starting in this release, all hyphens in process names will be converted to underscores in environment variables.

For example, a process named my-service will inject the environment variable MY_SERVICE_URL instead of MY-SERVICE_URL into linked containers.

#582