Deploy machine learning models in production

Cortex is an open source machine learning deployment platform that makes it simple to deploy your machine learning models as web APIs on AWS. It combines TensorFlow Serving, ONNX Runtime, and Flask into a single tool that takes models from S3 and deploys them as web APIs. It also uses Docker and Kubernetes behind the scenes to autoscale, run rolling updates, and support CPU and GPU inference. The project is maintained by a venture-backed team of infrastructure engineers with backgrounds from Google, Illumio, and Berkeley.

install • docs • examples • we're hiring • email us • chat with us

Key features

• Minimal declarative configuration: Deployments can be defined in a single cortex.yaml file.

• Autoscaling: Cortex automatically scales APIs to handle production workloads.

• Multi framework: Cortex supports TensorFlow, Keras, PyTorch, Scikit-learn, XGBoost, and more.

• Rolling updates: Cortex updates deployed APIs without any downtime.

• Log streaming: Cortex streams logs from your deployed models to your CLI.

• Prediction monitoring: Cortex can monitor network metrics and track predictions.

• CPU / GPU support: Cortex can run inference on CPU or GPU infrastructure.

How it works

Define your deployment using declarative configuration

# cortex.yaml

- kind: api
  name: my-api
  model: s3://my-bucket/my-model.onnx
  request_handler: handler.py
  compute:
    gpu: 1

Customize request handling

# handler.py

# Load data for preprocessing or postprocessing. For example:
labels = download_labels_from_s3()


def pre_inference(sample, metadata):
  # Python code


def post_inference(prediction, metadata):
  # Python code

Deploy to AWS using the CLI

$ cortex deploy

Deploying ...
http://***.amazonaws.com/my-api  # Your API is ready!

Serve real-time predictions via autoscaling JSON APIs running on AWS

$ curl http://***.amazonaws.com/my-api -d '{"a": 1, "b": 2, "c": 3}'

{ prediction: "def" }

Installation

# Download the install script
$ curl -O https://raw.githubusercontent.com/cortexlabs/cortex/0.8/cortex.sh && chmod +x cortex.sh

# Install the Cortex CLI on your machine: the CLI sends configuration and code to the Cortex cluster
$ ./cortex.sh install cli

# Set your AWS credentials
$ export AWS_ACCESS_KEY_ID=***
$ export AWS_SECRET_ACCESS_KEY=***

# Configure AWS instance settings
$ export CORTEX_NODE_TYPE="m5.large"
$ export CORTEX_NODES_MIN="2"
$ export CORTEX_NODES_MAX="5"

# Install the Cortex cluster in your AWS account: the cluster is responsible for hosting your APIs
$ ./cortex.sh install

See installation instructions for more details.

Examples

• Text generation with GPT-2

• Sentiment analysis with BERT

• Image classification with Inception v3