Philosophy and goals of the project going forward?

[creatorrr]

What is the project's philosophy and goals going forward? We recently started using cortex for managing NLU+ML workloads and love the tech you guys have built. Kudos to the cortex team for the impressive work!

We are using the 0.35.0 version currently and the 0.36.0 release departs significantly from it. I couldn't find any relevant thread but it'd be very helpful to have a section of docs dedicated to goals and philosophy of the project. Specifically, I'd like to know:

1. What was the motivation behind moving away from the handler interface to the containers one?
2. Are there any plans to re-introduce a similar abstraction in the future?
3. Loading and caching models from s3 was particularly useful for us, any plans on supporting it in the future?
4. What are plans for supporting <=0.35.0 releases with the handler API?

[deliahu]

Hey @creatorrr thanks for reaching out!

1. There were many factors that went into this decision, since as you mentioned, it is a significant API change. The main motivation was for us to focus on the aspects where we feel that we provide the most value (infrastructure automation), while at the same time allowing for the most flexibility in terms of what kinds of workloads that can run on Cortex. There are many ways to serve a model within a container (TensorFlow Serving, TorchServe, Triton, standard Python servers like Flask/FastAPI, etc), all which have their own advantages and disadvantages, and many of which are not written in Python. By making our interface agnostic, we’re moving to a more standard and portable interface, as well as allowing for more flexibility and use cases. I’d be happy to discuss this further if you’re interested in chatting.

2. There are not currently plans to bring back the Python interface.

3. AWS credentials still exist within the container’s runtime, so it is possible to download a model from s3 when your server initializes. Here is an example from our test folder (you would download your model in def startup): https://github.com/cortexlabs/cortex/blob/0.36/test/apis/realtime/text-generator/app/main.py . Also, some model servers (like TensorFlow Serving) support downloading models directly from s3. I’d be happy to jump on a call to help you more specifically with anything you get stuck on with the migration.

4. Our preference is to help you migrate to the new API. We will continue to help with simple questions or small tasks on the old version, but our main focus is on supporting the new API.

Feel free to email me at david@cortex.dev or DM me on the community Slack if you want to set up a time to chat about our thought process with the product updates or for help with migrating your APIs!

[deliahu]

I'll go ahead and close this issue since we had our conversation. Feel free to reach out if you have any questions as you migrate to the new version!

[lminer]

Just to add my two cents. I was super happy with the old API as it abstracted away a lot of the pain around serving and managing models (for example being able to update models without any downtime). It's a shame to see so many useful features being discarded. Taking a look at the new API, the rewrite that we would have to do is so extensive that we're considering moving to a different service.

[deliahu]

@lminer Thanks for following up. Yes, I do understand how some of the features that were removed were useful for you. Please see my response above for some of our thoughts that went into this decision, and I'd be happy to chat live if you'd like to hear more about our thought process and/or would like tips for migrating to the new version (should you choose to try it).

Regarding updating models without any downtime, this is still possible with Cortex today. There are two approaches:

1. You can bake your model into your Docker image, and then build a new image when you want to do a rolling update. If you reuse the same Docker tag (e.g. my-repo/my-model:latest), you would use the cortex refresh command, and if you change the docker tag (e.g. my-repo/my-model:v2), then you can update the image field in the api configuration file and run cortex deploy. Generally I would recommend updating the image tag (you would have a tag for each version of your model), since that makes it easier to roll back should you need to.

2. You can download your model from s3 during container initialization; if you use a model server which supports automatic reloading from s3 (e.g. TensorFlow Serving or Triton), then you would just upload a new version of your model to your s3 bucket, and the model server will automatically load the new version. We take this approach in our Inferentia test api (see the --model_base_path).

Option 2 is nice for development, but in production I would recommend going with option #1 since there are fewer moving parts, it's less likely to make mistakes when updating your models, APIs will likely be able to scale up faster, and you'd have a complete versioned "servable unit" for your model which does not rely on external resources.

I'd be happy to dive into this (or anything else) further if you're interested, feel free to email me at david@cortex.dev or DM me on the community Slack.

[wise-east]

I also agree that the previous versions (<=0.35) were better suited for my use case and was much easier to use, but I understand why the move to being language agnostic makes more sense in the longer run.

It would be nice though, to show how to set up previous example APIs using cortex with versions >0.35. I am not that great with Docker and it seems like the documentation has become very succinct on how to set up APIs with the new version. This would help me transition to the newer versions and also I am sure it will be helpful for many others who have used the previous versions of cortex.

[deliahu]

@wise-east There are some examples in our test folder, which could be helpful: https://github.com/cortexlabs/cortex/tree/master/test/apis

In terms of the Docker workflow, I agree that we could help show some examples for how to do this. For any of our examples, it should be just two commands; from a directory with a Dockerfile, it would be:

docker build . -f <dockerfile_path> -t <image_url>

followed by:

docker push <image_url>

<dockerfile_path> is the name of a Dockerfile (e.g. cpu.Dockerfile), and <image_url> is the URL for your image repository (e.g. 764403040460.dkr.ecr.us-west-2.amazonaws.com/cortexlabs/realtime-sleep-cpu for EKS, docker.io/cortexlabs/realtime-sleep-cpu for Docker Hub, or quay.io/cortexlabs/realtime-sleep-cpu for quay).