Customize dependency search path in cortex.yaml

[langep]

Description

Add the ability to specify the search path for dependency files such as requirements.txt, conda-packages.txt or dependencies.sh inside the cortex.yaml configuration.

Motivation

We want to be able to use different cortex.yaml configurations each with their own dependencies in the same code base. Our plan is to setup the active at the root of the repository and point to the predictor via the predictor.path setting. However, this would require that we also copy all the dependencies files to the root.

Adding an additional config option inside cortex.yaml to specify the search path relative to the cortex root similar to python_path would work for us. The new option could be called dependency_path.

[deliahu]

@langep Yes I think this is a great idea. We actually have a ticket for this (#1191), but I'll go ahead and close that one as a duplicate since your motivation section is more detailed.

Here is the proposed design from the other ticket, does it sound reasonable to you?

- name: my-api
  predictor:
    dependencies:
      pip: alternative-requirements.txt
      conda: alternative-conda-packages.txt
      shell: alternate-dependencies.sh

[langep]

Yes that design would work great, assuming that the individual paths can be relative to the root e.g. pip: some-subdir/alternative-requirements.txt.

[deliahu]

Yes, we would support nested paths. We'll keep you posted on our priority for this!

In the meantime, there could be a somewhat hacky workaround that might work: in theory it should be possible to set an environment variable via the env field in the predictor section of the API config, and use it in dependencies.sh to determine what commands to run.

E.g.

# cortex.yaml

- name: my-api
  predictor:
    env:
      api_name: my-api

# dependencies.sh

if [ "$api_name" == "my-api" ]; then
  pip install some-subdir/alternative-requirements.txt
fi

Another approach (which would have some added benefits as well) would be to build a custom docker image containing your dependencies (docs). The benefits of this approach are that it's faster to download from ECR/GCR than from quay.io, you would remove a dependency on quay.io (which could have downtime), and there would be no need to re-install dependencies on each api replica init().

[langep]

Thank you @deliahu, I was thinking about the dependencies.sh hack as well but didn't have a chance to testing if the environment variables from the API config are set at the time dependencies.sh runs. Thanks for confirming that they are in fact set at the time.

[deliahu]

@langep sounds good, let me know if you run into any issues or have any questions about the workaround