Add --reload flag to the serve run subcommand

[volks73]

Why are these changes needed?

I have implemented a custom watch using the watchfiles package, formerly named watchgod, for my own local development of Ray Serve applications. It would be nice if this functionality existed in the serve Command Line Interface (CLI). I have searched the GitHub issues and Ray forums. This feature has been requested off and on several times.

This PR adds the --reload flag to the serve run subcommand. It will listen for file changes in the specified working directory and redeploy the application. This is similar uvicorn and other web frameworks. It is useful for application development.

The watchfiles package is added to the requirements.txt file and the setup.py file as indicated.

Related issue number

• [Serve] Implement serve run --reload flag #26341
• [Serve] serve run --reload command #24790
• [serve] Add support for serve run command for development #16094
• Serve Deployment with Reload Option #15505

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

I have tested by creating an example folder in the project root, mkdir example && example, then creating the following example Ray Serve application, example_serve.py, based on the Key Concepts documentation:

from ray import serve

@serve.deployment
class MyFirstDeployment:
    def __init__(self, msg):
        self.msg = msg

    def __call__(self):
        return self.msg


my_first_app = MyFirstDeployment.bind("Hello World!")

I started the Ray Serve application with following command from within the example folder:

serve run --reload --working-dir=./ example_serve:my_first_app

This yields the following output:

2023-08-13 17:44:49,851 INFO scripts.py:420 -- Running import path: 'example_serve:my_first_app'.
2023-08-13 17:44:55,768 INFO worker.py:1612 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265
2023-08-13 17:44:55,772 INFO packaging.py:518 -- Creating a file package for local directory './'.
2023-08-13 17:44:55,773 INFO packaging.py:346 -- Pushing file package 'gcs://_ray_pkg_579ed46ebfd67af1.zip' (0.00MiB) to Ray cluster...
2023-08-13 17:44:55,773 INFO packaging.py:359 -- Successfully pushed file package 'gcs://_ray_pkg_579ed46ebfd67af1.zip'.
(ServeController pid=58220) INFO 2023-08-13 17:44:58,267 controller 58220 controller.py:357 - Finished recovering deployments after 0.00033402442932128906s.
(HTTPProxyActor pid=58249) INFO 2023-08-13 17:44:59,445 http_proxy 192.168.1.57 http_proxy.py:904 - Proxy actor 8f51acf84b79d0e8004b4c9901000000 starting on node 3f223d4ed49dce22ca1474482841f62b3e232754755644273ba0750f.
(ServeController pid=58220) INFO 2023-08-13 17:44:59,501 controller 58220 deployment_state.py:1358 - Deploying new version of deployment default_MyFirstDeployment.
(HTTPProxyActor pid=58249) INFO:     Started server process [58249]
(ServeController pid=58220) INFO 2023-08-13 17:44:59,604 controller 58220 deployment_state.py:1628 - Adding 1 replica to deployment default_MyFirstDeployment.
2023-08-13 17:45:01,512 SUCC scripts.py:461 -- Deployed Serve app successfully.

Then, I change the "Hello World" to "Hello Me", and I get the following output:

2023-08-13 17:45:13,460 INFO scripts.py:476 -- Files changed. Redeploying Serve app.
(ServeController pid=58220) INFO 2023-08-13 17:45:13,503 controller 58220 deployment_state.py:1358 - Deploying new version of deployment default_MyFirstDeployment.
(ServeController pid=58220) INFO 2023-08-13 17:45:13,614 controller 58220 deployment_state.py:1511 - Stopping 1 replicas of deployment 'default_MyFirstDeployment' with outdated versions.
(ServeController pid=58220) INFO 2023-08-13 17:45:15,942 controller 58220 deployment_state.py:1628 - Adding 1 replica to deployment default_MyFirstDeployment.
^C2023-08-13 17:45:57,611       INFO scripts.py:485 -- Got KeyboardInterrupt, shutting down...
(ServeController pid=58220) INFO 2023-08-13 17:45:57,692 controller 58220 deployment_state.py:1653 - Removing 1 replica from deployment 'default_MyFirstDeployment'.

[edoakes]

@volks73 thanks so much for adding this, it'll be a great UX improvement!

I left a few small comments on naming/behavior.

In addition, could you please add a test case to python/ray/serve/tests/test_cli.py? Should be of the form: start background process running serve run with reload, check app is deployed and returns initial response, update file in the working dir to return a new response, wait_for_condition that the app returns the new response.

[edoakes]

@volks73 in addition, you'll need to sign off all commits in the PR using the -s flag to git commit (that's why the "DCO" build is currently failing).

[volks73]

@edoakes I have signed the previous and current commits. Sorry about that.

I have added a test to the python/ray/serve/tests/test_cli_2.py file instead of the test_cli.py file because I saw all of the serve run tests are in the test_cli_2.py file. I am not sure if the test is correct because I added a fixture to use a temporary directory for the working directory and change to that directory.

[edoakes]

Test looks great, thanks! Just some small nits then LGTM.

[edoakes]

Test case is failing, pointed out what looks like the issue:
https://buildkite.com/ray-project/oss-ci-build-pr/builds/31759#0189f509-1af0-4fa7-be7c-320a541be7ac/6-2237

[edoakes]

@volks73 I pushed a commit to the branch. Had to fix a few issues:

• The app object wasn't being reconstructed when we picked up file changes.
• In addition to ^, the module itself needed to be reloaded using importlib.
• In the test, the updated file write was happening before the serve run command was in the file watching loop. Added a sleep, couldn't think of a better way to guarantee this :(

With these changes the test is passing locally and manual testing worked as well.

[volks73]

@edoakes Great! Thank you for fixing the test and the app reloading. I don't know of another solution than sleeping for a while either.

[edoakes]

Doc build failed due to missing dependency, hopefully fixed 🤞

[volks73]

@edoakes I am having trouble following the tests. Did updating the dependencies for the docs work?

[edoakes]

@volks73 all the relevant tests are passing (we have some typical flakiness in CI). Just need one more approval and should get merged in time for 2.7 branch cut!

[richardliaw]

move the dependency to serve instead of default?

[volks73]

@edoakes I have moved the watchfiles dependency to the serve dependencies as requested.

[richardliaw]

approving from docs side

[edoakes]

@volks73 merged 🎉 thanks again for contributing. This has been on the backlog for a long time and happy to see it done :)

[volks73]

@edoakes Your welcome and thank you for helping with the PR and overall for the Ray project.