Support local py_modules in jobs

[architkulkarni]

Why are these changes needed?

Related issue number

Previously local py_modules were not uploaded to the remote cluster during job submission, so jobs could only use py_modules from remote URIs. This PR uploads py_modules to the cluster, just like how it currently works for the non job-submission case.

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[edoakes]

This looks great, it's basically what I suggested on the other PR a few minutes ago :)

I think we should take it a step further and do the same for working_dir.

Also, it seems to me that the callback should simply take a local zip file (or bytes?) and return a URI rather than needing to implement its own handling for include_parent_dir and excludes.

[architkulkarni]

Also, it seems to me that the callback should simply take a local zip file (or bytes?) and return a URI rather than needing to implement its own handling for include_parent_dir and excludes.

@edoakes I agree that should be the final state. How about we merge this first, and then I make a followup issue for that refactor? I can also try to do it in #22368, since I need to support uploading local files/bytes anyway for that one.

[architkulkarni]

serve:test_cli is flaky on master.

[edoakes]

@architkulkarni merge conflict from @shrekris-anyscale revert and subsequent revert-revert :(

[architkulkarni]

tests:test_actor_client_mode, Windows tests:test_basic_2_client_mode, and doc build (concurrency limit) broken on master. Mac builds are stalled on master