feat: DH-19715: Python remote file source plugin

[bmingles]

DH-19715: Python remote file source plugin

• Python plugin that can be installed in a DH Core / Core+ worker
• Registers custom sys.meta_path finder and loader that will request source code from a remote client
• The plugin source includes a NodeJS based test client + a test workspace, but ultimately this plugin will be sourced by the VS Code extension (feat: DH-19715: VS Code - Remote Python execution vscode-deephaven#253)

Testing

There is a test Python workspace in plugins/python-remote-file-source/test-node-client/workspace. There are 2 test files that get run by the vitest test harness:

• test_relative_imports.py
• test_absolute_imports.py

Running the Tests

In the plugins repo root:

1. Install dependencies

# Install js dependencies
nvm install
npm install

# Setup python venv and install dependencies
python -m venv .venv
source .venv/bin/activate
pip install --upgrade -r requirements.txt
pip install click watchdog deephaven-server

2. Build the python-remote-file-source plugin

python tools/plugin_builder.py python-remote-file-source

3. Start DH server

Note that the -Dprocess.info.system-info.enabled=false is only needed for Mac M1/M2.

python tools/plugin_builder.py \
 python-remote-file-source \
 --server \
 --server-arg \
 --jvm-args="-Dauthentication.psk=plugins.repo.test -Dprocess.info.system-info.enabled=false"

4. Before running tests for the first time, install dependencies for the test client:

cd plugins/python-remote-file-source/test-node-client
nvm install
npm install
cd -

5. Run nodejs tests

npm run test:remote-file-source

VS Code Extension

There is a .vsix attached to the DH-19715 that can be installed in VS Code if you want to see how the plugin integrates with VS Code.

[cpwright]

I've looked at the Python and mostly skipped over the JS/TS. I've left micro-comments, but this is more of a macro design comment that you guys can think about. I might be misreading how this works as well.

One thing that I think we should consider is that the pipe to your Deephaven server instance is often narrow/has high latency. It may make sense to push more data to the worker rather than pulling it and being smarter on the VS Code side. In particular, from your demo I like how you have the ability to add several paths to the worker's import space.

I think that each of those top-level names would form a RemoteMetaPathFinder. if that is true, then we do have to request information from the client on each find_spec. Even when we are negative, the counterpoint ist hat negative requests in Python might be seemingly rare.

Instead of just sending the worker the top-level list of names, you could also send them the list of files that exist within each name; or maybe even contents. That would eliminate the need for "in-line" requests, because the worker could look things up in a cache.

It would however, also require significantly more front-end work; because we would need to watch the file system and send the server updated information whenever the watched file system paths change.

As I write this down, I think I am more convinced this is probably premature optimization for Python; and we can leave this as is.

[mofojed]

A few cases on the server where we should just report the error and kill the connection.
A couple of questions why we're not using existing JSON RPC libraries instead of rolling our own functions/types.
Looks good overall though

[mofojed]

Not that there's anything wrong with this, but in deephaven.ui we used the json-rpc library for handling requests: https://pypi.org/project/json-rpc/
Handles passing methods off to a dispatcher and adding the appropriate ID to responses, etc. Just thinking there might be some edge cases that aren't handled correctly, such as notifications, but shouldn't matter for this case I don't think...

[bmingles]

I didn't realize we were already using a json-rpc library. I just opted not to bring in a dependency since the number of message types involved is very small and it's a pretty simple schema.

[mofojed]

Why not use the json-rpc-2.0 library like we use in deephaven.ui?

[bmingles]

I had just avoided the extra dependency since number of message types is small. Also didn't realize we were using a library already to set some precedence.