Syncing local python project files with remote Jupyter server

[brunocous]

Feature: Notebook Editor, Interactive Window, Python Editor cells

Description

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer

Context

VSCode allows users to connect with a running remote Jupyter server (https://code.visualstudio.com/docs/python/jupyter-support#_connect-to-a-remote-jupyter-server). Using the Jupyter API it is able to start a kernel and execute cells of a locally saved notebook remotely.

Use case/problem

If you want to for example call a function from another python file (foo.py) from that notebook (local-notebook.ipynb), the remote kernel can't access local files. The remote kernel can access other files saved on the remote notebook server. The problem is that local (Python) files are not synced with the remote Jupyter server instance, for the remote Python interpreter (kernel) to access them.

Existing solutions

The standard way of achieving this is through rsync over SSH. However this default requires managing a SSH connection and SSH keys (which large entreprises servers not necessarily allow).
There are workarounds (manually uploading files through the notebook UI, and using git), but these inhibit development and iteration speed.

Proposal

Extend the VSCode Python extension to allow users to sync files with a remote running Jupyter notebook server. Under the hood, the Jupyter contents API can be used for this:

• https://jupyter-notebook.readthedocs.io/en/stable/extending/contents.html,
• https://github.com/jupyter/jupyter/wiki/Jupyter-Notebook-Server-API.

Authentication and authorization is handled through the API token that you need anyway to connect.

No SSH, git or manual hassle required.

Additionally, you can execute your local code (by calling it through the notebook) remotely without having to manage a remote Python SSH interpreter, or docker images. All you need is a running jupyter notebook.

[IanMatthewHuff]

@brunocous Thanks for the detailed suggestion. I do feel that this would be an interesting suggestion to consider. We'll discuss it at our triage meeting.

[brunocous]

Any news on this, or how can I help?

[brunocous]

Btw, there are other workarounds to this when you use a cloud provider anyway in your project (S3, data storage, blob,...).

For example, using AWS S3 you could perform aws s3 sync to sync local files to an S3 bucket. And then use a jupyterplugin(like https://github.com/uktrade/jupyters3) to sync that S3 bucket with Jupyter. Something similar probably exists for other cloud providers

[CmdQ]

I'd find this feature highly useful. The other ways are always cumbersome.

[fra-luc]

I'd like this very much as well.

[GF-Huang]

Any progress?

[rchiodo]

This is something we're investigating. Not sure if or when we'll release it though.

[205g0]

Just found this issue and indeed, this would be a feature making the entire experience more round.

The Jupyter remote server is on abstract level just a dumb number cruncher I use because:

• I want to have clear separation between my local dev machine and an external TPU machine, latter is rarely for development but just for the model creation and for having a reproduceable production system; besides, both machines couldn't be more different, the TPU machine is just about TFLOPS and a solid server and production Ubuntu while my local dev machine is about hipdi support, mobility, window management, etc.
• Because of the reproducibility I want to have also all the external imports on that remote machine
• But everything which comes from me, locally, from my current dev efforts should be locally/from my dev machine which I use to create code

The current separation is jarring (one file local but rest remote) and breaks an unmatched feature of VS Code.

Would love to see an update on this

[brunocous]

I'm already pleased that the VSCode team is even considering this feature. I dropped the same feature request for Pycharm some time ago, but not a single response or action was taken (https://youtrack.jetbrains.com/issue/PY-42649). So good job VS code team!
Even this feature gets a green light, it will take some non-trivial dev effort. For now, there are some suboptimal workarounds that gives you the same result (rsync, scripts that do something with git, syncing with any cloud blob or object store, etc).