Wrapping existing language server(s) in websockets, virtual file systems

[bollwyvl]

Some semi-related questions:
Are there any examples or documentation of wrapping/extending pygls to:

1. ... serve LSP over websockets?
2. ... to wrap existing LS implementations?
3. ... use non-traditional filesystems and formats?

Background

We've been looking at bringing LSP to the Jupyter ecosystem for a while now. While the execution message format serves well for executing code, completion, inspection, shared frontend/backend state, and rudimentary code modification, LSP has many additional features and implementations which would be valuable to augment our kernels.

One of the current attempts at [1,2] is jupyterlab-lsp, which presently has a hard dependency on nodejs by way of jsonrpc-ws-proxy, which allows a single websocket port to server many stdio language servers. I've been making the configuration more jupyter-forward with this wip PR, which in turn wraps jsonrpc-ws-proxy with jupyter-server-proxy.

I've run into some issues in making it work simply for the general case, especially around absolute file paths, which aren't necessarily a given due to [3] the Jupyter Contents API, which allows serving "files" from... well, just about anything.

Another thing I want is to be able to have multiple, language-oriented URLs (or some message envelope format) be answered by the same underlying process, e.g. javascript-typescript-langserver.

Happy to help explore this space in any way!

[danixeee]

Hello @bollwyvl,

Currently pygls is designed to work with a single client on a host machine. We are using our extension based on pygls with VS Code client and each instance of it runs its own LS process.

1. Although I haven't worked with python websockets, I quickly took a look at the library and I am pretty sure that adding support for websockets wouldn't require too much work, mostly because pygls LSP is separated from the communication protocol.

2. There are no docs for this.

3. As I said, LS process is started on the same machine, but, anyway, opened documents (tabs) are being kept in memory. When document is changed, its content is sent to LS and synchronized with in-memory document. You can easily inherit base LSP protocol and change default behavior of this methods (it is what I did here) and use Contents API to manage documents.

Do you plan to handle multiple clients with the same LS process by using websocket channels? If you want to keep websocket proxy server, could you communicate with LS via existing TCP connection?

It is possible to use same LS for multiple languages. Each opened document should have languageId property (by LSP specs) and based on that you could apply language support logic for that language.

I hope I understood your problem. I am willing to help further

[bollwyvl]

Thanks for the response! Currently *pygls* is designed to work with a single client on a host

machine. We are using our extension based on *pygls* with VS Code client and each instance of it runs its own LS process.

Right, and this setup is also the primary (or at least initial) use case for working with LSP in jupyter. We also support running the client, the server, and even the user's personal execution environment (the kernel) on different computers. Probably the best way to see what I'm talking about is to open my current wip on binder: https://mybinder.org/v2/gh/bollwyvl/jupyterlab-lsp/add-traitlets-proxy?urlpath=lab%2Ftree%2Fexamples%2FPython.ipynb - The hub creates your "server" computer in docker and starts the jupyter server process. - you request your client assets from the server - your browser builds your client - it routes to the example notebook - your client requests a kernel and a language server - the server starts the language server and kernel and proxies requests between them and your language server

1. Although I haven't worked with python websockets <https://websockets.readthedocs.io/en/stable/intro.html>, I quickly took a look at the library and I am pretty sure that adding support for websockets wouldn't require too much work, mostly because *pygls* LSP is separated from the communication protocol. The inspiration here is from @sourcegraph's work. They have a general web

socket adapter for the wire protocol, but it's all based in nodejs. While we use a lot of node to build stuff, I'm trying to protect end users from having to install it :p

1. There are no docs for this. Well, maybe we'll get some out of this, though they might be rough

prototypes first!

1. As I said, LS process is started on the same machine, but, anyway, opened documents (tabs) are being kept in memory. When document is changed, its content is sent to LS and synchronized with in-memory document. You can easily inherit base LSP protocol and change default behavior of this methods (it is what I did here <https://github.com/textX/textX-LS/blob/danixeee/textx-view-container/textX-LS/server/textx_ls_server/protocol.py>) and use *Contents API* to manage documents. Great, thanks for the pointer! 1. Do you plan to handle multiple clients with the same LS process by using websocket channels? If you want to keep websocket proxy server, could you communicate with LS via existing TCP connection?

Yes, it's feasible that multiple clients could be connected. Jupyter kernels have this concept, and I think users would expect it.

It is possible to use same LS for multiple languages. Each opened document should have languageId property (by LSP specs) and based on that you could apply language support logic for that language.

Now that's interesting. So instead of multiple websocket URLs, we could just have one that knew how to route the request based on the languageId.

I hope I understood your problem. I am willing to help further

I'm rather new to this myself! thanks for all the help!

[bollwyvl]

@danixeee i've figured out a lot of my problems over on https://github.com/krassowski/jupyterlab-lsp/pull/46, thanks for the thoughts for getting there!

Basically, python-jsonrpc-server provides enough of the plumbing to handle the language stdio multiplexing, and their examples already had the stuff i needed to get it working over websockets. I still haven't dug into how to do my fancy virtual files, but at least now it's all in-process in our server, and not hidden inside node.

Thanks again for the guidance! If you have any interest in making any of your language servers work in Jupyter(Lab), we might soon have just the ticket for you!

[danixeee]

@bollwyvl Good to hear that!

I have created the issue to implement web-sockets for pygls.

If you decide to try out the pygls for any of your future language servers, I would be open to help in any way, thank you!