Clarification (wiki material) about connection state

[ljw1004]

I'd like to contribute some "implementor's notes". It feels like the wiki would be a natural place for these, but Github doesn't support PRs for wiki contributions, so I'm writing it out as an issue.
(@dbaeumer, it's your call whether you think that implementor's notes are useful, and if so, then where they should go).

For the main wiki page:

Implementator's notes:

• [[Connection State]] How the canonical LanguageClient handles connecting to the server, crashes, retries, Initialize request, Shutdown request, and Exit notification

For a new wiki page called Connection-State.md:

When writing a client you'd like to know how to launch an LSP server, and how to respond to various events, e.g.

• If the server responds to Initialize request with a ResponseError, then the client pops up a message dialog, and the retry flag indicates whether the dialog should offer the user a button to press to retry initialization.
• If the server closes its connection to the client, then the client should restart.

It's useful to see how the canonical implementation of a LSP language client handles those and other events, as implemented in the class LanguageClient.

The relevant state of a LanguageClient object is characterized by three variables:

  private _state: Initial | Starting | StartFailed | Running | Stopping | Stopped;
  private _connectionPromise: Thenable<IConnection> | undefined;
  private _resolvedConnection: IConnection | undefined;

Here's the flow chart:

(INITIAL)--------------------------+
| _state == ClientState.Initial    |
| _connectionPromise == undefined  |
| _resolvedConnection == undefined |
+----------------------------------+
  |
  | [external] extension author invokes .start()
 \|/
(STARTING)-------------------------+
| _state == ClientState.Starting   |
| _connectionPromise = underway    |
+----------------------------------+
  |
  | We called resolveConnection(), which calls createConnection(),
  | which launches the LSP server and connects to its stdin/stdout.
  |
  | [external] either the process launches correctly and we have stdin/stdout, or not!
  |
  <>-----------------------------------+
  |                                    |
  |success:                            |failure:
  |                                   \|/
+------------------------------+     (STARTFAILED)-----------------------+
| _connectionPromise = success |     | _state == ClientState.StartFailed |
+------------------------------+     | _connectionPromise = failed       |
  |                                  +-----------------------------------+
  |                                    |
  |                                    | message to user
  | call initialize() to send an       | "Couldn't start server"
  | Initialize request to LSP server  \|/
  |                                    X
  |[external] get back response
  |[external] *** or connectionClosed/Error
  |
  <>-----------------------------------+
  |                                    |
  |ok:                                 |error:
  |                                    |
(RUNNING)--------------------------+   | message to user "Couldn't start server"
| _state == ClientState.Running    |   | The message has a Retry button if .retry
| _resolvedConnection = connection |   | flag was set on the reply from the server
+----------------------------------+   |
  |                                    | [external] user clicks button
  | everything's okay!                 | [external] *** or connectionClosed/Error
  |                                    |
  | [external] *** or                  |
  | connectionClosed/Error             <>----------------+
 \|/                                   |                 |
  X                                    |cancel:          |retry:
                                       |                 |
                                       |                 +-> jump to the initialize()
                                       |                     call a few lines up
                                       |
                                      \|/
                                     (STOPPING)----------------------+
                                     | _state = ClientState.Stopping |
                                     +-------------------------------+
                                       |
                                       | send Shutdown request to LSP server
                                       |
                                       | [external] get back response
                                       | [external] *** or connectionError
                                       |
                                       <>----------------+
                                       |                 |
                                       |ok:              |error:
                                       |                 |
                                       |                 +-> X. Nothing further.
                                       |
                                       | send Exit notification
                                       | to LSP server
                                      \|/
                                     (STOPPED)----------------------+
                                     | _state = ClientState.Stopped |
                                     | _connectionPromise = null    |
                                     | _resolvedConnection = null   |
                                     +------------------------------+
                                       |
                                       | Wait 2000ms, then do
                                       | "kill -0 $LSP_PID"
                                       | which checks whether the process still exists.
                                       | If it does, then do
                                       | "kill -9 $LSP_PID" or similar.
                                      \|/
                                       X

*** The vscode-jsonrpc library might signal two other events, which get extra
handling at the locations marked *** in the diagram above:

<>---...---+
|          |
|          |connectionError:
.          |
.          | If the error came from attempting to write to LSP's stdin,
.          | and we'd had no more than two successive errors, then ignore it.
           | Otherwise:
           |
           +--> jump straight to the (STOPPED) state.

<>---...--+
|         |
|         |connectionClosed:
.         |
.        \|/
.       +----------------------------+
        | _connectionPromise = null  |
        | _resolvedConnection = null |
        +----------------------------+
          |
          | Log the time at which this occurred.
          | Have there 5 or more closures in the past three minutes?
          |
          <>---------------------------------+
          |                                  |        
          | >=5 closures                     | <5 closures
         \|/                                 |
        +------------------------------+     +-> jump to the (STARTING) state.
        | _state = ClientState.Stopped |
        +------------------------------+

[ljw1004]

Note: this chart is particularly important for authors of "singleton LSP servers" -- LSP backend servers which are so heavyweight, e.g. ~10min startup+indexing time, that we use a single machine-wide persistent instance of the LSP server. Here's why...

• Imagine that one VSCode instance is launched. It will launch an LSP process which merely proxies requests through to the existing singleton backend. This instance will hold the authoritative source of truth for all documents which it has open/edited.

• Imagine a second VSCode instance is launched for the same project. If a given file is open+edited in both VSCodes, well, the poor singleton backend server can only trust one of them as its source of truth, and it will no longer be able to provide valid language services to the other one.

• So, one of the LSP proxies must be terminated. BUT this will cause VSCode to relaunch it automatically! (Automatic relaunch is desirable in some cases, e.g. for a benign and recoverable crash. But it's undesirable in this case.)

Question: Is there any way to solve this problem while staying within the bounds of the existing LSP?

Answer: Yes!!! First, observe that there are three causes of sending an Initialize message: (1) initial launch; (2) automatic retry after connection got closed; (3) in response to user clicking Retry button after the previous Initialize request gave back ResponseError. The LSP server can distinguish the third case from the first two, because the third case is the only one where the Initialize request was sent to the LSP server over a connection that had previously been used for a prior Initialize request.

That suggests the following design for singleton LSP servers:

1. When an LSP proxy gets a first Initialize request, it should first check whether any other LSP proxies are connected to the same singleton backend server. If they aren't, fine, it can proceed. If there are, then it should fail the Initialize request, and give back a ResponseError saying "Sorry another client is connected at the moment" with the retry flag set to true.

2. When an LSP proxy gets a subsequent Initialize request, then we know it must have come in response to the user explicitly clicking the Retry button (even though LSP spec doesn't explicitly convey this fact, we can still deduce it). So at this time it should kick off any other proxies connected to the same singleton backend server, causing them to terminate, and it should proceed with initialization as normal.

3. If an LSP proxy got kicked off, it should terminate. The corresponding VSCode instance (in a background window) will automatically retry launching the LSP proxy. This has already been handled by case [1], where it just pops up a banner inviting the user to retry.

4. If an LSP proxy crashes for its own buggy reasons (but there were no additional LSP proxies connected to the same server) then automatic retry will kick in. This has already been handled by case [1], where it just relaunches the LSP proxy with no user intervention required.

[arxanas]

@ljw1004: I think this diagram is inverted:

<>---...--+
|         |
|         |connectionClosed:
.         |
.        \|/
.       +----------------------------+
        | _connectionPromise = null  |
        | _resolvedConnection = null |
        +----------------------------+
          |
          | Log the time at which this occurred.
          | Have there 5 or more closures in the past three minutes?
          |
          <>---------------------------------+
          |                                  |        
          | <5 closures                      | >=5 closures
         \|/                                 |
        +------------------------------+     +-> jump to the (STARTING) state.
        | _state = ClientState.Stopped |
        +------------------------------+

You want to jump to the starting state and restart the process if you've had fewer than 5 closures, not more.

[ljw1004]

@arxanas you're right. I've fixed it.

[dbaeumer]

@ljw1004 very nice work. I will target this for the next minor release to find a good home for this.