x/tools/gopls: The Need For Gopls Extensibility (Proposal)

[marwan-at-work]

Background

Gopls is an incredible tool. Go programmers would truly not be as productive without it.

Gopls fully relies on go/packages to load and parse Go files. However, go/packages is known to be quite slow.

Therefore, the biggest benefit of gopls is arguably its caching of parsed Go files and their type system results that came from go/packages.

Once you have a bird's eye view of an Editor's workspace, the amount of features you can implement to make Go programmers productive are quite limitless. Gopls has a lot of features from code navigation, to code generation, and more.

However, it is practically impossible to fit all the features a programmer wants into gopls. The more features gopls adds, the more unmaintainable it becomes, the slower it gets, and the larger its codebase and resulting binary grow.

Therefore, there is a strong argument for gopls to support the minimal feature for it to be useful out of the box.

But that leaves huge swaths of incredible features that people can want from gopls that are still not worthy of putting into the Gopls codebase.

Below, I've listed a number of such use cases that I've personally run into. I imagine there are a lot more.

Problem

There are already many popular libraries in the community that leverage go/packages to do all sorts of work. To quickly do a reverse-lookup of those libraries, https://pkg.go.dev shows that go/packages is imported-by over 1000 packages in the wild not counting closed source Go code.

However, all of those packages suffer from the same problem that gopls has already solved: go/packages is too slow.

Projects that use multiple code generation libraries no longer enjoy the same speed typical Go programmers enjoy due to Go's fast compiler.

Anecdotally speaking, one company I interviewed with told me that their build system takes at least 10 seconds to generate all of their code and compile it.

Therefore, there is a strong need for such libraries to leverage the gopls cache without having to implement their features directly in the gopls codebase.

Proposal

I think the need for Gopls to be extensible is vital to the Go community.

Most tools out there write their code analysis tools in terms of go/packages but I'd love for many of these tools to take advantage of the already cached modules in their gopls session.

There are several ways to accomplish this task (I listed some ideas below) but I am indifferent with the implementation details. Because as a Go programmer, I have run into numerous scenarios where I wish gopls was extensible in any sort of way.

Use Cases

Code Generation

1. Ent: the Ent library uses Go code to define a SQL database schema and generates a full ORM with query and insert features that has become quite popular. The library uses go/packages to analyze your schema code and creates a number of packages that expose Go functions that can talk to a SQL server. As your schema grows, the time it takes to generate your code grows more.
2. gqlgen: gqlgen uses a GraphQL schema coupled with a config file to generate an entire Go GraphQL Server. The config file allows the code generator to know when to create a Go struct that maps to a GraphQL Object, or when to re-use an existing struct. Gqlgen heavily uses go/packages to analyze and generate code and can take many seconds.
3. entgql: interestingly, there is a library that combines both of the libraries above and you can imagine how slow that becomes.
4. Protocol Buffers: there are protobuf plugins out there like protoc-gen-twirpql that uses go/packages and could definitely leverage the gopls cache to generate its code. It's also worth noting that this is a more interesting case than the above because protobuf plugins are usually invoked from the protoc compiler so a plugin would have to somehow know how to latch on to a gopls session and not the other way around.
5. Google's Wire: just another popular example out there that uses go/packages for AST manipulation and code generation where it could be made a lot faster if it leveraged the gopls cache.

AutoComplete

Error Handling

Many projects has repeatable patterns for how they decorate and return errors in their functions. For example, Athens uses the Upspin Error Handling Pattern, CLIs just use fmt.Errorf("funcName: %w", err), pkgsite uses defers, and much more.

Gopls already has a great auto-complete snippet suggestion that just returns the error such as:

if err != nil {
  return <zeroValues...>, err
}

But an extension could be smarter and go beond just returning the error, such as:

if err != nil {
  return <zeroValues...>, errors.E(op, err) // Athens pattern
  // or
  return <zeroValues...>, fmt.Errorf("<enclosingFunc>: %w", err) // Go1.13 error wrapping pattern.
  // or
  return <zeroValues...>, errors.Wrap(err, "<enclosingFunc") // github.com/pkg/errors pattern.
}

Or in the case of http handlers, users can implement their own way of handling an error inside of a HandlerFunc. See this comment where it explicitly says:

We have been fairly hesitant about adding snippets, since it might seem like we're prescribing a "correct" or "Go team approved" way of doing things.

This is a perfect use case for extension so that gopls doesn't implement a single correct way, but projects can have their own snippet completions.

Snippet Completions

Besides error snippet completions, many companies and teams can have templates as starter points for their servers.

For example, Uber has fx, while the New York Times has Gizmo and they follow specific patterns. A company can publish an extension that could create snippets for starting their own framework as well as other LSP features.

Other Use Cases

1. Many companies have internal libraries and gopls could extend its Go: Add Import feature where a company can distribute an extension that lists such private modules as options to add to your go.mod or Go file.
2. Code Analyzers: Gopls already has a number of analyzers some of which are maintained by open source contributors. Some programmers don't use all of the analyzers. Therefore, it might make sense to pull some of them out into extensions that the open source maintainers can contribute to them directly.
3. Experimental features: many experimental features can be shipped as an extension first while they await proper review. Users can then give feedback and report errors so that it is more robust by the time it lands into upstream. A good example would be "the stub method" feature, but I'm sure there are many more.

Possible Implementations

There are several ways that can make gopls extensible:

1. Serialize the gopls cache so that other tools can quickly deserialize and load it.
2. Dynamically load extended features using the plugin library (which I have heard a few times that it is unstable and a bad idea)
3. Statically load extended features and compile them with gopls together. This makes installing gopls with extensions a bit more complex but avoids the need for serialization/deserialization or the use of the plugin library.

Difficulty

I completely understand that this task is nowhere near trivial.

The first thing that strikes me is the fact that the entirety of gopls is under an internal package path which makes it impossible for other codebases to re-use any of its type definitions and functions outside of the tools repository. Therefore, the first thing that gopls will need to do is export a lot of its types and functions so that extensiosn can use them. Perhaps just renaming internal to experimental is good enough as a start so that it doesn't block the progress of this task, until the gopls team is happy with the results.

Another thing that requires a lot of thinking is how gopls recognizes extensions and communicates with them to share its memory objects (Snapshots, Views, etc). I suggested some ideas above and I hope there are more ideas out there. But it is also not a trivial task to land at the perfect solution. I hope this issue can be a place to discuss those ideas and their tradeoffs.

[stamblerre]

Thank you for filing this thoughtful and well-researched proposal! We'd love to support something like this, but this is definitely a large project that we'd have to look into prioritizing. We'll follow up here after some team discussions, and perhaps we can discuss this at the next golang-tools call.

[hherman1]

Does it really make sense for a code generation tool to have a dependency on a running gopls instance? It seems like making these tools self contained would be a good thing.

For a long time devs at my company would compile Java programs using intellijs compiler, which hooked into ids state. But when the IDE got into a bad state, it would break your builds.

seems like not a great fit for codegen. Direct ide features make sense to me though

[marwan-at-work]

Does it really make sense for a code generation tool to have a dependency on a running gopls instance? It seems like making these tools self contained would be a good thing.

For sure. I'd be happy to accept any solution that would make code generation fast. The reality is, go list and therefore go/packages are too slow to be part of your development iteration and on large scale projects that do a lot of code generation, it becomes a very unenjoyable experience.

[myitcv]

cc @ianthehat who has investigated this before. We should also reference previous discussions on this proposal.

[findleyr]

I have thought about this a lot, both in the context of extensibility and in the context of simplifying gopls's implementation and improving gopls's performance.

The reason gopls has always had memory problems is that it naively loads the entire workspace (files+syntax+types), with the assumption that we need the workspace to be in-memory to satisfy queries like workspace symbols or references. But in fact, those workspace-wide queries can be offloaded to more efficient data structures that produce approximate results, and then we can refine our results with full package information as necessary -- I've already done this for workspace symbols, and the entire symbol data structure fits in ~30mb for Kubernetes, compared with a workspace memory footprint of ~3GB.

Therefore it's possible that we could drop a huge amount of syntax and type information, computing it on-demand as needed (or possibly loading types from export data), without any loss in gopls's functionality. The major barrier to doing this is the fact that our current package APIs inside of gopls assume that package information is already in memory. If we were to try to drop things from memory, we'd need a better API for package information. Such an API would need to be declarative, allowing the caller to specify exactly what information is needed, and for how long. Such an API would be very similar to go/packages, albeit with additional configuration for cache persistence.

So it has been in the back of my mind that an ideal final form of gopls might have an "in-memory go/packages", augmented with metadata for cache persistence. This could enable extensibility, maintainability, and performance. But the devil is in the details, and with the push for generics I haven't had time to dig into whether this would actually be possible.