CST memory optimization - #2191
Open
spoenemann wants to merge 11 commits into
Open
Conversation
Contributor
|
in my measurements we see only 35% memory gain (on the total heap) but this is still a huge improvement. for pure cst it is almost 50% |
cdietrich
reviewed
Aug 6, 2026
| npm run coverage # run tests with coverage report | ||
| ``` | ||
|
|
||
| You can also run the tests of a single package, e.g. `npm test --workspace=langium`. |
Contributor
There was a problem hiding this comment.
is this correct?
langium/packages/langium/package.json
Line 56 in 0be7d7e
npm test --workspace=langium
npm error Lifecycle script test failed with error:
npm error workspace langium@4.3.0
npm error location /Users/dietrich/git/langium/packages/langium
npm error Missing script: "test"
npm error
npm error To see a list of scripts, run:
npm error npm run --workspace=langium@4.3.0
Contributor
There was a problem hiding this comment.
maybe
npm test -- packages/langium
cdietrich
reviewed
Aug 6, 2026
| ## Prerequisites | ||
|
|
||
| For developing Langium, you require at least Node.js version 16 and at least npm version 7.7.0 to be able to use npm workspaces. | ||
| For developing Langium, you require at least Node.js version 20.10.0 (see the `engines` entry in [package.json](./package.json)). The npm version bundled with it supports npm workspaces, which this repository relies on. |
Contributor
There was a problem hiding this comment.
in the nvmrc we have 24, package json requires 22
cdietrich
reviewed
Aug 6, 2026
| When you add a dependency to a `package.json` npm resolves this from the configured registry (default is [registry.npmjs.org](https://registry.npmjs.org/)). In other dependency management systems you can create something like a dev or snapshot version and install it locally, but this is not possible as such with npm. If you want to achieve a similar behavior you need to link your packages globally (global refers to you user's account scope, local refers to the project's scope). When you do this you forcefully overwrite what was downloaded from the npm registry. Incrementing the version of Langium itself will lead to problems, because npm will not be able to resolve those from its registry and therefore fail any install attempts in projects referring to Langium. | ||
|
|
||
| If you execute `npm install` on the top-level of your Langium checkout all dependencies will be gathered locally inside `node_modules` and all four packages or sub-projects will be built. **Warning**: Do not run npm install directly inside the four Langium packages as it will mess up dependency resolution once you link packages globally. | ||
| If you execute `npm install` on the top-level of your Langium checkout all dependencies will be gathered locally inside `node_modules` and all packages or sub-projects will be built. **Warning**: Do not run npm install directly inside the individual Langium packages as it will mess up dependency resolution once you link packages globally. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR aims at improving Langium's memory usage, which was found to be relatively high in this article. The Conrete Syntax Tree (CST) implementation is the main reason for this. These measures have been applied:
rootis no longer stored in all CST nodes, but derived by walking up the container hierarchy.rangeis no longer stored in leaf CST nodes, but created on demand from integer values._astNodeand_rangeCache) are initialized withundefinedso the V8 engine includes them in the finalized object layout (this avoids the overhead of a separatePropertyArrayfor properties added later).contentis now areadonlyarray and must be manipulated via the CstNodeBuilder so we avoid the CstNodeContainer overhead.The first two changes save memory by doing a bit more computation for the
root,text, andrangeproperties, which mostly affects LSP-related services. The overhead of these computations is virtually negligible, and the benefits of the changes outweigh it by far as shown in the benchmarks below.I created four benchmarks to measure the impact of these changes on the CST size, parsing, and document building:
cst-node-memory-benchmark.tsmeasures the memory usage of a synthetically generated CSTcst-parse-memory-benchmark.tsmeasures the memory usage of a parsed CSTparser-time-benchmark.tsmeasures the execution time of the parserworkspace-build-benchmark.tsmeasures the execution time and heap size of a full workspace build