Pre-compute Suggestion DB during build time

[wdanilo]

Original issue is #184350816.

Motivation: Apparently we continue to re-parse documentation on every startup rather than reading the data from a cache. The parser is slow as it needs to be rewritten - tracked as #182497187, but not planned. Prototype implementation of caching on branch wip/jtulach/CacheSuggestions_184350816 shows 5-10% speed up by eliminating the generateDocumentation method all together. We should mitigate the urgent need to have a Rust parser for documentation by implementing the caching.

Plan: There is a build sequence that processes the standard libraries during sbt build (Regenerating manifest for [distribution/lib/Standard/Base/]) - we shall enhance it to pre-compute the Suggestion DB initial upload data and store them in JSON (preferably) next to the library manifest. During the initialization of the IDE/engine connection we check, if the pre-computed data are available next to the library manifest. If so, we send the prepared JSON and eliminate all computations what so ever.

Expectation: Following the current (and only) startup measurement methodology we are able to get down to 9600ms from the previous 11020ms - that's roughly the same as 1164ms wasted in generateDocumentation measurement suggests.

Conclusion: we can speed the engine initialization up by 5-10% by computing the suggestion database as part of the build of the standard libraries. Let's do it.

[enso-bot]

Dmitry Bushev reports a new STANDUP for yesterday (2023-02-09):

Progress: Started working on the task. Updated the SBT build logic to run the compilation of the standard library. Later it will generate the suggestion files. Resolved a copule of issues with the "triage" label. It should be finished by 2023-02-17.

Next Day: Next day I will be working on the #5068 task. Continue working on tasks

[enso-bot]

Dmitry Bushev reports a new STANDUP for today (2023-02-10):

Progress: Continue working on the task. Started working on the compiler logic to generate the suggestions during the compilation task. Was looking maybe I can reuse some of the existing module cache logic, but for now will only share the execution context. It should be finished by 2023-02-17.

Next Day: Next day I will be working on the #5068 task. Continue working on tasks

[enso-bot]

Dmitry Bushev reports a new STANDUP for yesterday (2023-02-13):

Progress: Continue working on the task. Created a stub for serialization manager and integrated it into the compiler pipeline. Started looking into the language server logic It should be finished by 2023-02-17.

Next Day: Next day I will be working on the #5068 task. Continue working on tasks

[enso-bot]

Dmitry Bushev reports a new STANDUP for today (2023-02-14):

Progress: Continue working on the task. Updated EnsureCompiledJob to compile only the required dependencies. Right now the filtering logic is not optimal, but I may look into the package repository to optimize it. Started to update the language server but then decided that I can reuse the existing notification, then the logic should stay the same. It should be finished by 2023-02-17.

Next Day: Next day I will be working on the #5068 task. Continue working on tasks

[enso-bot]

Dmitry Bushev reports a new 🔴 DELAY for yesterday (2023-02-27):

Summary: There is 14 days delay in implementation of the Pre-compute Suggestion DB during build time (#5068) task.
It will cause 0 days delay for the delivery of this weekly plan.

Delay Cause: Vacation

[enso-bot]

Dmitry Bushev reports a new STANDUP for yesterday (2023-02-27):

Progress: Catching up after the vacation. Was reviewing the changes in the PR. Started working on deserealization of the suggestions in the language server. It should be finished by 2023-03-03.

Next Day: Next day I will be working on the #5068 task. Continue working on the task

[enso-bot]

Dmitry Bushev reports a new STANDUP for yesterday (2023-02-28):

Progress: Continue working on the task. Started the refactoring to move the serialization logic to the package level. Jaroslav pointed out that the current runtime serialization will be replaced by the upcoming PR, so the serialization should remain in the runtime so far. Started working on preparing the appropriate notification to the language server instead It should be finished by 2023-03-03.

Next Day: Next day I will be working on the #5068 task. Continue working on the task

[enso-bot]

Dmitry Bushev reports a new STANDUP for yesterday (2023-03-01):

Progress: Continue working on the task. Updated and rebased the PR on the current develop branch. Implemented a suggestions serialization logic based on new caching infrastructure. Started testing. It should be finished by 2023-03-03.

Next Day: Next day I will be working on the #5068 task. Continue working on the task

[enso-bot]

Dmitry Bushev reports a new STANDUP for yesterday (2023-03-02):

Progress: Continue working on the task. Fixed the serialization logic. Ensured that the suggestions are successfully generated during the library building. It should be finished by 2023-03-03.

Next Day: Next day I will be working on the #5068 task. Continue working on the task

[enso-bot]

Dmitry Bushev reports a new 🔴 DELAY for today (2023-03-06):

Summary: There is 3 days delay in implementation of the Pre-compute Suggestion DB during build time (#5068) task.
It will cause 1 day delay for the delivery of this weekly plan.

Delay Cause: The lazy compilation caused a lot of failures in the runtime tests. Had to go through pretty much all of them and fix the imports.

[enso-bot]

Dmitry Bushev reports a new STANDUP for today (2023-03-06):

Progress: Continue working on the task. Started testing. Updated the language server logic, updated a bunch of failing tests, and fixed an issue with the checksum computation for the stored suggestions. PR is ready to review. It should be finished by 2023-03-06.

Next Day: Next day I will be working on the #5068 task. Continue working on the task

[JaroslavTulach]

After integration of this PR as well as #5568 and #5791 we managed to speed startup (according to our methodology - #5569) from 11s to 8s. I believe majority of the speed up comes from pre-computing the suggestion DB. Great work!

Still I see at least a second spend in SqlSuggestionRepo, so I suggest to continue with

• Speed start up by deferring post-startup ls/engine actions  #5040