Skip to content
This repository was archived by the owner on Jan 2, 2021. It is now read-only.

Parse module headers #511

Merged
merged 17 commits into from
Apr 27, 2020
Merged

Parse module headers #511

merged 17 commits into from
Apr 27, 2020

Conversation

pepeiborra
Copy link
Collaborator

@pepeiborra pepeiborra commented Mar 27, 2020

Instead of full modules, ghcide now parses only the module headers and avoids storing unnecessary ASTs in the Shake graph where possible. This happens in a couple of places:

  • Locating the imports of a module. Now this uses a ModSummary instead of a ParsedModule
  • For computing all the completions in a module. Now this uses a ModIface instead, i.e. a .hi file.
  • For computing source span infos, the parsed modules are only used when building against ghc-lib. Otherwise, ModIface is used instead.
  • Interface files created by ghcide now embed Haddocks

Altogether, ghcide only parses full modules for files of interest now, or for dependencies when an up-to-date .hi file is not available. The main benefits are

  1. shorter start-up times
  2. lower memory usage.

Joint work with @lazamar.

Fixes #505

lazamar and others added 7 commits March 25, 2020 17:38
- extract getModSummaryFromImports and fix diagnostics
- replace GetParsedModule by GetModSummary where possible
  There is only one usage of GetParsedModule left, and that is in GetSpanInfos
  for documentation. This the wrong approach, docs should be loaded from
  interface files and not from sources. TODO
Progress notifications are not being sent anymore
Instead, embed haddocks in interface files
@pepeiborra pepeiborra changed the title Mod summary Parse module headers Mar 27, 2020
Copy link
Contributor

@mpickering mpickering left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments but overall looks good.

@pepeiborra pepeiborra force-pushed the mod-summary branch 3 times, most recently from 50240fb to f3d397b Compare March 30, 2020 19:10
@mpickering
Copy link
Contributor

An annoying problem I immediately ran into, if a module fails to parse because of the Opt_Haddock flag then you get an error diagnostic. As soon as you open this module however, the diagnostic goes away because the Opt_Haddock flag is no longer enabled!

@mpickering
Copy link
Contributor

mpickering commented Mar 31, 2020

I am also getting a lot of traces which I assume should be removed.

Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/Match/Constructor.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/Binds.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/PmCheck.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/ListComp.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcSimplify.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/PmCheck/Ppr.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcInteract.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcCanonical.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/Expr.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/Binds.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcInteract.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/Match.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/ListComp.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/Match/Constructor.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/PmCheck/Ppr.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/Arrows.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/PmCheck.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/PmCheck/Oracle.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcSimplify.hs
Released stringbuffer for (preprocessor) /home/matt/core-typedth/compiler/GHC/Iface/Load.hs-boot
Released stringbuffer for (preprocessor) /home/matt/core-typedth/compiler/GHC/CoreToIface.hs-boot
Released stringbuffer for (preprocessor) /home/matt/core-typedth/compiler/GHC/Driver/Plugins.hs-boot
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcRnTypes.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcMType.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Driver/Plugins.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/FamInst.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Rename/Env.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Core/Make.hs
Released stringbuffer for (preprocessor) /home/matt/core-typedth/compiler/GHC/Driver/Pipeline/Monad.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Core/Unfold.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/Inst.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Core/Op/ConstantFold.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/basicTypes/MkId.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Iface/Load.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcFlatten.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Iface/Rename.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Core/Lint.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/CoreToIface.hs
Released stringbuffer for (preprocessor) /home/matt/core-typedth/compiler/GHC/HsToCore/Monad.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Iface/Binary.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/Utils.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/GuardedRHSs.hs
Released stringbuffer for (preprocessor) /home/matt/core-typedth/compiler/typecheck/TcHoleFitTypes.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Iface/Env.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcHsSyn.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/prelude/PrelInfo.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/iface/BuildTyCl.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/DsMetaTc.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcUnify.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Types/Name/Shape.hs
Released stringbuffer for (preprocessor) /home/matt/core-typedth/compiler/GHC/HsToCore/Binds.hs-boot
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Core/SimpleOpt.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Core/Rules.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcEnv.hs
Released stringbuffer for (preprocessor) /home/matt/core-typedth/compiler/GHC/Rename/Unbound.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/ClsInst.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/Match/Literal.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Core/Op/Monad.hs
Released stringbuffer for (preprocessor) /home/matt/core-typedth/compiler/GHC/Rename/Utils.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/HsToCore/Quote.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/GHC/Driver/Finder.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcTypeable.hs
Released stringbuffer for (preprocessor) /home/matt/core-typedth/compiler/GHC/HsToCore/Expr.hs-boot
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcErrors.hs
Released stringbuffer for (runCpp) /home/matt/core-typedth/compiler/typecheck/TcSMonad.hs

@pepeiborra
Copy link
Collaborator Author

pepeiborra commented Mar 31, 2020

Ah, my bad, those traces were not intended for this PR. Will fix

UPDATE: fixed

@pepeiborra
Copy link
Collaborator Author

pepeiborra commented Mar 31, 2020

An annoying problem I immediately ran into, if a module fails to parse because of the Opt_Haddock flag then you get an error diagnostic. As soon as you open this module however, the diagnostic goes away because the Opt_Haddock flag is no longer enabled!

Yup, that’s going to be annoying. First workaround that comes to mind is parse again without Opt_Haddock in case of error, not pretty though. @cocreature @ndmitchell any other suggestions?

If Opt_Haddock is not enabled we parse twice to capture Haddock parse errors
@pepeiborra
Copy link
Collaborator Author

An annoying problem I immediately ran into, if a module fails to parse because of the Opt_Haddock flag then you get an error diagnostic. As soon as you open this module however, the diagnostic goes away because the Opt_Haddock flag is no longer enabled!

Yup, that’s going to be annoying. First workaround that comes to mind is parse again without Opt_Haddock in case of error, not pretty though. @cocreature @ndmitchell any other suggestions?

I have gone with a different solution: parse files of interest twice, with and without Opt_Haddock, to collect Haddock parse errors consistently.

Copy link
Collaborator

@cocreature cocreature left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thank you so much! Sorry for taking ages to review this.

@cocreature cocreature merged commit 4f9c756 into haskell:master Apr 27, 2020
pepeiborra added a commit to pepeiborra/ide that referenced this pull request Dec 29, 2020
* Create rule to get ModSummary without parsing entire source file

* Load file source from disk if not available in memory

* Fix build after cherry pick

* Couple of fixes

- extract getModSummaryFromImports and fix diagnostics
- replace GetParsedModule by GetModSummary where possible
  There is only one usage of GetParsedModule left, and that is in GetSpanInfos
  for documentation. This the wrong approach, docs should be loaded from
  interface files and not from sources. TODO

* Fix watched file tests

Progress notifications are not being sent anymore

* Compat with GHC 8.6

* Avoid parsing source files for completions and documentation

Instead, embed haddocks in interface files

* Allow CPP in module

* Force things after parsing in order to release buffers

* avoid holding on to stringbuffer unnecessarily

* Skip unnecessary file contents read

* Drop HscEnv requirement

* Add comments on forcing things

* Add comments on GHC_LIB restriction

* Parse files of interest twice to capture Haddock errors

If Opt_Haddock is not enabled we parse twice to capture Haddock parse errors

* Parallelize two-pass parsing

* Update src/Development/IDE/Core/Compile.hs

Co-authored-by: Marcelo Lazaroni <lazaronijunior@gmail.com>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
pepeiborra added a commit to pepeiborra/ide that referenced this pull request Dec 29, 2020
* Create rule to get ModSummary without parsing entire source file

* Load file source from disk if not available in memory

* Fix build after cherry pick

* Couple of fixes

- extract getModSummaryFromImports and fix diagnostics
- replace GetParsedModule by GetModSummary where possible
  There is only one usage of GetParsedModule left, and that is in GetSpanInfos
  for documentation. This the wrong approach, docs should be loaded from
  interface files and not from sources. TODO

* Fix watched file tests

Progress notifications are not being sent anymore

* Compat with GHC 8.6

* Avoid parsing source files for completions and documentation

Instead, embed haddocks in interface files

* Allow CPP in module

* Force things after parsing in order to release buffers

* avoid holding on to stringbuffer unnecessarily

* Skip unnecessary file contents read

* Drop HscEnv requirement

* Add comments on forcing things

* Add comments on GHC_LIB restriction

* Parse files of interest twice to capture Haddock errors

If Opt_Haddock is not enabled we parse twice to capture Haddock parse errors

* Parallelize two-pass parsing

* Update src/Development/IDE/Core/Compile.hs

Co-authored-by: Marcelo Lazaroni <lazaronijunior@gmail.com>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
pepeiborra added a commit to pepeiborra/ide that referenced this pull request Dec 29, 2020
* Create rule to get ModSummary without parsing entire source file

* Load file source from disk if not available in memory

* Fix build after cherry pick

* Couple of fixes

- extract getModSummaryFromImports and fix diagnostics
- replace GetParsedModule by GetModSummary where possible
  There is only one usage of GetParsedModule left, and that is in GetSpanInfos
  for documentation. This the wrong approach, docs should be loaded from
  interface files and not from sources. TODO

* Fix watched file tests

Progress notifications are not being sent anymore

* Compat with GHC 8.6

* Avoid parsing source files for completions and documentation

Instead, embed haddocks in interface files

* Allow CPP in module

* Force things after parsing in order to release buffers

* avoid holding on to stringbuffer unnecessarily

* Skip unnecessary file contents read

* Drop HscEnv requirement

* Add comments on forcing things

* Add comments on GHC_LIB restriction

* Parse files of interest twice to capture Haddock errors

If Opt_Haddock is not enabled we parse twice to capture Haddock parse errors

* Parallelize two-pass parsing

* Update src/Development/IDE/Core/Compile.hs

Co-authored-by: Marcelo Lazaroni <lazaronijunior@gmail.com>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Avoid parsing sources for dependencies
4 participants