Quasiquoters rendering broken in hyperlinked sources

[geraldus]

Awesome hyperlinked sources have incorrect quasiquoters rendering, example http://hackage.haskell.org/package/yesod-form-richtext-0.1.0.0/docs/src/Yesod.Form.Summernote.html#snHtmlFieldCustomized

[mpickering]

The problem is that @mrhania implemented a custom lexer rather than reusing GHC's lexer. I tried to port it to use GHC's lexer a few months ago but there were some bad performance problems I didn't work out.

[harpocrates]

@mpickering Do you happen to have that branch lying around somewhere? I'd love to pick up the torch on this one (I've already played around in GHC's lexer and in Haddock), but it would be nice to hear what sorts of issues you had, advice you'd like to give.

Also, how were you measuring performance? I don't see any benchmarks in the .cabal files.

[alexbiehl]

@harpocrates IIRC this is the branch https://github.com/mpickering/haddock/commits/perf

I usually test performance for the hyperlinker with packages I have lying around, e.g. Cabal. Just make sure your test package doesn't use TemplateHaskell as this falsifies the the benchmarks as haddock needs to generate code for the package.

[harpocrates]

I think I've managed to get @mpickering 's branch back into shape. I fixed a handful of things and it seems to be mostly working now (I haven't tried running it on anything large yet, but the performance issues I felt initially went away after rebasing). I'll open a WIP PR when I get home from work. There are a couple fundamental issues that warrant discussion in any case though:

• CPP is not handled in the GHC lexer, yet Haddock will have to handle it. Currently, the solution is to recover from all lexical errors by turning problematic lines into comments (and then restarting the lexer from after that line). That seems like an OK failure mode to me, but I think it should be discussed...

• Language pragmas (actually all top-level pragmas) are not lexed as pragmas. This (probably) requires a GHC change. In the meantime, language pragmas would look like regular comments.

  GHC's handling of top-level pragmas is a (IMHO) huge lexer-level hack to try to avoid loading the whole file and we are feeling that here. (If bytestring ever becomes a stage 0 package, Data.ByteString.Lazy is what that really needs - not a hand coded list-based parser.)

In any case, the actual output looks great!

[alexbiehl]

This is fixed in #714. Thank you @harpocrates!

[mpickering]

The performance problems I had were obvious trying to use the modified version of haddock at all. It would just take ages on any file and use a lot of memory. Thanks for finishing this for me.

I don't understand the comment about language pragmas which seem to get lexed to ITlanguage_prag?

[harpocrates]

@mpickering I don't think language pragmas get lexed to ITlanguage_prag using lexTokenStream, but I'd love to be proven wrong. My understanding is that the GHC lexer can be initialized with 2 states: one of those is solely for lexing top-level file pragmas (see HeaderInfo) and the other is for lexing the whole file (lexTokenStream uses that). In the whole file version, ITlanguage_prag tokens never get generated - they just become comments.

My heuristic here was to give all {-# ... #-} comment tokens the TkPragma token style.