Implement a web backend for rustdoc_ng

[alexcrichton]

This large commit implements and html output option for rustdoc_ng. The
executable has been altered to be invoked as "rustdoc_ng html " and
it will dump everything into the local "doc" directory. JSON can still be
generated by changing 'html' to 'json'.

This also fixes a number of bugs in rustdoc_ng relating to comment stripping,
along with some other various issues that I found along the way.

The make doc command has been altered to generate the new documentation into
the doc/ng/$(CRATE) directories.

Previews

• http://www.contrib.andrew.cmu.edu/~acrichto/doc/std/
• http://www.contrib.andrew.cmu.edu/~acrichto/doc/extra/

Missing features

• Different versions of documentation on the same page (all possibly indexed as well?) I think that this needs to be thought out before action is taken. It's an awesome idea, but it should be done carefully.
• Source links are missing. This is a little dependent on getting versions working. In theory we should link back to github, but we should always link back to the exact version the documentation was generated from.
• Integration with other tools. It would be awesome to have rustpkg-style inference of the package name and version so they don't have to be specified anywhere. Additionally, I should be able to build documentation for a pkgid, not necessarily a crate file.

cc @cmr/@Seldaek

[alexcrichton]

A couple of cool things:

• I managed only hit one ICE while writing all this code. Either rust is getting stable or I'm subconciously trained to avoid particular patterns. I have a terrible memory, so I think rust is stabilizing!
• This is some seriously heavy usage of the new format! syntax extension. I really enjoyed using it, although I could be biased... I'm seeing rays of light for an HTML formatting library which escapes everything by default, but I couldn't quite wrap my head around doing so.
• The formatting of all the code into html templates is ugly as sin. This could seriously benefit from a formatting library.
• I thought include_str! was cool

Regardless, carry on!

[alexcrichton]

Also, this would seriously benefit from a markdown C library as opposed to forking off and executing pandoc. We're not really using any of pandoc's fancy extensions any more (except for all the tutorial/manual generation), so we're less attached to it now than before.

The reason it needs to be a C library is that instead of forking pandoc once per page, now it's once per docblock which is a lot more often. I've parallelized everything among a set number of workers, so it's not as bad as it could be, but it's still pretty bad.

[alexcrichton]

And of course, mad props to @cmr and @Seldaek for all their awesome work!

[thestinger]

You can use pandoc to compile small fragments without forking more than one process for it, since it's a library.

[alexcrichton]

@thestinger that would be awesome, but it wasn't clear to me at first how to do that. @Seldaek suggested coalescing a number of rendering requests into one batch to send to pandoc, but that was difficult to implement so I didn't end up doing it. If I could keep a long-running pandoc process running though, that'd be awesome.

[steveklabnik]

Ive also been thinking about an HTML auto-escaping library....