runnableExamples should preserve source code doc comments, strings, and (maybe) formatting

[timotheecour]

proc foo*()=
  runnableExamples:
    import std/strutils
    ## line_doc_comment
    # bar
    doAssert "'foo" == "'foo"
    ##[
    foo
    bar
    ]##
    doAssert: not "foo".startsWith "ba"

renders as:

import
  "pathto/runnableExamples/t03_bad_quoting.nim"

import
  std / strutils

block:
  doAssert "\'foo" == "\'foo"
  ## foo
  ## bar
  ## 
  doAssert:
    not "foo".startsWith "ba"

• the added quoting \' in "\'foo" , although not incorrect, is not needed and the way it's written in the source ("'foo"`)should be preserved
• the line_doc_comment is not rendered
• an extra ## with empty comment (below bar) is wrongly inserted
• [minor] formatting is a bit odd, eg `doAssert: not "foo".startsWith "ba" is in 1 line in source file but gets split in 2 when rendered which is rather ugly (and vertical space hoarder, when there are a bunch of these) ; not sure how easy it'd be to fix that since it's rendered

doAssert:
    not "foo".startsWith "ba"

these are correct:

• the multiline doc comment is rendered (although see note above regarding extra ##)
• the comment bar which is not a doc comment is not preserved (useful to distinguish what shd end up in docs vs not)

links

• pr_fix_8871_runnableExamples_verbatim (didn't finish)

[johnnovak]

+1 on this one, some more examples:

func swapEndian*(T: typedesc[SomeNumber], buf: var openArray[byte],
                 startIndex: Natural) {.inline, since: (1,1).} =
  ## Swaps the byte order of a number stored as a byte sequence in `buf`
  ## starting at `startIndex`. The type of the number must be passed in as an
  ## argument.
  runnableExamples:
    var buf: array[6, byte]
    buf[0] = 0xff
    buf[1] = 0xca
    buf[2] = 0xfe
    buf[3] = 0x42
    swapEndian(uint16, buf, 1)
    doAssert buf[0] == 0xff
    doAssert buf[1] == 0xfe
    doAssert buf[2] == 0xca
    doAssert buf[3] == 0x42

And here's the result. Note the literals have been reformatted, there's extra leading zeroes, which doesn't make much sense as these are bytes, and they're uppercased:

Ideally, we should just take the original text, strip the left margin, and just insert it into the documentation unchanged. We shouldn't parse it into AST and the regenerate the source from that, we're losing information that way. Just taking the text as it is seems to be the easiest thing to do anyway.

[timotheecour]

Ideally, we should just take the original text, strip the left margin, and just insert it into the documentation unchanged

my thoughts as well; the code is being parsed/run anyways with nim doc so there's no point in doing that work twice; only caveat is wrt non-doc-comments, which would be preserved even if not meant to be, but that's fine IMO.
for this, we can change strutils.indent to accept negative count (s/Natural/int/) and compute indentation from 1st line.
proc indent(s: string; count: Natural; padding: string = " "): string {...}

[johnnovak]

Well, I don't think there should be a doc comment/non-doc comment distinction in the runnableExamples blocks anyway, so I think it's a non-issue. The rule would be that the text would appear in the documentation verbatim, and that's the end of it, it's simple.

[johnnovak]

I'd even propose a stripMargin function, Scala has a similar one and it's quite handy.