WIP: Docstring refactoring

[mlechu]

This change moves the construction of docstring registration calls into lowering
instead of doing so in a macro. The original intent was just to fix #58630,
but some refactoring is necessary to do this in a good way.

Not all of the following is implemented yet. Feedback on the design is welcome!

Current design

Docstrings are stored in a module-scoped IdDict mapping Base.Docs.Binding to
Base.Docs.MultiDoc, which either contains a single docstring, or a map from
method signature to docstring.

Docstrings are parsed to @doc str x calls (this macro is also public and
used). @doc defers to Core.atdoc!, which figures out if it's one of the
many things we can document, tries to guess the signature of any methods if x
is a function, and returns a quoted call to Docs.doc!, which registers the
docstring.

A few things make our current design complex:

1. Docstring registration calls are computed during macro expansion, so we're
   trying to figure out what we're documenting purely by Expr. We need to dig
   through struct contents and call expressions to find field docstrings and
   signatures.
   • The implicit first argument in a signature tuple is excluded from our keys
2. @doc is sometimes responsible for evaluating the expression it's given, and
   sometimes not.

Complicating matters further are a few "doc-only" special cases where adding a
docstring changes the semantics of the following expression

"foo"
x         # Will never throw an UndefVarError

"foo"
Mod.x     # Same for x, but Mod needs to be defined.

"foo"
f(x::T)   # f isn't called.  Neither f nor x need to be defined, but T does.

This change

• Structures and the information we store (other than including signatures'
  first arguments in keys) are both unchanged.

• Change two-arg @doc to produce Expr(:doc, ...) instead of calling
  Core.atdoc

• New Expr(:doc, str, x, linenode) form. Generally, this lowers x
  as usual followed by a call like

  Core.setdoc(m::Module, docstr::Core.SimpleVector, linenode, x::Any)

  and returns the value of x if we're not in a doc-only case. Pending parser
  changes, this lets us prevent docstrings from changing what an expression
  returns (modulo doc-only cases).

• Unlike Core.atdoc, the new Core.setdoc gets the answers from lowering, and
  can query the runtime about the object to document. The "signature" key we
  use for method x is now exactly x.sig.

Misc

• Rename some @doc helpers to distinguish between the @doc x ("get docs for
  x") and @doc str x (set docstring str for x) code paths, which are
  pretty disjoint.
• Remove @doc "foo" -> bar syntax, which appears to be an undocumented special
  syntax. I'll put it back if people use it.
• @doc used to take all arguments and pass them to atdoc!, so had at least
  one secret undocumented argument define. Remove this option.

Potential issues

• Signature-guessing is necessary in determining the key for doc-only call
  expressions where there's nothing to lower. In its current form, this PR
  breaks these.
• The REPL uses the same signature-guessing logic for lookup and needs something
  to replace it.
• More lisp?! (but it's a relatively simple addition to lowering, and I
  don't mind being the one to add this to JuliaLowering too).

Todo

• Parse "str" x as Expr(:doc, ...) instead of a call to @doc (JuliaSyntax
  already does this internally before Expr-converting it)
• It might be worth cleaning up the semantics of Expr(:doc); so far I've
  tried to stay faithful to existing behaviour.
  https://github.com/mlechu/julia/blob/docsystem-wrangling/src/julia-syntax.scm#L2604-L2617
• Store our new signatures in a vector instead of an IdDict (see Doc system treats signatures incorrectly #58630).
• Fix up docs for (::T)(x) = x
  • We currently count these as docs for T
• Check for IR spam, use multidoc instead of duplicated strings
• Two-arg @doc should ideally return the same thing (Base.Docs.Binding) as
  before, probably by just calling one-arg @doc
• Implement docs for @__doc__ and quoted macro calls (should be easy)
• Struct field docs

[Keno]

Nice. This implementation makes me much happier. Whenever things guess at what lowering might do, trouble starts happening.

• Signature-guessing is necessary in determining the key for doc-only call
  expressions where there's nothing to lower. In its current form, this PR
  breaks these.

I think I'd be happy for this syntax to always just do the dumb thing of lowering f(::T, ::S) where A to Tuple{typeof(f), T, S} where A and not support any of the actual tricky cases (optional args, kwargs, etc).

[Keno]

Base.Docs.Binding

A bit of a side quest, but I've always hated this type. Can we just use regular Core.Binding for this?

[mlechu]

I think I'd be happy for this syntax to always just do the dumb thing of lowering f(::T, ::S) where A to Tuple{typeof(f), T, S} where A and not support any of the actual tricky cases (optional args, kwargs, etc).

We'd need to require f to be defined beforehand, but I'd also be happy with this.

Can we just use regular Core.Binding for this?

Would be nice. I'll see if it's feasible (and reasonable to do in this PR).

[vtjnash]

That is more of an implementation bug though (quite a few open issues about it), since it is not a Binding, but rather a StructFieldPath (which might sometimes happen to alias some globals)

[mlechu]

That is more of an implementation bug

What are you referring to, Docs.Binding? Also, what is a StructFieldPath?