WIP: Search harder for cross-module references

[timholy]

This provides a partial fix to #688, specifically for the case where we are building docs for multiple modules and some of them may not know about each other. As a partial fix I prefer it to #1351, though for cases this doesn't cover that may have merit too.

This is a WIP because (1) it's missing tests, and (2) while it solves the issue during the CrossReferences phase, HTMLWriter still has an issue:

[ Info: HTMLWriter: rendering HTML pages.
┌ Warning: invalid local link: unresolved path in reference.md
│   link.text =
│    1-element Array{Any,1}:
│     Markdown.Code("", "SnoopCompileCore.@snoopr")
│   link.url = "reference.html#SnoopCompileCore.@snoopr"
└ @ Documenter.Writers.HTMLWriter ~/.julia/dev/Documenter/src/Writers/HTMLWriter.jl:1697

I'm not entirely sure about the internal data structures here. I'm guessing I need something like

Documenter.jl/src/CrossReferences.jl

Lines 153 to 157 in 6b3efa6

	# Replace the `@ref` url with a path to the referenced docs.
	docsnode = doc.internal.objects[object]
	path = relpath(docsnode.page.build, dirname(page.build))
	slug = Utilities.slugify(object)
	link.url = string(path, '#', slug)

. I would appreciate some guidance. (In this situation, @snoopr is defined and documented in SnoopCompileCore, but SnoopCompile has a using SnoopCompileCore in it so @snoopr is visible from inside both SnoopCompileCore and SnoopCompile.)

[timholy]

Update, I'm not so sure the problem I thought I saw really is a problem. With this PR, the make.jl logs in SnoopCompile@1.6.1 are clean, if memory serves correctly. If there is interest in this I can presumably put a test together.

[mortenpi]

Just to make sure I understand: this fixes the case where you're referring to a binding that is defined in a module that is not being usingd in make.jl, but is loaded as a dependency of something and its source code lives in the current repository (i.e. the multi-package repo thing)? And without this patch we can't resolve such at-ref links, because they are not available in Main?

If I understand this correctly, then yes, I think it's a good idea to have this. I don't think this necessarily conflicts with #1351, but it is probably the better solution for the SnoopCompile use case.

Regarding tests: I think it should be easy to add something to the test/examples builds that test this code path.

[timholy]

Just to make sure I understand: this fixes the case where you're referring to a binding that is defined in a module that is not being usingd in make.jl, but is loaded as a dependency of something and its source code lives in the current repository (i.e. the multi-package repo thing)? And without this patch we can't resolve such at-ref links, because they are not available in Main?

I think it's probably also relevant to the following case: module A has two sub-modules, B and C, independent of one another, not exported, but which want to refer to one another.

That's what I've tried to add in my test. There are two senses of "works": do the tests pass according to Test, and do the links work when I click on them in a browser? On this branch, the answer to both questions is "yes." If I cherry-pick the "Test cross-module references" commit onto master, the links do not work when I click on them. However, the tests still pass. What can I add to make them fail, i.e., to force it to check whether the links work?

[mortenpi]

First, sorry for not getting back to this earlier. When I build test/examples locally, I get an error (stacktrace of which was wrong, #1363):

BoundsError: attempt to access (:Main, :Mod)
  at index [0]
Stacktrace:
 [1] getindex at ./tuple.jl:24 [inlined]
 [2] getindex(::Tuple{Symbol,Symbol}, ::UnitRange{Int64}) at ./range.jl:295
 [3] resolve_reference(::Module, ::Expr, ::Set{Module}) at /home/mortenpi/Julia/JuliaDocs/Documenter/src/CrossReferences.jl:208
 [4] docsxref(::Markdown.Link, ::String, ::Dict{Symbol,Any}, ::Documenter.Documents.Page, ::Documenter.Documents.Document) at /home/mortenpi/Julia/JuliaDocs/Documenter/src/CrossReferences.jl:132
 [5] basicxref(::Markdown.Link, ::Dict{Symbol,Any}, ::Documenter.Documents.Page, ::Documenter.Documents.Document) at /home/mortenpi/Julia/JuliaDocs/Documenter/src/CrossReferences.jl:52
 [6] xref(::Markdown.Link, ::Dict{Symbol,Any}, ::Documenter.Documents.Page, ::Documenter.Documents.Document) at /home/mortenpi/Julia/JuliaDocs/Documenter/src/CrossReferences.jl:44
 [7] #1 at /home/mortenpi/Julia/JuliaDocs/Documenter/src/CrossReferences.jl:34 [inlined]

That is also failing the CI. However, it looks to be a heisenbug.. sometimes it doesn't throw that error, which very interesting...

• I took the liberty of cherry-picking Fix exception handling in test/examples/make.jl #1363 into here, so that stacktrace would be correct if it happens.

Regarding the tests:

• Actually, we call makedocs with modules = [Mod, AutoDocs, AutoDocs.A, AutoDocs.B, AutoDocs.Filter], so I am not sure if, when we put the test docstrings into A and B, it will actually test what we want? For clarity, I created separate submodules for that now.

• I also added another test to check the cross-module case (i.e. both A and B are directly in Main, closer to the SnoopCompile case). I would say that those tests should be sufficient.

• To actually test if the links are present, I would suggest adding some occursins to check if the appropriate href="#foo" is present. I dropped a proof of concept example into the tests, but I couldn't really make it work at the moment due to the error above.

That's how far I got at the moment, I can take another look later in the week.

[mortenpi]

Actually, we call makedocs with modules = [Mod, AutoDocs, AutoDocs.A, AutoDocs.B, AutoDocs.Filter], so I am not sure if, when we put the test docstrings into A and B, it will actually test what we want?

Ok, I just saw that this was added here. Passing the submodules by hand to makedocs should not be necessary since we expand the list automatically anyway:

Documenter.jl/src/Documents.jl

Lines 349 to 352 in 8226880

	blueprint = DocumentBlueprint(
	Dict{String, Page}(),
	Utilities.submodules(modules),
	)