[clang-doc] add a JSON generator

[evelez7]

Adds a JSON generator backend to emit mapped information as JSON. This will enable a better testing format for upcoming changes. It can also potentially serve to feed our other backend generators in the future, like Mustache which already serializes information to JSON before emitting as HTML.

This patch contains functionality to emit classes and provides most of the basis of the generator.

[evelez7]

• [clang-doc] add a JSON generator #142483 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[evelez7]

I think this patch is mostly ready in terms of functionality, but we should decide if the emitted files should follow the HTML or YAML style of creation. Right now, since I just ripped the Mustache generator code, it creates folders for each namespace and emits the nested entities there. This actually represents a problem for template specializations because the code tries to write another JSON object (the specialization) to the same file as the base template, which is invalid.

With YAML, each entity is emitted in a file that uses the entity's USR as a name. All files are emitted to the output directory without any folders. That conveniently solves the above template problem, but also results in ugly file names that are potentially bad for multi-file testing (right now, we use regex to find a USR file name to test YAML).

I'd say the HTML layout is nice, but then we'd need to decide what to call the specializations' files or whether to just include them as an object inside the base template's file.

[ilovepi]

This is a good start, but I thin we'll want some more tests, and probably some unit test coverage. Unittesting is especially nice, since I believe this backend doesn't need any of the asset files, right?

As for whether this should follow the pattern of YAML or HTML ... I'm conflicted for the same reasons you are. But maybe there's a better way for us to generate the filename that doesn't need the USR? like could we get the mangled name and use that? I'd hope those would be different enough to not conflict. Alternatively, maybe we should look at a way we can merge all the related docinfo together at an earlier step.

[ilovepi]

I think this is a mistake in the mustache impl, since I believe we normally allow the user to set the the line numbering schema in the existing other generators.

[ilovepi]

This whole bit may be nicer after #142273

[evelez7]

This is a good start, but I thin we'll want some more tests, and probably some unit test coverage. Unittesting is especially nice, since I believe this backend doesn't need any of the asset files, right?

Yeah, this doesn't need assets, so I'll be able to call things directly for unit tests.

As for whether this should follow the pattern of YAML or HTML ... I'm conflicted for the same reasons you are. But maybe there's a better way for us to generate the filename that doesn't need the USR? like could we get the mangled name and use that? I'd hope those would be different enough to not conflict. Alternatively, maybe we should look at a way we can merge all the related docinfo together at an earlier step.

I'll investigate this. Although right off the bat, the names in the Infos aren't mangled. Name and FullName are the same for both records, which is unfortunate.

[ilovepi]

I'll investigate this. Although right off the bat, the names in the Infos aren't mangled. Name and FullName are the same for both records, which is unfortunate.

OK, well, let's give it a try, and if its too hard lets ... IDK go w/ the YAML thing, so its correct? Testing will be kind of hellish so I'm loathe to go that path, but if one is obviously broken, then we should probably avoid it.

[ilovepi]

Right now, since I just ripped the Mustache generator code, it creates folders for each namespace and emits the nested entities there. This actually represents a problem for template specializations because the code tries to write another JSON object (the specialization) to the same file as the base template, which is invalid.

After re-reading this, I have a vague memory that this used to happen for everything, and it somehow got fixed for YAML. Probably back when Brett was working on this.

[evelez7]

If we're open to adding a flag in the base Info like IsClassSpecialization, then we can probably easily deal with these by trying to reconstruct the specialization's arguments (will be something like "Foo<T, int>.json"). Functions don't produce their own files so function specializations wont be a problem. Then we can keep this layout. @ilovepi

That or we manually change the name when serializing during mapping. I'm not so sure how well merging the infos would be ergonomically. Specializations can have all of their own unique members and methods which need documentation.

I also found a way to get a nice ugly mangled name :)

[ilovepi]

If we're open to adding a flag in the base Info like IsClassSpecialization, then we can probably easily deal with these by trying to reconstruct the specialization's arguments (will be something like "Foo<T, int>.json"). Functions don't produce their own files so function specializations wont be a problem. Then we can keep this layout. @ilovepi

That or we manually change the name when serializing during mapping. I'm not so sure how well merging the infos would be ergonomically. Specializations can have all of their own unique members and methods which need documentation.

This sounds promising. I'm fine w/ adding a field to track this. BTW, what does clang do? I'm wondering if we should track more than 1-bit of info here.

I also found a way to get a nice ugly mangled name :)

Nice! IIRC that's what I've seen a lot of document generators use. If its unique enough for ODR, is probably good enough for docs ... right?

[evelez7]

This sounds promising. I'm fine w/ adding a field to track this. BTW, what does clang do? I'm wondering if we should track more than 1-bit of info here.

As far as I can tell, inside the AST Clang makes use of the isa<> mechanisms (which we could also leverage) or pointer unions like

llvm-project/clang/include/clang/AST/DeclCXX.h

Lines 486 to 487 in f427d3a

	llvm::PointerUnion<ClassTemplateDecl *, MemberSpecializationInfo *>
	TemplateOrInstantiation;

[ilovepi]

This sounds promising. I'm fine w/ adding a field to track this. BTW, what does clang do? I'm wondering if we should track more than 1-bit of info here.

As far as I can tell, inside the AST Clang makes use of the isa<> mechanisms (which we could also leverage) or pointer unions like

llvm-project/clang/include/clang/AST/DeclCXX.h

Lines 486 to 487 in f427d3a

	llvm::PointerUnion<ClassTemplateDecl *, MemberSpecializationInfo *>
	TemplateOrInstantiation;

Lets keep it simple for now. Clang-Doc doesn't have LLVM style RTTI yet, and I'm cautiously optimistic that we can keep it that way. So lets just add a bool or some bitfields or whatever.