api.xml doc parser

[Redth]

The existing doc parsers are great for when the documentation is available for download publicly, however we are facing a situation with Android Support libraries where documentation available online is either lagging behind releases or is simply not updated in the usual locations for consuming.

The documentation is available on the web server, however since we can't redistribute this content, we need to distribute the information (parameter names) in an intermediate format which the binding build process can use as a source for parameter name data.

For Android Support libraries, we are starting to produce parameter names in an intermediate format which mimics the existing api.xml structure (albeit with much less information in the attributes).

This PR creates a parameter name doc parser for the api.xml format.

[monojenkins]

Hello! I'm the build bot for the Mono project.

I need approval from a Mono team member to build this pull request. A team member should reply with "approve" to approve a build of this pull request, "whitelist" to whitelist this and all future pull requests from this contributor, or "build" to explicitly request a build, even if one has already been done.

Contributors can ignore this message.

[dnfclas]

Hi @Redth, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!

The agreement was validated by .NET Foundation and real humans are currently evaluating your PR.

TTYL, DNFBOT;

[Redth]

It's not clear to me why ApiSource is set multiple times here, so I've done the same by setting the DocletType twice.

[atsushieno]

I guess they are just extraneous, but it was @dellis1972 who wrote this class-parse bits so he might have some comments.

[dellis1972]

@Redth you know what I can't remember why.. Its probably a left over from before I implemented the first one.
Try removing it and seeing what happens :)

[atsushieno]

The overall changes looks good to me, but I'm not sure if we'd like to make this feature public. Namely, I have no idea how people practically generate such api.xml referene file. If that can be represented just as Metadata fixups, then I don't see any point of introducing this new feature.

[Redth]

@atsushieno currently we are using the docs to generate metadata fixups (so that we can publish something to the public repo which users can reliably reproduce the same build we make), however there are a couple of problems with it:

1. It seems precedence of multiple transform rules affecting the same parameter name is undefined. So, if we need to manually correct a parameter name (so we don't break previous API), we'd need to manually edit the otherwise automated metadata transforms file - this is tedious and somewhat error prone to do for every version update.
2. It's non-trivial to try and automate splitting up the metadata transforms into the support library binding projects they belong to (it's not as simple as splitting by package name - some package names traverse multiple java android support libraries). This means we end up adding all the metadata fixups to all the projects (some 9000 rules), which slows the build down but more importantly causes a lot of meaningless warnings in the build output.

We are already, today, generating such an api.xml file to use with this parser for android support from the online docs.

I get that it's not too useful to the public, but I need some way of providing an alternate parameter name lookup mechanism to our support library (and play services) binding projects going forward. This seems to be the least friction.

[jonpryor]

It seems precedence of multiple transform rules affecting the same parameter name is undefined

Have an example? Metadata transforms are executed in-order, from top-to-bottom, when processing the metadata file.

[Redth]

@jonpryor my example is multiple files.... let's say we have:

Metadata.Manual.xml:

<attr path="/api/package[@name='android.support.v7.app']/interface[@name='ActionBar.TabListener']/method[@name='onTabSelected']/parameter[2]" name="managedName">fragmentTransaction</attr>

Metadata.Generated.xml

<attr path="/api/package[@name='android.support.v7.app']/interface[@name='ActionBar.TabListener']/method[@name='onTabSelected']/parameter[2]" name="managedName">ft</attr>

Which order are the files processed in, and can they be guaranteed to always be processed in the same order?

[Redth]

@jonpryor rather ^

[jonpryor]

Which order are the files processed in, and can they be guaranteed to always be processed in the same order?

Kinda?

On the one hand, yes: generator --fixup=A --fixup=B ... will always process the fixup files in the specified order: A, B.

On the other hand...things get complicated. Nobody calls generator directly. Instead, people use the <BindingsGenerator/> MSBuild task, which uses the @(TransformFile) Build action, which is up to the handling of the IDE and MSBuild.

On the plus side, the order of @(TransformFile) should be the same as the <TransformFile/> elements within your .csproj. On the minus side, that order isn't controllable from the IDE. It could in theory be changed at any time, based on the whims of the IDE.

So there is control, but it's not reliable control, unless you're willing and able to hand-edit your .csproj file to ensure that the order is what you require.

[Redth]

@jonpryor that makes sense, and is maybe even workable for our needs (although good luck to the poor soul who inherits that gem).

Otherwise, the main issue is dealing with all the warnings our binding projects get with the extra transforms that won't match in a given project. Again, I can't think of a reasonably simple way to have my scraper split up metadata transforms it generates into multiple files, according to which java libraries they actually should be used for, so I end up including the same huge metadata transforms file (some 9000 lines of it) in every (~26) support library binding project, slowing the build down and generating a lot of meaningless warnings.

[jonpryor]

the main issue is dealing with all the warnings our binding projects get with the extra transforms that won't match in a given project.

We've never produced warning-free code, and I don't see that changing anytime soon (though generator could use some more contributors! )

As such, what's the difference between hundreds of warnings and thousands of warnings? It's not like you're actually going to read them...

Alternatively, we could plausibly add a way to hide warnings, akin to csc /nowarn:....

[Redth]

I do actually read through many of the warnings. Some of them are actually important and can affect the generated c# binding API. So adding thousands more warnings is just more noise to deal with.

If we could ignore warnings in an entire metadata transform file that could be useful, but ignoring a type of warning is still not ideal since the same types may be meaningful coming from other metadata files.

Can we do something like <metadata ignoreWarnings="true"> ?

[atsushieno]

There is a handful of missing arguments here (@Redth should have written everything here, not internal discussion). What he is using to generate api.xml is this web documentation scraper:
https://github.com/xamarin/components/tree/master/AndroidDocUtil

That's why #97 (comment) says "some package names traverse multiple java android support libraries" and that matters (otherwise, the API XML should not just include such non-existent classes/members at all, period).

So it's more like a API database that contains more than the relevant API that the jar binding needs to fill.

With all that in mind, I still don't think this "add this non-javadoc documentation format" is a good approach, but having IAndroidDocScraper (the name is argurable) and supporting some custom scraper is a good idea (maybe like supporting custom CodeDom type with assembly qualified name in several tools).

That is my ideal solution, and in the meantime I think this change in general can be merged. With some change:

• I still wouldn't like making this feature public, so the value "ApiXml" should become something like "_ApiXml" to indicate that it is not public.
• So as use in xamarin-android MSBuild tasks (once we integrate this)

[Redth]

I've named the doclet type to _ApiXml as suggested, I think that's sensible.

I agree that in some sense this isn't quite like a java doc, however it is generated from java docs, so the data is really the same. Also, for Android support libraries, we don't get individual java docs for each .jar library, so for each binding project, even if we used the stock droid docs, we still have way more data than is relevant to the actual binding project, so this is really not a different situation, it's just that android support libraries are a bit of a different beast than most other bindings.

I do think it sounds reasonable to eventually break apart the concept of java docs and other data sources for parameters, however is it really necessary if there are no other obvious use cases at the moment than this particular one? The only other thing I can foresee happening is Google possibly eventually providing us with API information in a more machine readable format than droid docs, but so far we have not received anything like that. If that does happen, perhaps that's the time to revisit separating these ideas out?

[atsushieno]

No more comments, merging.