Binding: Vision

[jonpryor]

The current binding infrastructure is "janky":

1. "Somehow obtain" a .aar or .jar file to bind.
2. Add it to a Binding project, and set the appropriate build action, e.g. @(LibraryProjectZip) or @(EmbeddedJar).
3. Build it.

In practice, (1) and (3) can be incredibly annoying. (At least for @jonpryor, it's not always obvious where to obtain the files referenced by a Maven artifact, and (3) is full of peril.)

(3) will remain "out of scope" for this vision; fixing (3) involves ensuring that generator can always emit compilable output, and that we expand the Java constructs that generator supports. We should absolutely do this, but addressing these deficits involves finding the "less-than-ideal" constructs, filing issues for them, and fixing them.

Which leaves (1) and (2): can anything be done there?

Artifact Retrieval & Binding

Instead of a .aar or .jar file as being the "primary starting point", can we instead have a Maven artifact ID be the primary point?

Then, given a set of Maven artifact IDs:

1. Specify the Maven IDs.
2. Obtain the entire dependency graph for the maven artifacts.
3. Find existing bindings for everything within (1).
4. For any dependencies not already bound, bind them.

Specify the Maven IDs

How should the Maven IDs be specified? What's the "user interface"?

Obtain Maven Dependency Graph

I believe @moljac has code to do this, as part of Binderator: https://github.com/xamarin/XamarinComponents/tree/master/Util/Xamarin.AndroidBinderator

Find existing bindings

It appears that NuGet.org allows searching arbitrary package metadata. If we ensure that all new NuGet packages contain NuGet.org-searchable package metadata containing the Maven artifact ID & version, then it should be possible to find existing bindings.

@moljac is working on this.

Bind unbound packages

For any maven artifacts in the dependency graph which haven't been bound, presumably including the "root" dependencies, we should bind them.

Proposal: Item Group?

We can imagine using an MSBuild item group to specify the "root" maven IDs, either directly:

<MavenArtifact Include="androidx.car" ArtifactId="car" Version="1.0.0-alpha7" />

or by using a file (file format? Gradle? JSON? XML?) to contain the information.

Implicit in this idea is that these are like @(PackageReference) item group: the containing .csproj can have as many of them as desired, which in turn means that the .csproj could emit "many" output assemblies (dozens?), one per bound library as well as the "intentioned" output assembly.

Additionally, how would @(TransformFile) (e.g. metadata.xml) work in such an environment? All @(TransformFile)s apply to all unbound artifacts?

Proposal: dotnet new?

Instead of using an item group, add a new tool, e.g.

dotnet new maven-binding --group-id androidx.car --artifact-id car --version 1.0.0-alpha7

This command would in turn emit a solution, with one .csproj project per unbound artifact. This "answers" the @(TransformFile) question: each project uses has its own set of @(TransformFile)s.

Less well defined is this: how does one update an existing "binding solution"? Changing the root artifact version will "ripple" across the entire solution; is it possible to update such a directory tree in-place? Or would we need to re-invoke the dotnet new maven-binding command into a new directory, and require the user manually copy over any required @(TransformFile)s & partial classes and such?

[jonathanpeppers]

is it possible to update such a directory tree in-place?

When working on #5348 I found you can do dotnet new maven-binding --force to write on top of existing directories and files. It does not delete any existing files when using --force.

[jpobst]

It sounds like this proposal is basically doing 2 things for the user:

1. Downloading the binary from Maven
2. Adding NuGet or Project dependencies automatically based on Maven metadata

#1 probably isn't too interesting. I don't think it used to be this way, but now you can just search on Maven and it provides an easy link to download the artifact: example.

We could probably even automate this without much effort in today's projects if we wanted to and resolve it at compile time:

Change:

<InputJar Include="okhttp-4.9.0.jar" />

to something like:

<MavenInputJar Group="com.squareup.okhttp3" Id="okhttp" Version="4.9.0" />

#2 is definitely more interesting. I'm not sure how much effort it normally takes.

Additional Considerations to Flesh Out

NuGet Metadata

From my reading of that link, I don't think NuGet supports searching arbitrary metadata. It seems to only support searching the official NuGet supported metadata. Thus we would need to decide which of those fields we would use.

Additions considerations:

• How do we handle collisions? ie: More than one package says they are the package for com.squareup.okhttp3.okhttp.
• Is this a security risk? Does it matter if someone publishes a malicious package claiming to be androidx.core on NuGet?

Visual Studio UX

How extensible is the built-in VS template UI system? Can we prompt the user to enter Maven information that we can use? Is the template system robust enough that it can run our custom logic to query Maven and resolve dependencies in order to generate projects?

If we can only do this outside of VS, do we think users will find and use it?

Updates

Once a package has been bound and a new release is posted to Maven, what does the update process look like for updating our binding?

Cases to consider:

• Update the bound package version
• New version has a new dependency that is on NuGet
• New version has a new dependency that is not on NuGet
• New version removed a NuGet dependency
• New version removed a not-NuGet dependency

Publishing to NuGet

We need to understand what the target user is going to do with the binding: consume it locally or publish to NuGet.

If publishing to NuGet then there are a few additional issues like:

• Specifying the NuGet package version
• How do we convert project dependencies we created into NuGet dependencies once published?

[moljac]

It sounds like this proposal is basically doing 2 things for the user:

1. Downloading the binary from Maven

Trivial.

2. Adding NuGet or Project dependencies automatically based on Maven metadata

Not only Maven. What if we have bindings? maven gives a sheet about nuget.org

From my reading of that link, I don't think NuGet supports searching arbitrary metadata.
It seems to only support searching the official NuGet supported metadata.
Thus we would need to decide which of those fields we would use.

I am testing client API for search

https://github.com/HolisticWare-Xamarin-Tools/HolisticWare.Xamarin.Tools.Bindings.XamarinAndroid.FassBinderMeister/blob/master/source/HolisticWare.Xamarin.Tools.NuGet/NuGetClient.cs#L36-L73

NOTE:

                                                    new SearchFilterType
                                                    {                                                       
                                                    }

I did not play with that more than 15 minutes.

How do we handle collisions? ie: More than one package says they are the package for com.squareup.okhttp3.okhttp.

dunno. Trying to have something - then I can make new steps...

Cases to consider:

Update the bound package version
New version has a new dependency that is on NuGet
New version has a new dependency that is not on NuGet
New version removed a NuGet dependency
New version removed a not-NuGet dependency

Writing xummary and pseudo-code. Coming soon

[moljac]

Obtain Maven Dependency Graph
I believe @moljac has code to do this, as part of Binderator:
https://github.com/xamarin/XamarinComponents/tree/master/Util/Xamarin.AndroidBinderator

Nope. Side project. Will be easily embeddable. Or code copied with whatever changes needed (namespace changes etc)

Proposal: Item Group?
We can imagine using an MSBuild item group to specify the "root" maven IDs, either directly:

<MavenArtifact Include="androidx.car" ArtifactId="car" Version="1.0.0-alpha7" />

MavenArtifact would not normaly appear in .NET library or project, but Xamarin.Android. Thus we could add some convetion-based rules... Like checking for serialized data files (json, xml) with artifacts to bind...

If the ItemGroup is defined we could:

1. check for serialized files and merge lists
   or
2. ignore files

Why working with files?
It would be easier to have such files as "interfaces" for design and testing.

Now more details:

We have 2 sources od ultimate divine truth (priority list):

1. nuget.org
2. maven repos (currently I work only with maven.google.com)

Pick this:

<MavenArtifact Include="androidx.car" ArtifactId="car" Version="1.0.0-alpha7" />

What I am trying to implement right now:

1. query nuget.org if there is already something like "androidx car car" on nuget.org. Grab versions and metadata
   https://github.com/HolisticWare-Xamarin-Tools/HolisticWare.Xamarin.Tools.Bindings.XamarinAndroid.FassBinderMeister/blob/master/source/HolisticWare.Xamarin.Tools.NuGet/NuGetClient.cs#L43

2. If there is NO such nuget package or the version is GREATER-THAN Version="1.0.0-alpha7" put in the List TODOs_ToBind

3. Check dependencies from maven POM

   foreach (Artifact a in a.POM.Dependencies.Leaves()) // Leaves is GraphTheoretical term for flattening
   {
       TODOs_ToBind = TODOs_ToBind.Concat
                                   (
                                       ! a.CheckNugetOrgPackages()     // everything not on nuget.org must be bound 
                                                                       // .. or other BuildActions 
                                   );
   }
   
   foreach (Artifact a in TODOs_ToBind)
   {
       Process.Run("dotnet new maven-binding {a}");
   }
   '''
   
   

[moljac]

Artifact a  = new Artifact().POM;

POM does not have stable name (yet) it is basically Maven 4.0.0 XSD schema transformed into C# classes

https://github.com/HolisticWare-Xamarin-Tools/HolisticWare.Xamarin.Tools.Bindings.XamarinAndroid.FassBinderMeister/blob/master/docs/maven-4.0.0.xsd

https://github.com/HolisticWare-Xamarin-Tools/HolisticWare.Xamarin.Tools.Bindings.XamarinAndroid.FassBinderMeister/blob/master/source/HolisticWare.Xamarin.Tools.Maven/Models.GeneratedFromXSD.Original/maven-4.0.0.cs

I used XML to from AndroidX and GPS-FB to generate C#, - BUT there were some fields missing. So last week I generated C# from maven XSD

[moljac]

My plan is 20201202 for next 2 weeks:

1. create minimal API for step 3 in Binding: Vision #5352 (comment)

2. pack/publish some HolisticWare packages with API

3. create Cake scripts for testing that can be copied ANYWHERE (C# libs for MSBuild tasks or dotnet tools)

[jpobst]

An additional consideration from @Redth:

Maven is like NuGet, there can be many feeds, some of which may require authentication. So we may need to support specifying a URL, and possibly a username/password as well.

[Redth]

We should really back up a step here and sit down to discuss a few things:

1. What are the different scenarios in which customers care about bindings
2. What are the challenges in those scenarios
3. What can we do to help with those challenges.

[moljac]

1. What are the different scenarios in which customers care about bindings

In very few cases they do. Usually expect to be done by Microsoft or some partner. Basically wishes to have some SDK bindings are submitted as issues in (AndroidX/AndroidSupport, GPS-FB and XamarinComponents). Some are submitted as internal mails and very few (for me personally) over social media.

2. What are the challenges in those scenarios

1. Bindings knowledge and skills which influence productivity where most users give up.
2. Bindings process
   2.1 find/download artifact,
   2.2 analyze artifact (dependencies and nuget packages)
3. Test bindings (samples or unit tests)

Challenges I personally have:

• Does bindings package exist for given artifact?
• Is update necessary?
• What are details on NuGet.org?
  • Dependencies (transitive)
• Where to get artifact[s] (download) and info about it?
  I use following repositories (in this order)
  https://maven.google.com/web/index.html
  https://mvnrepository.com/
  https://search.maven.org/
• What are details (info about) artifact from Maven repo?
  • Dependencies (transitive)
    • Is something to be bound in transitive dependency chain?
      For published bindings on NuGet.org this is not necessary
      • How to handle transitive dependencies?
        dotnet new?
        BuildAction?

3. What can we do to help with those challenges.

Improving productivity will most likely increase abstraction (add one or more levels of APIs) that might drive some users away from bindings.

Needed APIs:

• NuGet querying for info about bindings

  After few days of more intensive work with NuGet client API the conclusion is that there is much more work
  than expected. NuGet.org does not have perfect search, so client API is limited too.
  Few problems:

  • Search does not retrieve DependencySets, so filtering by TargetFramework is more complicated then I though
  • Search API input is string as keyword retrieving tons of results.
    Current solution is to provide API with Predicate or planned PredicateBuilder for custom search
    and to extend it with Metadata query which retrieves Package DependencySetsProperly

• Maven APIs (extended - more than in MavenNET)

• API that uses data from NuGet API and Maven API and prepares bindings

NOTE: More details to come.

Maven is like NuGet, there can be many feeds, some of which may require authentication. So we may need to support specifying a URL, and possibly a username/password as well.

I work with Google's maven right now, but I do plan to add others (mavenCentral and jcenter/Bintray).

I would not bother with private repos or repos that need authentication. There must be some reason artifacts are not public. We could add that later.