Automatic API compatibility validation

[mockitoguy]

Problem

Producers and consumers of software libraries don’t have an easy life. As a consumer of a library I don’t know if new version of the library brings API incompatibilities until I try to use it and it blows up, hopefully during compilation or testing, and not on production! As a producer of a library I can unintentionally sneak in API incompatibilities, unless I am very careful and/or great engineers review my code and spot the problems.

Let’s make the life of producers and consumers of libraries delightful!!!

Solution

What if producer gets nice report or even build failure if he changes the API without bumping major version? What if consumer gets nice report, or even early build failure if known API incompatibilities exist in the new version of a library he attempts to use? What if the incompatibilities are computable statically and it is not necessary to compile code/run tests to identify issues? What if the tools can tell you with good accuracy what version of a library you can consume safely?

First steps

While we don’t necessarily need to scope the entire solution we can to scope down an initial feature set that can help us validate if the problem is worth solving and how. Below features are suggestions and should be critically reviewed by whoever decides to own this feature.

Does my change break API compatibility?

To get started, we can create a Gradle task that compares 2 binaries, identifies API incompatibilities and reports them. Tools that identify API incompatibilities already exist and can be leveraged (mockito/mockito#738). In the project we already have code that pulls down the previously released binaries (implemented as part of #84).

Suggested implementation

Suggested implementation should be enough to get started. However, please discuss and suggest different approach as you see fit.

• Create a new plugin ApiCompatibilityPlugin that will add new Gradle task of type CheckApiCompatibilityTask (new task type). The new plugin is applied by our continuous delivery plugin, however, we don’t hook up the new task to the workflow yet.
• When the task runs, it pulls down previously released binaries reusing the code we already have
• Then, it reports binary incompatibilities. For example it produces a report file.
• Bonus: add configuration to the task to produce report AND throw exception if there are incompatibilities
• Bonus 2: add configuration to the task to specify what are public packages and what are “internal” packages. Incompatible changes to internal packages are OK and should not fail the task. In the compatibility report it is easy to discriminate changes to “internal” API VS changes to “public” API.
• Bonus 3: make bonus 2 feature configurable on a task (property on the task).

Future ideas

Below future ideas are wild brainstorming :)

• Create a service that reports API incompatibilities and attaches information to GitHub Pull Request (similar to how Travis CI reports build results, codecov reports coverage).
• Create a concept of “API snapshot” that is stored in a file and ships to binary repository along with the publications (jar, sources, javadoc). Tools can use the snapshot to compare APIs between arbitrary versions of a library.
• Create plugin for consumers so that they can check compatibility with libraries they consume. The plugin should aid consumer in bumping to new version of a desired library.

[metlos]

I've came across this by a complete accident, but you might want to take a look at what I'm doing with https://github.com/revapi/revapi.

There's no gradle plugin for it just now, but the maven plugin does quite a bit:

• "What if producer gets nice report or even build failure if he changes the API without bumping major version?"
• What if consumer gets nice report, or even early build failure if known API incompatibilities exist in the new version of a library he attempts to use?

• Yes, if the API change in the used library is "exposed" through some possible call-chain from the API of the consumer project

• What if the incompatibilities are computable statically and it is not necessary to compile code/run tests to identify issues?

• the code needs to be compiled, but no tests need to be run (unless you also want to check for semantic compatibility, which quickly becomes NP complete ;) )

• What if the tools can tell you with good accuracy what version of a library you can consume safely?

• https://diff.revapi.org
• Animal Sniffer

If you find it interesting, I would love to answer any questions you might have!

[smoothreggae]

The Gradle team recently integrated the japicmp-gradle-plugin into their build to do something similar. Would this help meet some of your needs?

[metlos]

Japicmp is a good tool for detecting binary incompatibilities but IMHO is a little bit lacking in terms of pluggability and extensibility.

The above mentioned https://diff.revapi.org uses a custom "reporter" to output the findings in json that is then consumed by the webpage. The Spoon project for example uses Revapi to check the API changes in all of their PRs and uses a single Freemarker template (used by Revapi) to produce nice reports such as one at INRIA/spoon#1771.

This would have been much harder with japicmp.

Also japicmp seems to be focused on binary compatibility and doesn't detect as many source incompatibilities as Revapi does as far as I know..

On the other hand, japicmp seems to have a larger community..

[mockitoguy]

Thank you for suggestions and links. Indeed, revapi looks interesting!

[magneticflux-]

@metlos Japicmp looks like exactly what is needed, it even comes with a --semantic-versioning that tells you which version to bump when comparing two jars!

Config options in Gradle Japicmp that fit requested features:

• onlyBinaryIncompatibleModified - Outputs only classes/methods with modifications that result in binary incompatibility. Type: boolean. Default value: false
• packageIncludes/packageExcludes
• accessModifier - Sets the access modifier level (public, package, protected, private). Type: String. Default value: public

[metlos]

I'm obviously biased because I'm the author of Revapi, but :-)

While japicmp certainly fits the bill, I really do think that Revapi might be a better choice because of its pluggable design. Japicmp is a single purpose tool with not much choice when it comes to reporting, while Revapi was from the beginning designed with pluggable"reporters".

I also think that reducing the compatibility to just binary compatibility is not the right choice in today's containerized world where stuff gets rebuilt from source. Revapi seems to have a richer set of detected problems in both binary and especially source area. Uniquely, Revapi is able to detect problems like nonpublic classes sneaking into the API through its use chain tracking abilities.

Japicmp on the other hand has a Gradle plugin that Revapi lacks as of now.

All in all I think both solutions would fit your use case, each with its own set of drawbacks ;-)

Whatever you choose in the end I'm very much excited about the prospect of shipkit...

[magneticflux-]

@mockitoguy I've been mulling over this feature for some time now and I feel that implementing it may require a fundamental change in the way versions are incremented. This is how I see Shipkit currently incrementing versions:

1. Merge a PR with version=1.0.1 and previousVersion=1.0.0
2. Travis CI begins a build
3. xyz-1.0.1.jar is uploaded to Bintray
4. New properties are written: version=1.0.2 and previousVersion=1.0.1
5. Changelog, etc. updated
6. Shipkit release commit made, commit tagged, etc.
7. Shipkit pushes release commit, build succeeds

Japicmp/Revapi can determine the version=1.0.2 property, but if they were executed in the same spot that the "New properties are written" step is executed they would be useless since the jar was already uploaded under the current version and there have been no changes yet.

I can see several ways around this issue:

1. Create a separate task autoIncrementVersion to be run by the user before a PR is merged that replaces version=1.0.1 with an automatically incremented version number based off of the current sources and the artifact from previousVersion=1.0.0. This would require an extra step from the user, but it would allow users to make sure what the next version would be called before merging.

2. Keep everything the same except for allowing users to write version=auto instead of a fixed version number. Then, before publishing, fill in the number with an automatically incremented number, upload the binary, set previousVersion to the auto-generated version, and set version=auto again. This would allow users to force a given version to be released but also rely on automatic increments most of the time.

3. Get rid of the version property and always use some algorithm to increment the previousVersion number. The jar version would solely depend on the compatibility of the jar to the previous version. This would free users of Shipkit to concentrate on writing code, and users of the shipped library to know guaranteed binary compatibility.

Do you have any other solutions for this? Which do you think is the most promising?