At each commit to master, automatically build an APK and send it to Google Play alpha

[nicolas-raoul]

Since we are using branches, the master branch should always work. In other words, if a bug appears in the master branch then we want to know it immediately.

We have a good number of beta testers (Commons users with no development skills), but betas are rather rare because building and uploading them wastes a lot of the maintainer's time.

Solution:
Whenever something is committed to master:

• Increase the beta minor version number
• Build the APK
• Sign it with the real keys
• Upload it to Google Play's beta

It might sound a bit challenging, but I believe all of the pieces to set this up exist:

• Use Travis to build and save an APK: https://gist.github.com/domenic/ec8b0fc8ab45f39403dd
• Use Google's API to upload the APK: https://developers.google.com/android-publisher/#publishing

[misaochan]

Wow, interesting. I had no idea we could automate that! If we could get it to work, indeed it would benefit us tremendously - less work for me uploading betas and easier testing of the master branch.

The only issue I'm seeing with this is that there will be a LOT of versions being uploaded without changelogs so it may be difficult to keep track of all of them?

[misaochan]

Edit: Also, if we decide to do this we will probably need to ask WMF if they will be okay with the very frequent releases. I know it notifies me in the dev console every time the Wikipedia app is updated, so they might find themselves getting spammed by our updates if they don't adjust their notifications (and I'm not sure if it's possible to turn off notifications for one particular app).

[nicolas-raoul]

With a beta for each commit, the changelog is available without any effort: It is just the list of Git commits.
It makes it very easy to spot what commit introduced a bug. Testers can proceed by dichotomy and tell us exactly what commit triggers the problem, that too saves a lot of developer time.

I don't see any problem with "too many" releases. https://en.wikipedia.org/wiki/Release_early,_release_often

[maskaravivek]

We don't use Crashlytics for beta releases but otherwise Crashlytics is a very good option for such frequent beta releases. I guess using Crashlytics would have privacy concerns, right?

[psh]

Using Fabric Beta to distribute the app is different than using the Crashlytics crash reporting tool. The privacy concerns are different in this case and offer people a clear opt-in / opt-out - users can still use the real app, but if they have concerns about getting test builds then they have a clear path for avoiding them.

There would be no runtime component baked into the Commons app if we used Beta for distribution. That said, there's a minimal build.gradle change to give you the crashlyticsUploadDistributionDebug build task that does the work and some keys to manage.

My current project team uses Fabric Beta to distribute QA builds of the app as it's a one-stop-shop for comparing different built versions and simple for the QA team to receive builds. Plus it plays nice with CI servers for automated builds (we use Bitrise). Prior to production releases we push to the Google Play "alpha channel" to smoke test the app before it rolls out to anyone external. Finally, the Google Play "beta channel" is used for friends & family beta testing before a wide release goes out containing newsworthy new features.

[maskaravivek]

Yes, I was also suggesting usage of Fabric Beta for distribution of test builds. If we get a go ahead that it wont violate any of our privacy policies for users then the integration would be quite simple and can be automated.

[tshradheya]

Can I take this up?

[misaochan]

@psh Interesting! Fabric Beta doesn't actually use the Google Play channels, right? The entire system appears to be self-contained. I would personally be OK with this, but would probably need to check with WMF.

If we don't use Fabric Beta, and elect to use a script to upload to Google Play Store instead... I wonder if we should create an alpha testing group for this purpose? There are a few reasons why I think this might be preferable to uploading to the Play Store's beta:

• We have 300+ beta testers, and master is not always stable. Theoretically it SHOULD be, but realistically not always. :) I always try to thoroughly test manually-released versions, so there is an extra check between master and the users in manually-released versions. I'm not sure if we really want to release code that hasn't been double-checked to 300 users (10% of our userbase)?

• Such frequently-released betas will result in no single version getting more than a day or two of testing time. I think ideally we would like to have the 300 beta testers as a buffer between the alpha users (who will get almost-daily builds) and production users.

So if we go with alpha testing, I figured we can invite people within our close group to join that, and do the automatic build/upload with that. That will still help us hugely with testing, as it will allow the non-devs to help us test the current master, and is also easier than building from master all the time. Then I will do the beta and production releases as per normal. @nicolas-raoul what do you think?

[nicolas-raoul]

Yes, Google Play's alpha is perfect for this use :-)
Personally I would choose the alpha even if it means constants updates, but most testers would probably choose beta.

[misaochan]

Reposting from @ujjwalagrawal17 at #1482

We can also add Fastlane for automatic deployment of the beta App. (Maybe on successful merge to release branch or master branch)
Link: https://docs.fastlane.tools/getting-started/android/beta-deployment/