Mediawiki MobileView API deprecated, Mobile Content Service deprecated too, migrate to WMF-recommended solution

[nicolas-raoul]

https://phabricator.wikimedia.org/T210808

https://www.mediawiki.org/wiki/Mobile_Content_Service

[awanishyadav967]

Hi @nicolas-raoul Please assign to me ,I will work on this.

[nicolas-raoul]

@awanishyadav967 It is yours, thanks! Please let us know about your progress or share your findings every few days.

[kartikaykaushik14]

@nicolas-raoul The issue seems to be unassigned. Could you please assign the issue to me?

[nicolas-raoul]

@kartikaykaushik14 Of course, thanks!

[kartikaykaushik14]

@nicolas-raoul I was going through the mobile content service API Documentation and just last week they posted a decommissioning notice and asked to migrate to Page Content Service - Reference

[nicolas-raoul]

@kartikaykaushik14 Thanks for the hint!

It looks like Parsoid is JavaScript, it might be usable via WebViews.

One could argue that we should follow what the Android Wikipedia app is doing, which is apparently the Page Content Service ("PCS"), but a counter-argument would be that it is unstable.

The first task would be to list our different use cases of the API, to see how well these use cases are covered by Parsoid and PCS, possibly via a tiny prototype app.

[kartikaykaushik14]

I was trying to analyze which REST endpoints are impacted by the deprecation of MobileView API and their possible alternatives from Page Content Service.

1. To GET the Lead Section and initial metadata of a given title. Alternative: getSectionsLead
2. To GET the remaining sections of a given title. Alternative: getSectionsRemaining
3. To GET the remaining sections request the URL of a given title. However, this endpoint is only to get URLs for the remaining sections and once we have an alternative to get the remaining sections directly, we do not need this endpoint.

But, the endpoints are currently unstable as per current documentation.

[kartikaykaushik14]

Even these endpoints in Page Content Service are now deprecated.

However, mobile-html endpoint uses Parsoid HTML. But it's currently in the experimental phase.

[mnalis]

Are there any updates on Mobileview API replacement?
July 2023 (when this API is supposed to stop working) is approaching rapidly.

Also, what is expected to break if MCS API is disabled on server side if commons app doesn't have it replaced yet?
Will the commons app remain usable (fully or partially), or stop working completely?

[sivaraam]

@kartikaykaushik14 Are you still working on this issue? If so, could you update your progress on this?

[sivaraam]

@mnalis Thanks for the nudge. As far as I could see, MCS has not yet been decommissioned. It still works but it has been deprecated [ ref ].

So, I suppose we should explore migrating to a different API soon.

[nicolas-raoul]

I think that could be a GSoC project. If too easy, this plus #3083.

[kartikaykaushik14]

@sivaraam Sorry I couldn't continue further on the issue. Will be a little free from work in the upcoming weeks. But feel free to reassign this issue or make it a part of next GSoC.

[sivaraam]

No issues @kartikaykaushik14 ! Kindly feel free to work on this if you're able to find time.

At a later stage, if there's anything that's left to be converged, we could convert that remaining part into a GSoC project.

[sivaraam]

Also, what is expected to break if MCS API is disabled on server side if commons app doesn't have it replaced yet? Will the commons app remain usable (fully or partially), or stop working completely?

We pinged the WMF team for some inputs on how to go about with the migration. @dbrant kindly took a look at our code and provided the following clarification:

And regarding Mobile Content Service deprecation: sorry there's been a
fair amount of confusion about this. From a cursory look at your code, it
doesn't seem like you actually use any of the endpoints that are
deprecated. Even though those functions and model classes are declared in
your "data-client" subproject, they seem to be only used in unit-testing
methods, all of which can be safely removed. (unless I missed a use-case
for these functions in your code.)

So, I think the only thing left to do here is to identify the stale portions of code and remove them :-)

[nicolas-raoul]

@kartikaykaushik14 Are you planning to work on this? :-)

[psh]

I've spent some time digging through the data-client module to remove "dead code" that appears to be unused. If you're curious, it's on a branch that you can browse.

If we're looking for a migration path to a different API, I hope you find the investigation useful!

[nicolas-raoul]

@psh Wonderful! Feel free to send a pull repuest. 🙂

[psh]

From what I have seen, now that dead code is pretty much gone, is that we are using 5 things from the data-client module that could all be pulled over and incorporated into the main commons app code and the module deleted -

• Base data model classes for wikimedia actions API requests / responses
• CSRF Token management
• Notifications
• Login
• Thanks request/response

In terms of size, "thanks" and "notifications" are probably the easiest and mirrors other self-contained API calls we make in the app as far as work goes. Something like CategoryInterface and CategoryClient might be a template for how it would look.

Login already exists in the app, so we'd be moving 4 classes over from the data client. The CSRF client makes use of it, so maybe those have to move together. 🤔 Seems like a logical grouping to me.

If we leave the data model base classes for last, then (I think) each of the other steps is self-contained and you wont end up with strange circular dependencies of the data client module depending on the main app.

[sivaraam]

Sounds like a good plan. Thanks a lot for taking the time to help with the clean up of the data-client module! 🙂

[psh]

OK, quick "state of the union" update for those who are interested -

• Commons app is now 100% self-contained as far as the API service interfaces are concerned. They're created and managed consistently across the board and we should be able to see exactly the APIs we're using
• Slightly deeper, the OkHttp and Retrofit caching is now more efficient and shared across more of our API classes. I may be a little biased, but I think performance and memory usuage have improved 😄
• Each of the PR so far has (a) moved to newer Kotlin and (b) reduced overall lines of code

So, what's left?

• The AppAdapter abstraction is 90% unused, and digging into it, manages just cookie persistence and our OkHttp instance. Since OkHttp uses the CookieJar these amount to roughly the same issue and will be addressed in an up-coming PR. The AppAdapter will then go away.
• Model classes in the "data-client" will need to merge with their commons app sibling (as appropriate) or move.
• Deletion of the data-client - LOTS of files will be touched in the form of deletions. It will be a pretty big commit but most of the work has already been done to move away from it.

[nicolas-raoul]

Wonderful! Thanks for all of this great work! 🙂

[psh]

Quoting from earlier in this issue -

And regarding Mobile Content Service deprecation: sorry there's been a
fair amount of confusion about this. From a cursory look at your code, it
doesn't seem like you actually use any of the endpoints that are
deprecated. Even though those functions and model classes are declared in
your "data-client" subproject, they seem to be only used in unit-testing
methods, all of which can be safely removed. (unless I missed a use-case
for these functions in your code.)

So, I think the only thing left to do here is to identify the stale portions of code and remove them :-)

With the most recent commit (thanks to @nicolas-raoul for his patience as it was pretty huge) the "data-client" module is gone. The small portion of code from it that we were using is incorporated into the commons app proper.

I believe that wraps up the work we wanted to do here, unless there are other API related cleanup operations to address? Maybe we can gather our thoughts in this thread, spin off separate issues if we spot things we want to do, and close this one?

[sivaraam]

I believe that wraps up the work we wanted to do here, ...

I suppose so too. Thanks a lot for taking care of this in review friendly part-by-part fashion, @psh! 🙂

... unless there are other API related cleanup operations to address?

I can't recollect any API cleanup similar to this one that we've left out. On a tangent, there are a few API related issues such as #5244 / #3179 that are still open in our issue tracker. Feel free to take a stab at them if you're interested.

If you're looking for other kind of issues, feel free to elaborate. We could identify relevant ones.

Maybe we can gather our thoughts in this thread, spin off separate issues if we spot things we want to do, and close this one?

Most certainly 👍🏼

[nicolas-raoul]

Thanks a lot Paul for this fantastic work! 🙂