[DEPR]: Replace cs_comments_service

[regisb]

Proposal Date

2024-04-16

Target Ticket Acceptance Date

2024-10-16

Earliest Open edX Named Release Without This Functionality

Sumac - 2024-10

Rationale

The Open edX forum is an essential feature of the platform. In technical terms, it is a web application that is developed in the cs_comments_service repository. This application is implemented in the Ruby programming language. Only a few Open edX contributors have expertise in Ruby: most developers are skilled in Python and NodeJS. This makes it extremely difficult to implement new features and extend the forum.

Moreover, the cs_comments_service application currently stores data in MongoDB, which is neither a consistent or a relational data storage solution -- as opposed to MySQL. This results both in operational and architecture complexity, as some data needs to be synced up between the two data stores (notably for user data).

Removal

The cs_comments_service repository will be removed. Note howerver that the tutor-forum plugin will be maintained, though it will be considerably simplified.

Replacement

The Ruby application will be replaced by a Django app, currently being developed in the following repository: https://github.com/edly-io/forum/ (soon to be migrated to the openedx organization, as per this public-engineering issue).

Deprecation

The cs_comments_service repository will include a deprecation notice in its README file. Migration will be transparent for most users running Tutor, as the tutor-forum plugin will smoothly transition from running the former ruby app to installing the new forum application in the edx-platform runtime.

Migration

Strategy

Let's call the legacy cs_comments_service "forum v1" and the new forum application "forum v2".

In forum v1, edx-platform communicates with the Ruby application via an HTTP rest API. All calls are made from the backend, Client-side API calls are made to edx-platform, which then forwards parts of the requests to forum v1.

In forum v2, these API calls take the form of a native Python API. This is made possible by the fact that the forum v2 application is installed in the same runtime as edx-platform.

This means that the diagram above will remain the same in the transition from v1 to v2. The only difference is that in step 3, communication will change from an HTTP API call to a native Python function call.

For storage, forum v2 will implement two different backends: one for MySQL (in the form of Django models) and one for MongoDB. The MongoDB backend is only preserved to ease the transition from v1 to v2; it is expected that it will be eventually removed.

Opt-out

Forum v2 will make use of two CourseWaffleFlag (source code). For every course, platform administrators will have the choice to:

1. Send HTTP requests to the forum v1 app or native API calls to the forum v2 app.
2. If calling the forum v2 app, use either the legacy MongoDB backend or the new MySQL backend.
3. Run a dedicated migration command that will copy the course data from MongoDB to the new MySQL models.

By default, users running Tutor (and the tutor-forum plugin) will automatically transition from v1 to v2 and the MySQL models. In that sense, the transition is opt-out rather than opt-in.

Proposed timeline

• July - October 23rd 2024: implementation of forum v2
• October 23rd 2024: Forum v1 marked for deprecation: no new change accepted in the cs_comments_service repository.
• October 23rd - December 9th 2024: Sumac testing, fixing issues in forum v2.
• December 9th 2024: Sumac release
  • By default, Tutor will run forum v2. The storage backend will depend on whether or not it's a new tutor installation
    • Existing platforms will keep using MongoDB: instructions to migrate will be printed on tutor ... launch and be available in the tutor-forum README.
    • New platforms will default to the MySQL backend.
  • It will be possible to override this default behavior on a course-per-course basis.
• December 2024 - April 2025: all platforms are expected to migrate from forum v1 to forum v2, with the MySQL storage backend.
• June 2025: Teak release:
  • the forum v1 integration is removed from edx-platform
  • the mongodb backend is removed from forum v2

Additional Info

• This project was first announced in April 2024 here: https://discuss.openedx.org/t/axim-funded-contribution-proposal-forum-rewrite-from-ruby-mongodb-to-python-mysql/12788
• DEPR Announcement: https://discuss.openedx.org/t/deprecation-removal-forum-cs-comments-service-437/14056
• Then the technical implementation was discussed here: https://docs.google.com/document/d/1oa-L-ETJkXIPEOxLLOlcENE49o3kSh_HNVp-5gouwAc/edit
• Implementation of the forum v2 app started in July 2024 and project was managed publicly here: Forum rewrite: Ruby -> Python / Mongodb -> MySQL (view)
• DEPR announcement: https://discuss.openedx.org/t/deprecation-removal-forum-cs-comments-service-437/14056
• For reference, here are the guidelines for deprecation announcements: https://open-edx-proposals.readthedocs.io/en/latest/processes/oep-0021-proc-deprecation.html

Task List

• Migrate the edly-io/forum repository to the openedx organisation: [GH Request] Move edly-io/forum repository to the openedx organization axim-engineering#1272
• Merge the following pull request in edx-platform: Edly: Integrate Forum V2 Application into edx-platform edx-platform#35671
• Update the tutor-forum plugin to replace forum v1 by forum v2
• Document the opt-out strategy in forum v2
• Testing:
  • Forum v2 and MySQL storage: tested during the Sumac testing period
  • Migration from MongoDB to MySQL: coordinate with 2U to test the correctness of the migration scripts.

[kdmccormick]

It sounds like the "Earliest Open edX Named Release Without This Functionality" will actually be Teak, not Sumac?

[regisb]

@kdmccormick Yes, I wasn't sure to indicate Sumac or Teak in this field. We are proposing that cs_comments_service be removed in Teak but, technically speaking, Sumac will be the first release to ship with the replacement of cs_comments_service by default. In the end I decided to announce the change for Sumac to avoid taking people by surprise.

[ormsbee]

@hurtstotouchfire: Do you folks have time to give some feedback on this proposal?

Thank you.

[regisb]

I'll recap here the decisions that were reached in the BTR working group meeting that happened yesterday:

• Forum v2 support will be merged in edx-platform in Sumac.
• Platforms not running Tutor will keep running forum v1 by default.
• Platform running Tutor will run forum v2, but with different data backends:
  • Existing platforms will keep using MongoDB: instructions to migrate will be printed on tutor ... launch and be available in the tutor-forum README.
  • New platforms will default to the MySQL backend.
• Platforms running Tutor and which need to stick to forum v1 will have to create their own Tutor plugin, likely based on top of this version.
• Legacy forum v1-related code will be removed from edx-platform before Teak.
• Legacy MongoDB related code will be removed from forum v2 before Teak, with the exception of the migration script, which will remain available in Teak.

[feanil]

I've updated the timeline in the description to match this recap.