feat: Upstream Sync with Content Library Blocks

[kdmccormick]

Changes:

• Create a new UpstreamSyncMixin which is added to all CMS blocks, including:
  • New fields upstream, upstream_version, downstream_customized, and upstream_display_name, and upstream_max_attemps
  • A customized save() method which updates blocks
• Add three Python API functions for managing upstream syncing: sync_from_upstream, inspect_upstream, and inspect_upstream_as_json
• Add ajax endpoint POST /api/contentstore/v2/upstream_sync/{usage_key} for pulling the latest updates from upstream into a block
• Add link_to_upstream option to the paste API so that blocks copied from libraries can be linked back to their source
• Expose version_num from ContentLibraryXBlockMetadata

ADR: https://github.com/kdmccormick/edx-platform/blob/kdmccormick/upstream-proto/docs/decisions/0020-upstream-block.rst

Demo PR (not working at the moment): openedx/frontend-app-authoring#1202

[kdmccormick]

@bradenmacdonald So I've got this draft version of upstream sync working, implemented with XBlock fields and handlers. Using the XBlock fields and handlers for this was nice because it gave me persistence, URL routing, and OLX serialization for free.

But now I'm wondering if this is bad approach because we'll need to support Units and Subsections in libraries next, and we've agreed that those structural blocks should stop being XBlocks eventually.

The alternative I see is to:

• use Django models to store upstream links, upstream defaults, and upstream overrides,
• define REST/Python APIs for managing them, and
• handle OLX serialization as special case

Thoughts?

[kdmccormick]

use Django models to store upstream links, upstream defaults, and upstream overrides,

Okay, I started prototyping this, and it's easier said than done. Particularly, if you imagine a model like this, there's a question of whether we FK to entities or entityverisons:

class UpstreamDownstreamLink(models.Model):
    """
    Link a piece of content (the downstream) to a source for updates (the upstream).
    """
    # Pretty straightforward -- FK to a particular upstream version, and then update the
    # version whenever sync happens.
    upstream = models.ForeignKey(PublishableEntityVersion)

    # Which one of these is right?? Neither?

    # This one would apply to all versions of the downstream component, so it would play weird
    # draft/publish, and would fail if we ever added an "undo" button to studio.
    downstream = models.OneToOneField(PublishableEntity)

    # This seems more correct, but would require updating every time a new version of the downstream
    # component is created. Perhaps this would work fine as a cache, but it seems wrong as the authoritative
    # data source.
    downstream = models.OneToOneField(PublishableEntityVersion)

It seems to me, then, that we do want to persist upstream metadata using XBlocks fields, so that version management works as we expect. Still, I'm open to feedback on whether we should use XBlock handlers vs. build a separate REST API for getting/settings links and syncing from upstream.

[open-craft-grove]

Sandbox deployment failed 💥
Please check the settings and requirements.
Retry deployment by pushing a new commit or updating the requirements/settings in the pull request's description.
📜 Failure Logs
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[bradenmacdonald]

@kdmccormick I think it's fine to implement this with fields and handlers for now; as you said, it gives you a lot for free. It even gives us most of the Unit and Subsection functionality for free, I think. Yes, we want to implement those differently in the future but we aren't going to be able to make that change anytime soon, and I think we can worry about how library sync works in that case after we have the new design for how that stuff works. Right now it's mostly an unknown.

e.g. I have an idea for an "outline" object that encodes the course hierarchy and library content and much more, and if we implemented it that way, we'd have to re-implement even your django model based prototype. Or previously I had proposed this compositor architecture which is different yet again. So I think it's not worth planning for that too much until we've got an ADR and buy-in about what the future direction is for the non-XBlock way to represent hierarchy.

[open-craft-grove]

Sandbox deployment successful 🚀
🎓 LMS
📝 Studio
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[open-craft-grove]

Sandbox deployment successful 🚀
🎓 LMS
📝 Studio
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[open-craft-grove]

Sandbox deployment successful 🚀
🎓 LMS
📝 Studio
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[open-craft-grove]

Sandbox deployment successful 🚀
🎓 LMS
📝 Studio
ℹ️ Grove Config, Tutor Config, Tutor Requirements

[ormsbee]

This would get called during course import, right? If so, maybe just log this as an error for now? At some point we may add other supported key types that people will have in their Teak-generated content exports. Sumac might not recognize those or be able to hook it up correctly when importing, but it should try not to explode the import process.

[ormsbee]

I think we should do that thing we talked about where we have a fixed set of known overridable fields, and not generally pull from the settings scope. Otherwise we're going to make pretty much the entire VideoBlock overridable:

edx-platform/xmodule/video_block/video_xfields.py

Lines 33 to 122 in ca46c20

	youtube_id_1_0 = String(
	help=_("Optional, for older browsers: the YouTube ID for the normal speed video."),
	display_name=_("YouTube ID"),
	scope=Scope.settings,
	default="3_yD_cEKoCk"
	)
	youtube_id_0_75 = String(
	help=_("Optional, for older browsers: the YouTube ID for the .75x speed video."),
	display_name=_("YouTube ID for .75x speed"),
	scope=Scope.settings,
	default=""
	)
	youtube_id_1_25 = String(
	help=_("Optional, for older browsers: the YouTube ID for the 1.25x speed video."),
	display_name=_("YouTube ID for 1.25x speed"),
	scope=Scope.settings,
	default=""
	)
	youtube_id_1_5 = String(
	help=_("Optional, for older browsers: the YouTube ID for the 1.5x speed video."),
	display_name=_("YouTube ID for 1.5x speed"),
	scope=Scope.settings,
	default=""
	)
	start_time = RelativeTime( # datetime.timedelta object
	help=_(
	"Time you want the video to start if you don't want the entire video to play. "
	"Not supported in the native mobile app: the full video file will play. "
	"Formatted as HH:MM:SS. The maximum value is 23:59:59."
	),
	display_name=_("Video Start Time"),
	scope=Scope.settings,
	default=datetime.timedelta(seconds=0)
	)
	end_time = RelativeTime( # datetime.timedelta object
	help=_(
	"Time you want the video to stop if you don't want the entire video to play. "
	"Not supported in the native mobile app: the full video file will play. "
	"Formatted as HH:MM:SS. The maximum value is 23:59:59."
	),
	display_name=_("Video Stop Time"),
	scope=Scope.settings,
	default=datetime.timedelta(seconds=0)
	)
	#front-end code of video player checks logical validity of (start_time, end_time) pair.
	download_video = Boolean(
	help=_("Allow students to download versions of this video in different formats if they cannot use the edX video"
	" player or do not have access to YouTube. You must add at least one non-YouTube URL "
	"in the Video File URLs field."),
	display_name=_("Video Download Allowed"),
	scope=Scope.settings,
	default=False
	)
	html5_sources = List(
	help=_("The URL or URLs where you've posted non-YouTube versions of the video. Each URL must end in .mpeg,"
	" .mp4, .ogg, or .webm and cannot be a YouTube URL. (For browser compatibility, we strongly recommend"
	" .mp4 and .webm format.) Students will be able to view the first listed video that's compatible with"
	" the student's computer. To allow students to download these videos, "
	"set Video Download Allowed to True."),
	display_name=_("Video File URLs"),
	scope=Scope.settings,
	)
	track = String(
	help=_("By default, students can download an .srt or .txt transcript when you set Download Transcript "
	"Allowed to True. If you want to provide a downloadable transcript in a different format, we recommend "
	"that you upload a handout by using the Upload a Handout field. If this isn't possible, you can post a "
	"transcript file on the Files & Uploads page or on the Internet, and then add the URL for the "
	"transcript here. Students see a link to download that transcript below the video."),
	display_name=_("Downloadable Transcript URL"),
	scope=Scope.settings,
	default=''
	)
	download_track = Boolean(
	help=_("Allow students to download the timed transcript. A link to download the file appears below the video."
	" By default, the transcript is an .srt or .txt file. If you want to provide the transcript for "
	"download in a different format, upload a file by using the Upload Handout field."),
	display_name=_("Download Transcript Allowed"),
	scope=Scope.settings,
	default=False
	)
	# `sub` is deprecated field and should not be used in future. Now, transcripts are primarily handled in VAL and
	# backward compatibility for the video blocks already using this field has been ensured.
	sub = String(
	help=_("The default transcript for the video, from the Default Timed Transcript field on the Basic tab. "
	"This transcript should be in English. You don't have to change this setting."),
	display_name=_("Default Timed Transcript"),
	scope=Scope.settings,
	default=""
	)

And we're going to run into that weird issue where the ProblemBlock's markdown field is settings scope, but changes to it try to write to the content scoped data field and just mess everything up:

edx-platform/xmodule/capa_block.py

Line 323 in ca46c20

markdown = String(help=_("Markdown source of this module"), default=None, scope=Scope.settings)

[kdmccormick]

Suggested change

	self.upstream_overridden.append(field_name)
	self.upstream_overridden.add(field_name)

upstream_overridden is now a Set, not a List

[ormsbee]

What do you think (overall) of the current implementation?

Overall, I think the approach is good–especially given where we are today. My only review comments are on lower-level details.