Rearrange PyPI projects, break out adapters

[ssteinbach]

Overview

The goal of this issue is to refactor the PyPI projects for OpenTimelineIO into two pypi projects from the current one:

• OpenTimelineIO (the current package) becomes ONLY the core python wrapped library
  • Versioned whenever a new release of the core comes out from GitHub (similar to the current release process)
• OpenTimelineIO-Plugins (a new package) is OpenTimelineIO + a curated set of plugins
  • Versioned whenever a plugin is added or removed from the list, or the OpenTimelineIO releases
  • A plugin may be removed from the list if it remains unsupported and falls behind the core, as determined by the OTIO TSC
  • PRs may be made to add or remove plugins from the list
  • The Plugins project will be in its own repository, with a README that describes briefly some criteria that the TSC is looking for when assessing whether to include a project or not
  • Plugins that are not present in OpenTimelineIO-Plugins may still be present on pip. Users may install extra plugins through pip that aren't included in the OpenTimelineIO-Plugins project.

Outstanding tasks:

• Remove the dependency on the contrib project from the top level setup.py
• Add a note to the readme about the OpenTimelineIO-Plugins
• Rename umbrella repository to OpenTimeilneIO-Plugins
• Create pypi project for OpenTimelineIO-Plugins
• add CI integration for pushing to pypi from Plugins repo
• CI integration for checking new versions of OpenTimelineIO from the Plugins repo We'll leave this to adapter repo maintainers to do if they see fit
• register pypi packages for the adapter plugin projects <- at least for the ones that we plan on having the umbrella project include
• break out adapters
  • as a TSC group we identify which adapters we want to include in the default installation
  • each adapter gets its own package, which can be individually installed
  • remove explicit references to the contrib package from the main library (see comment below) and documentation
  • remove the contrib directory from the repository
  • recommend a pinning strategy for how the adapters should handle their version dependencies + some CI for testing against new versions of the CORE
• add version freezing to release script for parent projects No Longer needed, we'll allow leaf adapter projects to set version constraints when needed
• Add versioning strategy (noted below) to documentation (CONTRIBUTING.md or the main documentation)
• Remove references to the adapters from the main documentation (or update them to point at the individual adapters, mention that they're accessible via the -Plugins project)
• CI for adapter projects need create new versions on manual activation rather than when tags are created
• Show the plugin version in the otiopluginfo command output
• Add a documentation page that describes project layout and how that relates PyPI package
• Make sure adapter repos are version pinning to OpenTimelineIO>=0.17.0.dev1
• Stage PRs in adapter repos changing version pinning to OpenTimelineIO>=0.17.0

Versioning strategy

• OpenTimelineIO-Plugins project pins to a specific OpenTimelineIO project, corresponding to the current release, using the == constraint
• Each adapter picks a versioning strategy against the OpenTimelineIOversion number
• the -Plugins project depends on a set of adapters in a floating way, which lets the adapter repos individually deal with their dependency on the core
• OpenTimelineIO-Plugins gets versioned only when plugins get added or removed, or the pinned version of OpenTimelineIO is updated

Why the OpenTimelineIO Does Not Include The Plugins ("OpenTimelineIO-core vs OpenTimelineIO-

Currently, if you pip install opentimelineio you get the contrib adapters + the core library. This suggestion switches that around such that pip install opentimelineio is only the core without the contrib adapters. We decided to make this shift for several reasons:

• We want to make it clear to what degree the main project provides a warranty for components, and we want to emphasize the core format
• We can update the version of the main OpenTimelineIO package without the plugins being updated if there is a compability issue
• For software vendors, its more clear to identify the project version a software package uses or provides compatability for

Question

• if we break an adapter from a change to OTIO core, how/when do we catch that? Do we [re]introduce smoke testing in the Plugins repo?

[mikemahony]

Are we imagining all the packages (OpenTimelineIO umbrella, OTIO-core, and other adapters) would live in the same, existing repo? Or do we want them each split out into their own git repositories?

[timlehr]

I'd be in favour of splitting them into their own repos.

[jminor]

The adapters should definitely go into separate repos (one per adapter) so that experts in each one can be given ownership over them.

As for the core and the umbrella, I don't know. Is there a tradeoff worth considering? I can sort of imagine what a separate umbrella repo would look like, but I'm unclear on the details. What would it look like to have them together in one repo?

[apetrynet]

Is checkbox number two in the list the GitHub project itself or the project on pypi?
I'm assuming the latter(?)

So when pip install OpenTimelineIO you get core and default adapters?

Would it make sense to follow the pip install OpenTimelineIO[core] style?
Then we could do:

• core (core only)
• coreadapters (only the main adapters maintained by the otio project)
• contrib (a curated list of contrib adapters)
• all (all of the above)

I'm not sure if this is possible though, as I don't have enough experience with setting such a structure up. The contrib key might be tricky to update without releasing new main packages?

And I agree about splitting adapters to separate repos

[JeanChristopheMorinPerso]

I would see the "core" as a separate package (but still in the opentimelineio repo), opentimelineio-core. Installing opentimelineio should install the core I think. From there, we have two options:

1. Adapters are installed with pip isntall opentimelineio[adapters]. From what I remember we kind of already ruled this option out because the user experience isn't great.
2. Adapters are installed with opentimelineio (with the core too). This gives a nice experience to users.

As for the core and the umbrella, I don't know. Is there a tradeoff worth considering? I can sort of imagine what a separate umbrella repo would look like, but I'm unclear on the details.

If the core and the meta package (the umbrella) live in different repos, I think it could make it confusing to users. Our main repo would be named OpenTimelineIO like it is right now. But the meta package repo... How would it be named? Also, wouldn't it be awkward to have a repo that doesn't match the package name? Though that would be the case for the core package, but I don't see that much as a problem for core.

One thing that we might want to consider is versioning and version constraints and how we will want to bundle things. Will the meta package define its dependencies with lower bound versions or without constraints at all? What happens when new features are added to OTIO (or breaking changes)? Should adapters be compatible with multiple versions of OTIO? In that same line of thought, should they use lower bound constraint (>=0.14)) on OTIO version or both lower and upper bounds (>=0.14, <=1.17)? (specifying a maximum would probably cause problem down the line).

[jminor]

@mikemahony and I made this empty repo for experimentation. We can sketch this out and see how it fits before deciding on the final layout: https://github.com/OpenTimelineIO/otio-umbrella

I agree that having the repos and the pypi packages with clear names is important. The version history of a separate umbrella repo seems appealing too, but not essential.

[ssteinbach]

Note that there are a few explicit references to the contrib package in the code:

OpenTimelineIO/src/py-opentimelineio/opentimelineio/plugins/manifest.py

Line 327 in 5e27e78

# the contrib plugin manifest (located in the opentimelineio_contrib package)

Search for "contrib" in manifest.py. I see some in the autogen documentation as well, might be some others:

OpenTimelineIO/src/py-opentimelineio/opentimelineio/console/autogen_plugin_documentation.py

Line 348 in 5e27e78

contrib_manifest_path = manifest_path_list[1]

I added a todo list item to remove these references.

[mikemahony]

Putting these here for something to look at:

Option A - Core/Umbrella Separate Repositories
----------------------------------------------

 Owner            | Repository          | Package
 -----------------+---------------------+-----------------
  ASWF            |  OpenTimelineIO     |  otio-core
  OpenTimelineIO  |  otio-umbrella      |  opentimelineio
  OpenTimelineIO  |  otio-aaf-adapter   |  otio-aaf-adapter
  gilou           |  otio-drp-adapter   |  otio-drp-adapter
  apetrynet       |  otio-mlt-adapter   |  otio-mlt-adapter
  OpenTimelineIO  |  otio-xges-adapter  |  otio-xges-adapter   <-- owner ok?
  OpenTimelineIO  |  otio-ale-adapter   |  otio-ale-adapter   <-- owner ok?
  OpenTimelineIO  |  otio-rv-adapter    |  otio-rv-adapter   <-- owner ok?
  ...             |  ...                |  ...

Option B - Core/Umbrella Combined Repository
--------------------------------------------

 Owner            | Repository          | Packages
 -----------------+---------------------+-----------------
  ASWF            |  OpenTimelineIO     |  otio-core, opentimelineio (umbrella)
  OpenTimelineIO  |  otio-aaf-adapter   |  otio-aaf-adapter
  ...             |  ...                |  ...