Description
This issue has been opened by SuperFly.tv on behalf of EVS Broadcast Equipment (henceforth EVS).
We propose to extend the Sofie API to present an ingest API to allow ingest data to be sent to Sofie from external sources via HTTP rather than DDP.
Motivation
There is a desire for an external system to be able to "push" data to Sofie via an API. As the HTTP API has been used in other areas of EVS work, it is desirable to also have this external system use the HTTP API.
Proposal
A draft of the shape of the API has been published and a PR has been opened.
There are a couple of patterns to point out in this draft. The first is the use of ETags and If-None-Match, the second is that the proposed API allows for external Ids to be used interchangeably with Sofie-internal Ids. Both patterns are explored below.
ETags and If-None-Match
The ETag
and If-None-Match
HTTP headers have been included as part of this proposal as a way of identifying the version of each ingest item (e.g. Rundown, Segment) when it is pushed to Sofie. Using these headers means that an external system can provide us a version string for some ingest data via the ETag
and tell us to only update an existing item if it has a different ETag
to the one provided. This means that we have a chance of defending against too many updates being sent by an external system, so long as the external system provides us with a sensible ETag
to use as the version of the data (e.g. timestamp that the data was last modified).
Using this method also allows the entire API to be expressed as PUTs while still allowing for "forced" updates - e.g. in the case of re-syncing data. We achieve this by specifying that any update that is not accompanied by an If-None-Match
header is always treated as a "create" event rather than an update, meaning that we will overwrite any existing data in this case.
This approach also allows for some performance improvements to be made in some cases, as we will be able to discard an update without performing any costly work within blueprints if we have received data that has the same version string as the object already in the database.
However, to use this version information we will need to store it somewhere which means that this proposed change does mean a change to the data present in the ingest data collections in the form of an optional version: string
property. Also of note is the fact that bulk updates (e.g. a PUT
for a list of Rundowns) will use If-None-Match
to optimize bulk updates, but will not accept an ETag
for these updates, so the next update to each individual element within a collection will be treated as a create. This is in-line with the recommended usage of ETag
. The effect of this is likely to be that we will be able to optimize the initial ingest of data at the expense of unnecessary updates in the future when script edits are carried out on individual elements of the show.
Interchangeable Ids
In the current proposal, external systems can make calls to the Ingest API using either the Id internal to Sofie's database, or using an arbitrary external Id (so long as this Id is sufficiently unique). This allows the external system to interact with Sofie purely through its own Ids, without having to perform lookups to Sofie's API to maintain its own map between Ids. This also means that we can express the entire API as a series of PUTs rather than needing data to be created via a POST initially, which reduces the need for the external system to have an awareness of the data that is currently held within Sofie and by extension reduces the amount of knowledge the external system needs to have about the lifecycle of Sofie.
The cost of this approach should be minimal as it can be implemented using the $or
operator in Mongo queries. Also, Ingest operations are generally not so time-sensitive that this lookup will cause excessive delay.
It may be useful to extend this approach to other areas of the API but that is outside the scope of this RFC.
Other things of note
The design of this API is such that the ingest objects contain a generic payload
object, the exact shape of which is left to be agreed between the blueprints authors and the authors of the system that is pushing data to Sofie - in much the same way that the existing ingest gateways work. This also follows the pattern established by the Studio and ShowStyle config APIs, so anything that is developed in terms of validation and schema discovery for those methods can be shared with this API.
As part of the Playlist ingest objects, a resyncURL
can optionally be provided that will be used by the "Reload data" action within the Rundown UI. This URL will receive a POST
with an empty body to prompt an external system to resync data with Sofie, which is described as a request for all data for a Playlist to be re-sent to Sofie without the If-None-Match
header present.
Request For Comment
- Does this make sense to expose over the HTTP API?
- Does the proposed shape make sense?
- Are we happy to employ this pattern of mixing external Ids with Sofie-internal Ids?