[RFC] Push Notifications API

[omarroth]

Related issues: #195, #427

Currently, in order for an application to receive updates from a channel, it needs to send a request to /api/v1/channels/latest/:ucid or similar endpoint. This is inefficient when the number of channels being requested reaches even moderate size. This propsal is for a /api/v1/notifications that uses EventSource to push notifications in near real-time.

The EventSource protocol pushes updates using a persistent connection to an HTTP server. It is supported in all major browsers with a polyfill available for IE. The EventSource API can automatically reconnect to the source on errors, and provides a simple API for handling events. There is also a wrapper reconnecting-eventsource for handling more challenging network conditions.

The EventSource protocol is quite simple and can be implemented easily in any language where it isn't already supported. It sends comments (lines prefixed with :), or events containing: one or more data fields, an optional id, and an optional type. After each message is sent a trailing newline. Example from the spec:

: test stream

data: first event
id: 1

data:second event
id

data:  third event

/api/v1/notifications

The endpoint /api/v1/notifications would support a topics parameter containing a list of comma-separated resources. Each resource would be a UCID or any other supported ID. /api/v1/notifications would support a maximum of 1000 topics per call.

Opening a connection to /api/v1/notifications creates a persistent connection between the server and the client, with :keepalive #{TIMESTAMP} messages sent every 20-30 seconds to ensure the connection is not dropped.

Each event would contain a data field with the updated resource as JSON. Schema would be the same as /api/v1/channels/latest/:ucid.

Example output from cURL:

$ curl https://invidio.us/api/v1/notifications?topics=UCAAAA,UCBBBB
:keepalive 1554427320

id: 0
data: {"title":"...","videoId":"AAAAAAAAAAA","author":"...","authorId":"UCAAAA","authorUrl":"/channel/UCAAAA","videoThumbnails":[{"quality":"...","url":"...","width":120,"height":90}],"description":"...","descriptionHtml":"<div>...</div>","viewCount":1,"published":1554474212,"publishedText":"1 days ago","lengthSeconds":10,"liveNow":false,"paid":false,"premium":false}

:keepalive 1554427333

:keepalive 1554427335

id: 1
data: {"title":"...","videoId":"BBBBBBBBBBB","author":"...","authorId":"UCBBBB","authorUrl":"/channel/UCBBBB","videoThumbnails":[{"quality":"...","url":"...","width":120,"height":90}],"description":"...","descriptionHtml":"<div>...</div>","viewCount":1,"published":1554474212,"publishedText":"1 days ago","lengthSeconds":10,"liveNow":false,"paid":false,"premium":false}

Example handling in JavaScript:

var notifications = new EventSource('https://invidio.us/api/v1/notifications?topics=UCAAAA,UCBBBB');
notifications.onmessage(event) {
    var notification = JSON.parse(event.data); // => {title: "...", videoId: "AAAAAAAAAAA", ...

    // Update feed, notify user, or any other necessary handling for the event
}

Polling for notifications after being offline

In order for an application to receive notifications after being offline , a ?since=TIMESTAMP should be added to /api/v1/channels/latest/:ucid. Using ?since= would return up to 15 videos uploaded after the specified timestamp. If there are more than 15 videos that have been uploaded since the given date, /api/v1/channels/latest/:ucid will return the 15 latest.

It may also make sense to support an ?until=TIMESTAMP to create a "window" where specific videos can be queried, although this would require authentication as mentioned below.

Final thoughts

Notifications would only be supported for channels tracked by the instance, which means that one or more users on that instance are subscribed to them. Requesting that a new channel be tracked by an instance must be authenticated to prevent abuse. Authentication will be handled in a second RFC.

Important to note is that notifications will be sent when a channel changes a video's description or title, not just when it is uploaded.

In accordance with this spec, a "protocols":["eventsource"] field should be added to /api/v1/stats, example. See also #356.

I would like to discuss if there are any issues with this approach, or any other issues that haven't been considered to ensure this is provides as much utility as possible.

Thanks!

[omarroth]

A couple thoughts on the "Polling for notifications after being offline" section after some discussion in the Matrix server:

The ?since=#{TIMESTAMP} param should be added to /api/v1/notifications, not (or in addition to) /api/v1/channels/latest/:ucid. This would provide all published events for the subscribed topics up to a maximum of 15 since the specified TIMESTAMP, then continue serving events as normal. If there are more than 15 videos that have been uploaded since the given date, /api/v1/notifications will return the 15 latest from each channel.

Example output from cURL:

$curl https://invidio.us/api/v1/notifications?topics=UCAAAA,UCBBBB&since=1554597448
:keepalive 1554597449

id: 0
data: {"title":"...","videoId":"AAAAAAAAAAA","author":"...","authorId":"UCAAAA","authorUrl":"/channel/UCAAAA","videoThumbnails":[{"quality":"...","url":"...","width":120,"height":90}],"description":"...","descriptionHtml":"<div>...</div>","viewCount":1,"published":1554474212,"publishedText":"1 days ago","lengthSeconds":10,"liveNow":false,"paid":false,"premium":false}

id: 1
data: {"title":"...","videoId":"BAAAAAAAAAA","author":"...","authorId":"UCAAAA","authorUrl":"/channel/UCAAAA","videoThumbnails":[{"quality":"...","url":"...","width":120,"height":90}],"description":"...","descriptionHtml":"<div>...</div>","viewCount":1,"published":1554474212,"publishedText":"1 days ago","lengthSeconds":10,"liveNow":false,"paid":false,"premium":false}

id: 2
data: {"title":"...","videoId":"CAAAAAAAAAA","author":"...","authorId":"UCAAAA","authorUrl":"/channel/UCAAAA","videoThumbnails":[{"quality":"...","url":"...","width":120,"height":90}],"description":"...","descriptionHtml":"<div>...</div>","viewCount":1,"published":1554474212,"publishedText":"1 days ago","lengthSeconds":10,"liveNow":false,"paid":false,"premium":false}

:keepalive 1554597470

:keepalive 1554597493

id: 3
data: {"title":"...","videoId":"BBBBBBBBBBB","author":"...","authorId":"UCBBBB","authorUrl":"/channel/UCAAAA","videoThumbnails":[{"quality":"...","url":"...","width":120,"height":90}],"description":"...","descriptionHtml":"<div>...</div>","viewCount":1,"published":1554474212,"publishedText":"1 days ago","lengthSeconds":10,"liveNow":false,"paid":false,"premium":false}

Although this would need to run an expensive query against the DB depending on the number of topics, so calls using ?since=TIMESTAMP should be authenticated.

[cloudrac3r]

It's unclear how authentication ties in to all of this.

[omarroth]

In order for an instance to provide notifications for a channel, said channel needs to be tracked. To track a channel means the instance is subscribed to a channel's RSS feed and/or has a job running to pull all videos/update older videos from a channel.

Tracking channels can be resource intensive so currently it is only done when at least one user has previously subscribed to a channel (it will still track after unsubscribing). To ensure that a channel is being tracked, a client should send a POST /subscriptions/:ucid as mentioned here, which can then be followed by a DELETE /subscriptions/:ucid if there isn't a need to generate a corresponding subscription feed.

Authentication is needed so that a client can ensure it will receive notifications for a channel, while not allowing an easy vector for abuse.

[omarroth]

Additionally, supporting a ?since= parameter would need to run a query against all the specified topics, so that would also require authentication, since the query could get quite expensive.

[omarroth]

Added /api/v1/auth/notifications with fb7068d. ?since=TIMESTAMP has not yet been implemented.

Since each message contains the entire /api/v1/videos/:id response, it is recommended to use the fields API (see #429) to request only necessary data for providing notifications. For example &fields=videoId,title,author,authorId,published,isLive.

Important to note is that notifications will also be sent whenever a video is updated for example when a description or title is changed. It is recommended to use logic similar to here to provide consistent notifications for new videos.

[omarroth]

Added ?since=TIMESTAMP with 54d250b.

[omarroth]

Added POST /api/v1/auth/notifications with b3e083d.

Although there isn't technically a limit on the size of a GET request, things can start to fail in weird ways after a certain limit. This provides a way to access the advertised 1000 topics per call by putting the topics field into the body.

If you're using something like sse.js, you can do:

notifications = new SSE(
    '/api/v1/auth/notifications?fields=videoId,title,author,authorId,publishedText,published,authorThumbnails', {
        withCredentials: true,
        payload: 'topics=' + subscriptions.join(','),
        headers: { 'Content-Type': 'application/x-www-form-urlencoded' }
    });

notifications.onmessage = function (event) {
    if (event.data === '') {
        return;
    }

    var notification = JSON.parse(event.data);
    // ...
}