Transmux TS to fMP4

[joeyparrish]

On platforms that support Transport Stream via MediaSource, we are able to play TS content. Currently, that is just Edge and Chromecast. (Safari is not working due to #743.)

This is consistently surprising to developers, and not everyone can transmux their content to fMP4 ahead of time. Developers expect that working content will work everywhere.

To meet this expectation, we should start offering transmuxing of TS content in the browser. We should optimize this, if needed, to reduce impact on CPU and battery life. If the process is too expensive, we should warn developers and provide guidance on transmuxing content en masse on the server, to reduce the burden on clients.

We should examine the transmuxing implementation of hls.js (which is also Apache-licensed) before we begin work. There may be a way to collaborate with hls.js to reuse their transmuxer rather than build a new one.

[forbesjo]

https://github.com/videojs/mux.js is another library you could look at (I may be biased).

[joeyparrish]

@forbesjo, thank you for the suggestion. We will look into it!

[joeyparrish]

We have decided to use https://github.com/videojs/mux.js as the basis of our transmuxing feature.

Here's a rough outline:

Today

1. We parse the manifest and create our internal data structures describing the content
2. We query the platform for the MIME types and codecs we find in the content
3. If the container isn't supported, the platform rejects the MIME type, and we reject the content

With transmuxing

1. We parse the manifest and create our internal data structures describing the content
2. We query the platform for the MIME types and codecs we find in the content
3. If the container isn't supported, we check if mux.js has been loaded
4. If mux.js is available and can demux the container, we install transmuxing filters, relabel the content as MP4, and try again
5. Transmuxing filters will use mux.js to transmux each segment after download

If the application developer is not interested in transmuxing, no changes are required. If the application developer wants TS transmuxing, they merely have to upgrade Shaka Player to v2.3+ and load mux.js in parallel with Shaka.

@forbesjo, how does mux.js manage API changes? If we ask our users to load mux.js, and we have tested with v4.1.5, will there be any breaking API changes in v4.2? Should we advise them only to use specific versions? What's the best way to stay informed of important changes to mux.js?

[forbesjo]

mux.js follows semver so there will only be breaking changes in major versions. I would only advise users to use specific versions if they prefer certainty or reproducible builds over automatic updates.

Right now updates aren't communicated beyond watching the repo but we plan on announcing updates in the #announcements channel in http://slack.videojs.com/ as well as posting change log notes to Github releases.

[joeyparrish]

@forbesjo, thanks!

[forbesjo]

Users can also be directed to the video.js #playback channel for questions/information that doesn't fit in a Github issue.

[tchakabam]

mux.js is probably a really good choice! great to hear that, looking forward to TS-based HLS support in Shaka!

Do you see the need to run the transmuxing inside a worker script? This is how it was done in Hls.js and also in videojs-contrib-hls.

So in my experience transmuxing, parsing the TS, and building/writing the fMp4 is rather effortless task (it's not like decoding anything or so, "just" read/writing container data). We have been doing this for audio streams, but of course with video it will generally be a higher data throughput.

But it's probably better to not do this on the UI thread still :)

I have been building a library for transmuxing, that currently supports output of MP4, but the usecase it was tested with was only MP3 (for HLS with MP3 elementary streams playback on Firefox, that only accepts MP3 when contained as fMP4, Chrome does support Mp3 "elementary" btw). It is a general pipelining framework with parsers, demuxers, muxers, etc.. It's a bit GStreamer inspired :)
It also supports running stuff in workers. The mp4-packager was based on a Mozilla experimental project under Apache license.

See: https://github.com/tchakabam/multimedia.js

There is also another great library for reading & building MP4 data: https://github.com/madebyhiro/codem-isoboxer

and here is a node-based Transport-Stream parser which even seems to be DVB-T compliant: https://github.com/mkrufky/node-dvbtee

[tchakabam]

@joeyparrish If you are still looking for Hls.js know-how, I have been contributing a bit to it :) So you can shoot me your questions if you have any.