Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Media overlays in the streamer #110

Open
qnga opened this issue Jan 23, 2020 · 6 comments
Open

Media overlays in the streamer #110

qnga opened this issue Jan 23, 2020 · 6 comments

Comments

@qnga
Copy link
Contributor

qnga commented Jan 23, 2020

How media overlays should be implemented in the streamer?
The question has first been raised here.
There media overlays are introduced as a service provided over HTTP. It's not clear for me whether they should be able to be accessed both in-memory and over HTTP depending on the format. In any case, the fetcher seems to me to be the right place to convert SMIL media overlays to in-memory Synchronized Narration.

@HadrienGardeur
Copy link
Contributor

It mostly depends on the plaform.

On the Web, "media overlays" would only be served using HTTPS:

  • a Web App would fetch a RWPM over HTTPS
  • through this manifest, a number of links to Synchronized Narration documents are discovered
  • the navigator used by this Web App would then fetch over HTTPS the Synchronized Narration document, along with the HTML and audio resources referenced by that document

On mobile, apps that use strictly native code would only rely on HTTP to fetch the HTML and audio resources, everything else would be handled in-memory.
But apps that rely on a navigator built entirely in JS may behave exactly like Web Apps and fetch the manifest and Synchronized Narration document using HTTP as well.

@qnga
Copy link
Contributor Author

qnga commented Jan 25, 2020

On mobile, apps that use strictly native code would only rely on HTTP to fetch the HTML and audio resources, everything else would be handled in-memory.

Everything else doesn't include images and CSS, does it? Why this choice? I understand the idea of passing the manifest in memory, but I didn't think that concerned resources too. It seems to me to be conceptually twisted as it relies on two separate mechanisms for resource handling, and moreover prevents remote resource access for some resource types.

Now, assuming this choice in the Kotlin app, SMIL files may be parsed and their content be added as a kotlin property into the link objects of overlaid Html files (and not into dedicated links). Is that your vision?

I though of still another way: converting SMIL into sync narr format and inject them right into concerned Html files in the fetcher. This is particularly interesting if you plan to use audio element to play sound in synchronized narration. A Js lib might be used to play sound and highlight active text, though controls could probably still be native. I really don't known how you are planning to implement all of this.

@HadrienGardeur
Copy link
Contributor

Everything else doesn't include images and CSS, does it? Why this choice?

They're served over HTTPS as well. In my explanation, I only mentioned the resources that are fetched directly, not the ones that are fetched indirectly as HTML gets rendered (images, CSS, JS and fonts).

Now, assuming this choice in the Kotlin app, SMIL files may be parsed and their content be added as a kotlin property into the link objects of overlaid Html files (and not into dedicated links). Is that your vision?

I though of still another way: converting SMIL into sync narr format and inject them right into concerned Html files in the fetcher. This is particularly interesting if you plan to use audio element to play sound in synchronized narration. A Js lib might be used to play sound and highlight active text, though controls could probably still be native. I really don't known how you are planning to implement all of this.

We'll have to do both and the in-memory model will be based on Synchronized Narration anyway.

@qnga
Copy link
Contributor Author

qnga commented Feb 3, 2020

So, as far as I understand, in mobile apps:

  • SMIL resources are mapped to Links containing the Synchronized Narration data
  • <item media-overlay="idref"> is mapped to a Link whose alternates property contains the link corresponding to the idref as described above
  • when the navigator or the testapp (I don't know) encounters a link with type "application/vnd.syncnarr+json", it doesn't fetch the resource over HTTP, but directly use Publication.linkWithHref method to retrieve the link containing the synchronization data.

Questions:

  • What should be the href of the Synchronized Narration link computed from the SMIL resource? The original SMIL path contains the ".smil" extension, but a different href raises an issue with href unicity.
  • How should synchronization data be stored in Link objects? In a synNarr member? In otherProperties?

@qnga
Copy link
Contributor Author

qnga commented Feb 5, 2020

I think I was again mistaken.
NOT

  • when the navigator or the testapp (I don't know) encounters a link with type "application/vnd.syncnarr+json", it doesn't fetch the resource over HTTP, but directly use Publication.linkWithHref method to retrieve the link containing the synchronization data.

BUT

  • when the navigator or the testapp (I don't know) encounters a Link with type "application/vnd.syncnarr+json", it doesn't fetch the resource over HTTP, but directly use the synchronization data contained in the Link.

Questions remain to be answered.

@qnga
Copy link
Contributor Author

qnga commented Feb 5, 2020

My last interpretation has the drawback of getting the manifest dirty with Synchronization Narration data right inside. I suggest acting as if Media Overlays would be served over HTTP in the decidated resource with SyncNarr being the native format, more like my first interpreration.

Concretely:

  • a Publication object should have a syncNarr collection, as resources and readingOrder. This collection would not to be relevant for the manifest, but just a trick in mobile apps.
  • The Epub parser would parse all SMIL files and put synchronisation data into syncNarr collection by means of dedicated Links (or another class) that are not directly contained in alternates but instead referenced with an href, as other resources.
  • When the client encounters a link with type application/vnd.syncnarr+json (most likely in alternates, it just looks into syncNarr collection for the href.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants