Dynamically manipulating <source> src attribute *should* yield an effect

[benwiley4000]

tl;dr

Due to confusion about the nature of my request I want to clarify I am not suggesting I should be able to recycle the source list concept in HTML as a sequential playlist. I am merely suggesting I should be able to go from this:

<audio>
  <source src="song1.ogg">
  <source src="song1.mp3">
</audio>

to (by mutating the src attributes):

<audio>
  <source src="song2.ogg">
  <source src="song3.mp3">
</audio>

And the currentSrc should update to use one of the new src attributes on the <source> children.

jsfiddle showing current behavior

original posting

I was reading the <source> element spec and came across the following while trying to troubleshoot a bug in my library:

Dynamically modifying a source element and its attribute when the element is already inserted in a video or audio element will have no effect. To change what is playing, just use the src attribute on the media element directly, possibly making use of the canPlayType() method to pick from amongst available resources. Generally, manipulating source elements manually after the document has been parsed is an unnecessarily complicated approach.

Respectfully - I disagree! My component allows a user to specify a sequential playlist of songs to cycle through, and a list of potential sources for each song in the playlist, to be used by the browser as the browser sees fit.

The above quotation implies I am only able to benefit from the browser's automatic source selection for the first song I load. I have to take a totally imperative approach for subsequent tracks, which is a shame in a context like React where I would like to be able to declaratively specify my source list as a function of the currently chosen song index.

What reads simpler to you:

render() {
  const index = this.state.activeIndex;
  const sources = this.props.sourcesByIndex[index];
  return (
    <audio ref={ref => this.audio = ref}>
      {sources.map(source =>
        <source key={source.src} src={source.src} type={source.type} />
      )}
    </audio>
  );
}

or:

componentDidUpdate(prevProps, prevState) {
  const newIndex = this.state.activeIndex;
  if (newIndex !== prevState.activeIndex) {
    for (const source of this.props.sourcesByIndex[newIndex]) {
      const mimeType =
        source.type ||
        useSomeDatabaseLookupLibraryToFindMimeType(source.src);
      if (this.audio.canPlayType(mimeType) {
        this.audio.src = source.src;
        break;
      }
    }
  }
}

render() {
  // same render implementation as above
}

Manually iterating through my source options, checking canPlayType() for each, and updating the src attribute seems to me an unnecessarily complicated approach! 😄

Sidenote

While reading through both the spec and MDN I came across a notably discrepancy.

Spec

(Emphasis mine)

If the type attribute is not specified, the user agent will not select a different source element if it finds that it does not support the image format after fetching it.

The quotation refers specifically to images, but I'm assuming it applies to audio and video as well.

MDN

(Emphasis mine again)

If the type attribute isn't specified, the media's type is retrieved from the server and checked to see if the user agent can handle it; if it can't be rendered, the next <source> is checked.

Which is it - implementation-wise? If the MDN version turns out to be what's really in the wild, then we're missing a critical feature - automatic mimetype retrieval - when forced to restort to the method of source updating suggested in the spec.

ETA

In my original code example I mapped this.props.sources in the render method - I meant to just map sources. This is updated.

[guest271314]

You can

1. Request all of the media files first, then play the media in sequential order when the current media ended event is dispatched;

2. Request all of the media first, then utilize AudioContext to concatenate N AudioBuffers into a single AudioBuffer, then play that single file.

[benwiley4000]

@guest271314 Sorry, I don't understand how your suggestions are related to the problem I described. Could you provide some code examples? Thanks! 🙂

[guest271314]

The specification section referenced at #3588 (comment) appears to be clear as to dynamically modifying a <source> element.

The use case

specify a sequential playlist of songs to cycle through, and a list of potential sources for each song in the playlist, to be used by the browser as the browser sees fit

can be achieved using various approaches; you can <audio> elements or new Audio() by beginning the next media rendering (output to speakers) at the ended event of the current media rendering, for example, by iterating an Array of media sources, creating the element, attaching the event handler.

The above quotation implies I am only able to benefit from the browser's automatic source selection for the first song I load.

It should only be necessary to check if the media element can play certain types of media once, before including multiple <source> elements within the HTML.

The media types that are not capable of being played at the media element do not need to be included in HTML at all at <source> elements.

[benwiley4000]

you can elements or new Audio() by beginning the next media rendering (output to speakers) at the ended event of the current media rendering

This makes sense; however it's undesirable. In building a React component I want my <audio> element to be accessible in the mounted DOM, and the only way to do this is to defer to React to render the element (unless I want to apply a quite dirty and error-prone hack). If I want React to create a new <audio> element when the song has ended I need to use some non-conventional means to make my entire DOM subtree regenerate, which would be non-performant anyway.

It should only be necessary to check if the media element can play certain types of media once, before including multiple elements within the HTML.

This is a very good point. However, it then doesn't seem like I should be using <source> children at all. Instead I should just be reinventing that functionality by pre-calculating a single src attribute to pass directly to my <audio> element.

I understand the capacity to do what I want is there (I already showed a way in my code in the original post). However I'm suggesting that the given <source> children API is not very useful at all for a context where we want to cycle through multiple songs.

[guest271314]

However I'm suggesting that the given <source> children API is not very useful at all for a context where we want to cycle through multiple songs.

How did you draw the conclusion that the HTML <source> element is intended to be used to cycle through multiple songs?

Have no experience using React. React is not related to HTML <source> element, the HTML specification, or the use case of playing multiple songs in any particular order.

Again, there are various approaches which could be used to play media in a specific order using either one or more <audio> elements, or Web Audio API. If the requirement is to use only a single <audio> node, ended event can still be utilized to render audio in a particular order at that single element.

[benwiley4000]

How did you draw the conclusion that the HTML `<source>` element is

intended to be used to cycle through multiple songs? HTML supports dynamically updating src attributes on single audio elements to handle multiple songs, and thus a parallel expectation that the same would hold for source elements is reasonable. Le ven. 23 mars 2018 14 h 39, guest271314 <notifications@github.com> a écrit :

…

However I'm suggesting that the given <source> children API is not very useful at all for a context where we want to cycle through multiple songs. How did you draw the conclusion that the HTML <source> element is intended to be used to cycle through multiple songs? Have no experience using React. React is not related to HTML <source> element, the HTML specification, or the use case of playing multiple songs in any particular order. Again, there are various approaches which could be used to play media in a specific order using either one or more <audio> elements, or Web Audio API. If the requirement is to use only a single <audio> node, ended event can still be utilized to render audio in a particular order at that single element. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3588 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AM7h7dAa_TinXp-xU74coGAc8vz5pIi-ks5thUFOgaJpZM4S4Laa> .

[guest271314]

If the requirement is to use multiple <source> elements as children of a single <audio> or <video> node to implement a "playlist", you can

1. Set type attribute to the MIME type of the media resource;
2. At loadedmetadata and error event handlers of the parent HTMLMediaElement .canPlayType() returns false, remove the <source> element from the parent HTMLMediaElement;
3. At ended (or pause) event handler of the parent HTMLMediaElement if <source> elements children .length is greater than 1, insert the first <source> "beforeend" of the parent HTMLMediaElement, call .load() method of HTMLMediaElement

which should "cycle" the playback of the media resource set at src attribute of <source> element.

<style type="text/css">
/* provide notification for currently playing media */
figure figcaption:before {
  content: "Playing " attr(data-playing);
}
</style>
<figure>
  <audio controls="true"
    preload="auto" 
    onloadedmetadata="this.querySelectorAll('source').forEach(source => {if (!source.parentElement.canPlayType(source.type)){source.parentElement.removeChild(source)}}); this.closest('figure').querySelector('figcaption').dataset.playing = new window.URL(this.currentSrc).pathname.split('/').pop();"
    onerror="this.querySelectorAll('source').forEach(source => {if (!source.parentElement.canPlayType(source.type)){source.parentElement.removeChild(source)}}); this.closest('figure').querySelector('figcaption').dataset.playing = new window.URL(this.currentSrc).pathname.split('/').pop();"
    onended="if (this.querySelectorAll('source').length > 1) {this.insertAdjacentElement('beforeend', this.querySelector('source')); this.load();};"
    onplaying="console.log('Playing ' + this.currentSrc); this.closest('figure').querySelector('figcaption').dataset.playing = new window.URL(this.currentSrc).pathname.split('/').pop();">
      <source src="/path/to/ogv.ogv" type="video/ogg"/>
      <source src="/path/to/webm.webm" type="video/webm"/>
      <source src="/path/to/mp4.mp4" type="video/mp4"/>
  </audio>
  <!-- provide notification for currently playing media -->
  <figcaption data-playing=""></figcaption>
</figure>

[guest271314]

new window.URL() is not necessary; .split() with .pop() can be chained to .currentSrc. .load() is necessary.

[benwiley4000]

I'm not sure I follow your 3 steps, much less the code example...

Allow me to re-approach my initial query for a moment. I understand the spec is clear about dynamically updating <source> elements. I am suggesting this stance should be revisited by whatwg, as the workarounds mentioned have been quite convoluted, and it makes intuitive sense for src attribute updates to <source> children to be treated the same as src attribute updates to media elements. Particularly, this lends itself much better to declarative paradigms (e.g., but not exclusively React) which are becoming increasingly popular for authoring applications on the web.

Separately, could any of the browser engine implementers speak to my question about the sharp discrepancy between the spec and MDN on fallback <source> selection without specified mimetype?

[guest271314]

The 3 steps explained as an example reflect the requirement of using <source> elements at a single HTMLMediaElement as discrete media resources which can be used to "cycle" media playback as a "playlist". The code is intended to be only an example within the bounds of the original issue. Have you tried the code https://jsfiddle.net/qko0uwm4/?

1. Your original code checks .canPlayType().The MIME type should be set at <source> element type attribute uniformly.

2. Check .canPlayType() at both loadedmetadata and error events. If .canPlayType() returns false, remove the element from HTMLMediaElement.

3. At ended event, insert the <source> element at index 0 of <source> elements as last element of parent HTMLMediaElement (though we probably should also check if any <track> elements are siblings); call .load() method on parent HTMLMediaElement (see ".load() is necessary."; below)

9. Invoke the media element's resource selection algorithm.

6. Otherwise, if the media element does not have an assigned media provider object and does not have a src attribute, but does have a source element child, then let mode be children and let candidate be the first such source element child in tree order.

Not certain if you are expecting the <source> element to automatically render as a "cycle" "playlist"? If that is the case, perhaps the authors of the specification could chime in; as do not gather that as the intended purpose of the element, though could very well be incorrect.

[benwiley4000]

@guest271314 sorry, now I think I understand what you are demonstrating.

I was not suggesting I think a list of concurrent source children to a media element should operate as a playlist (this is not how HTMLMediaElement works! 😄 ).

I am suggesting I think I should be able to go from:

<audio>
  <source src="song1.ogg">
  <source src="song1.mp3">
</audio>

to (by mutating the src attributes):

<audio>
  <source src="song2.ogg">
  <source src="song3.mp3">
</audio>

And the currentSrc should update to use one of the new src attributes on the <source> children.

[guest271314]

That currently does occur.

[benwiley4000]

This did not appear to be the case based on my own testing, and as I pointed out in my intro post, the spec specifies this will not happen. I'm suggesting that spec should be revisited.

[guest271314]

Have you read the code and checked the console at linked jsfiddle? It is not entirely clear what you are suggesting should be revisited?

[benwiley4000]

Please see this code to understand what I mean. https://jsfiddle.net/w0wye78a/ Updates to the <source> src attribute bear no effect. P.S. I have updated the top of the original posting to reduce ambiguity. I understand why my request was confusing.

[benwiley4000]

I tried finding when/where it was decided that changes to src attribute on <source> should have no effect. The furthest I can trace the change in git is to this commit by @Hixie.

I can understand potential motivations for this - the most obvious being the fact changing a bunch of <source> elements to have src attributes matching a new source set would create awkward intermediate states before currentSrc can be reliably updated, and the method of figuring out when exactly to make that switch could be opaque to a user.

I'd accept this as a reasonable explanation, but at the least, some more specific explanation in the spec could be nice.

On the other hand, I'm still eager to know about the reason for the discrepancy on whether failed <source> elements with no type can defer to fallback siblings.

[benwiley4000]

Update - I've determined that specifying src on all the sources as well as the media element itself provides a suitably declarative-friendly way of accomplishing the behavior I want. Example

Here's a function I'm using for determining the best source to use on subsequent updates:

let testAudio;
function getProbableSource(sources) {
  testAudio = testAudio || new Audio();
  let probableSource;
  for (const confidence of ['probably', 'maybe']) {
    probableSource = probableSource || sources.find(source => {
	  return testAudio.canPlayType(source.type) === confidence;
	});
  }
  return probableSource ? probableSource.src : sources[0].src;
}

[guest271314]

Again, the code does not call .load() https://jsfiddle.net/w0wye78a/1/

[guest271314]

Which should resolve

Updates to the <source> src attribute bear no effect.

[benwiley4000]

@guest271314 I see! I didn't understand that about your comment earlier (the code example was somewhat dense). It would be useful to include a comment about that in the spec itself under the <source> element description (although I see something like this exists already on MDN).

In my case it's somewhat challenging to implement this well, since load() doesn't just sweep for <source> changes, it also resets everything about the element (so I need to not accidentally call load() anywhere I'm not supposed to). However I'll consider this approach for the future. Thanks!

ETA: Seems it's sufficient to just check if the source inputs have changed and call .load() if so... everything is working for me! thanks again.

[annevk]

Closing per the last comment. Let me know if I drew the wrong conclusion. Thanks!

[benwiley4000]

@annevk I think you're correct, although there were a couple unresolved items:

• in the part of the spec that says directly modifying <source> has no effect, there could be a note that actually, calling <audio>.load() afterward will suffice (it could be explained that this happens automatically when <audio>.src is updated because it's an atomic operation, but the user must manually invoke the load algorithm for <source> child updates).
• The sidenote mentioned in the original post about the discrepancy between MDN and the spec in how audio source resolution works for sources with no type attribute - I was wondering how this has actually been implemented, and if one of those resources should be updated.

[chrisdavidmills]

The sidenote mentioned in the original post about the discrepancy between MDN and the spec in how audio source resolution works for sources with no type attribute - I was wondering how this has actually been implemented, and if one of those resources should be updated.

I'd be interested in this too — does anyone have an answer here? I'd be happy to help update the MDN doc if required (or someone else can just edit it if they feel like it)