IPython.display.Audio not working for WAV audio

[mcmar]

Bug: Notebook Editor, Interactive Window, Editor cells

Issue has previously been discussed, but I have new information that is relevant. I'm forced to create a new issue because all previous issues have been closed and locked.
Please do not close it without considering this new information and responding to it.
I'm only looking for support for uncompressed wav audio format. No need for mp3, mpeg, ogg, or anything else.
New information:

1. Previous discussion with @rchiodo was limited to supporting ffmpeg codec. Support for wav audio files is not dependent in any way on ffmpeg codec. Wav audio can be played back without any codec. No dependency on ffmpeg exists for playing wav audio, which is the most common type used for machine learning because it can be sampled and reconstructed easily (ie wavenet/wavernn).
2. Microsofter and VSCode creater, @bpasero, claims that sound may be played back without DOM support via node.js Add ability to play sound vscode#421 This would eliminate previous reasons for closing this issue in the past.
3. @mjbvz says that ffmpeg codec is unsupported by the core VSCode, but:
   a) does not state that ffmpeg cant be supported by an extension like vscode-python
   b) does not state that any codec is required in order to play wav audio files
   Audio is not supported in webview vscode#66050
4. @mjbvz again states here that only certain types of media are unsupported and mentions a dependency on ffmpeg VSCode >= 1.13 greys out audio controls in previewHTML? vscode#32540
   He does not state that all audio playback is unsupported.

Steps to cause the bug to occur

Type the following code in VSCode, and run the cell (e.g. using SHIFT+ENTER):

import numpy as np
import IPython
T = 2.0    # seconds
sr = 22050 # sample rate
t = np.linspace(0, T, int(T*sr), endpoint=False) # time variable
x = 0.5*np.sin(2*np.pi*440*t)                # pure sine wave at 440 Hz
IPython.display.Audio(x, rate=sr)

You will get a disabled player. The same code in the browser will render a usable player.

Actual behavior

IPython.display.Audio renders a disabled player:

Expected behavior

A valid, playable audio control should be rendered, same as in the browser:

Your Jupyter and/or Python environment

Please provide as much info as you readily know

• Jupyter server running: Local
• Extension version: 2020.5.80290
• VS Code version: 1.45.1
• Setting python.jediEnabled: true
• Setting python.languageServer: Microsoft
• Python and/or Anaconda version: 3.7.6
• OS: Linux (Ubuntu 18.04 LTS)
• Virtual environment: conda

Developer Tools Console Output

Far too big to fit here. No relevant information. Just deprecation warnings and info output.

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer, @joyceerhl

[rchiodo]

Thanks for the feedback. The control that the python kernel creates doesn't work without the appropriate codecs installed. We don't have plans at this time to rewrite the control ourselves.

[Astlaan]

I too, support the inclusion of audio playing capabilities in the python interactive mode. Not having it makes it difficult for people working with audio processing, inc for machine learning.

[mcmar]

@rchiodo I don't understand what the roadblock is regarding the codec. WAV audio is just PCM. You can output it directly to speakers without any codec. You just write the buffer from the file to the audio interface. That's exactly what python-sounddevice and naudiodon do, sans codec. You can even use those libraries to do it.

Is there any way to add support for the ffmpeg extension in vscode-python, that way we're not dependent on the upstream repo? It looks like you asked here, but you never got an answer. microsoft/vscode#66050

Also, VS Code team states that they "strip out ffmpeg and other media support from our release builds". In that case, if I compile from source will IPython.display.Audio suddenly work? I'd be glad to host re-compiled VS Code binaries if that's the roadblock. microsoft/vscode#32540 (comment)

[rchiodo]

Not sure if hosting your own VS code would work. It might.

We closed this bug because there's not a lot of up votes on audio support and it would take us rewiring the ipython.display.audio javascript/html into some other control that plays the audio.

[rchiodo]

Additionally I should have specified that if you want to try doing the conversion to another control yourself, we willingly accept submissions.

Likely in the celloutput.tsx file you'd have to detect the audio control somehow (mime type is likely just text/html) and load/translate the control to something that would actually playback a WAV file.

[Honghe]

It would be nice if it can play audio.

[portogasDace]

I too have this bug can't have a work around?

[rchiodo]

Sorry the work around would be to write your own transform for text/html to create your own audio control that uses the WAV data without FFMpeg

[kirianguiller]

Hi everyone, the WAV reader still seems to be not working for vscode jupyter notebook. Is there any plan to fix it in a near future ?

[rchiodo]

It's unlikely this will be fixed. It's by design:
microsoft/vscode#32540 (comment)

We'd have to write something to hack it to get it to work and I don't see that happening unless we get a LOT of upvotes on getting audio for WAV to work.

[clouds56]

Maybe we should first leave this open as it doesn't get fixed.

[matangover]

Hi all, I've found a workaround using the Web Audio API. Save the following into a file called vscode_audio.py:

import IPython.display
import numpy as np
import json

def Audio(audio: np.ndarray, sr: int):
    """
    Use instead of IPython.display.Audio as a workaround for VS Code.
    `audio` is an array with shape (channels, samples) or just (samples,) for mono.
    """

    if np.ndim(audio) == 1:
        channels = [audio.tolist()]
    else:
        channels = audio.tolist()

    return IPython.display.HTML("""
        <script>
            if (!window.audioContext) {
                window.audioContext = new AudioContext();
                window.playAudio = function(audioChannels, sr) {
                    const buffer = audioContext.createBuffer(audioChannels.length, audioChannels[0].length, sr);
                    for (let [channel, data] of audioChannels.entries()) {
                        buffer.copyToChannel(Float32Array.from(data), channel);
                    }
            
                    const source = audioContext.createBufferSource();
                    source.buffer = buffer;
                    source.connect(audioContext.destination);
                    source.start();
                }
            }
        </script>
        <button onclick="playAudio(%s, %s)">Play</button>
    """ % (json.dumps(channels), sr))

In your notebook, use:

from vscode_audio import Audio
Audio(audio, sr)

where audio in a numpy array with shape (channels, samples) or just (samples,) for mono.

This is quite hacky and slows the whole notebook for longer signals, can be improved.
Anyway I suggest to reopen this issue.