Native Audio Reading Experience

[mekarpeles]

Once upon a time, we used to take archive.org books and generate tts using server-side festival API
https://git.archive.org/ia/petabox/-/blob/master/www/datanode/BookReader/BookReaderGetTTS.php

We then added client-side browser-synthesized WebSpeechSynthesis TTS to bookreader

@MarcCoquand has been working on a native mobile audio reading UI experience that works when we already have wavs/mp3 etc files:

This UI is currently prototyped within http://reader.archive.org for librivox books using capacitor.js. With a small shim (using e.g. PocketTTS similar to https://speech-server.readium.org/demo, instead of festival), I imagine this could power a ~native audio book experience for any Internet Archive book.

In an ideal world (at least for a prototype), the audio reader client is able to run PocketTTS client side in webasm via something like https://github.com/KevinAHM/pocket-tts-onnx-export.

Prototype Approach:
To create a proof of concept, I'd:

• Pull down the bookreader repo and create a new audioreader branch
• Look at how the existing TTS plugin logic works (the majority of the infrastructure is already done an working)
• Scale it back to a minimal working API (ideally that doesn't fetch any book page images at all) / scaffolding that speaks to Archive.org to fetch data
• Implement the minimal bookreader UI that Marc has implemented
• Wire up the demo to use pocket-tts-onnx-export
• Try to always have up to 5 paragraphs in the buffer / cached (load in the background).
• If/while the current paragraph is not loaded yet, have a loading spinner.
• On first load or when patron clicks next/previous to jump around, they will not want to wait for load times. In this case (either initial load or when next/previous pressed), the UI should not initially render an entire paragraph at once. Instead, it should render... the first 3 words as one chunk (a "landmark" or comprehensible unit that helps the patron understand if they are at the right section), and then the rest of the resentence as another chunk, and then the rest of the paragraph, and then the following 4 paragraphs until the buffer is hydrated.
• We want some sort of clever throttling or debouncing load strategy (whichever will be a better loading experience) so that as a patron seeks around w/ prev/next, we are not queuing up more loads every time the button is pressed. Either... (a) when the button is pressed [again], we immediately clear any in-flight rendering and re-initiate for that block and/or (b) one clever scheme could be to use a hybrid of PocketTTS and WebSpeechSynthesis (much faster) to provide an instant fallback preview while the higher quality voices are loading
• On the initial page load (before the patron presses play) the priority should be on preloading/buffering (1) the first 3 words, (2) the rest of the sentence, (3) the rest of the first paragraph, and then when that finishes, queuing up the next 3 paragraphs (as a callback, so they get loaded in order and don't overload the client cpu)

Note: the bookreader also uses webasm for in-book translations mode, so this may be another reference point.

UI:
When one loads the bookreader via the audioreader branch, one should see the minimal view (not the typical bookreader view) with the cover, the table of contents if it exists, and the controls. Pocket-tts should load on the client. There's a world in which there's an option to to show e.g. that sows up to 5 words that are within the reading window and shows the word being read stationary in the center / highlighted. This is a future accessibility version or option that is not enabled by default.

[fasuizu-br]

Affiliation: I am with BRAINIALL, which operates a buffered text-to-speech route returning WAV bytes. I will not open a PR, synthesize book text, or send provider credentials. Since #1580 compares the legacy server-TTS path with a client-side PocketTTS prototype and already buffers WAV or MP3 assets, would one maintainer-selected public-domain <=300-character paragraph receipt help establish only the server-generated buffered-WAV baseline before choosing a browser-only path? If you opt in, I can post the text hash and character count, response MIME and byte count, observed RIFF/WAVE fields, parsed duration, elapsed time, and observed charge. This would not claim streaming, client-side or WASM execution, word timing or highlighting, accessibility, voice quality, caching, BookReader compatibility, or rights for any book. If useful, what exact browser, byte-size and latency gate should the baseline use?