Skip to content

feat: preserve youtube transcript segments - #401

Open
lorenzozane wants to merge 1 commit into
kepano:mainfrom
lorenzozane:feat-youtube-preserve-segments
Open

lorenzozane wants to merge 1 commit into
kepano:mainfrom
lorenzozane:feat-youtube-preserve-segments

Conversation

@lorenzozane

Copy link
Copy Markdown
Contributor

Possible implementation for #176

YoutubeExtractor currently groups transcript segments into readable blocks. This is useful for reading, but some projects could benefit from keeping the individual segments to have more granular timestamps.

Added extractors.youtube.preserveTranscriptSegments, defaulting to false, to make this optional.

Example

The segments:

0:00 The quick brown
0:02 fox jumps over the lazy dog.
0:04 Then it ran
0:06 away quickly.

are currently grouped as:

0:00 The quick brown fox jumps over the lazy dog.
0:04 Then it ran away quickly.

With preserveTranscriptSegments: true, each segment keeps its own timestamp:

const result = await new Defuddle(document, {
  extractors: {
    youtube: {
      preserveTranscriptSegments: true
    }
  }
}).parseAsync();

Implementation

Added getTranscriptSegments() to either preserve the parsed segments or pass them through the existing grouping logic. Both fetched XML captions and transcripts extracted from the desktop/mobile DOM use this method.

Text decoding, normalization and HTML escaping still apply. When preservation is enabled, speaker markers remain literal text and speaker turns are not inferred.

The option is also forwarded through synchronous extraction, asynchronous extraction and fetchAsyncVariables(). Default behavior remains unchanged.

Tests

Added coverage in youtube-transcript.test.ts for option forwarding, both XML formats, desktop/mobile transcripts, normalization, chapters and fractional start times. The existing HTML escaping test now runs with preservation both enabled and disabled.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant