feat: Implement spectrogram visualization for AudioPlus

[cloudmark]

Spectrogram visualization to Audio Component

Reason for change

This PR adds spectrogram visualization support to the audio editor, enabling users to visualize frequency content over time in audio recordings. This feature enhances audio annotation capabilities by providing visual frequency analysis tools, particularly useful for tasks like speech analysis, music transcription, and sound event detection.

The implementation includes:

• FFT-based spectrogram rendering with configurable parameters
• Multiple color scheme options with live preview
• Mel-scale frequency mapping support
• Real-time parameter adjustment
• Performance optimizations for smooth rendering
• Modular code organization for maintainability

Screenshots

1. Configuration Interface

Shows the labeling interface configuration with the new spectrogram="true" property in the XML configuration, demonstrating how the feature can be enabled through the labeling interface.

2. Color Scheme Selection

Demonstrates the color scheme selection interface with:

• Multiple predefined schemes (Autumn, Bathymetry, Blackbody, etc.)
• Visual preview for each scheme
• Real-time application of color changes
• Smooth transition between schemes

3. Time Display on Hover

Shows interactive features:

• Precise time indicator (00:00:00.447) on hover
• Clear visualization of frequency content
• Smooth rendering of the spectrogram
• Integration with the waveform display

4. Playback and Spectrogram Settings

Comprehensive control panel featuring:

• Playback speed adjustment
• Audio zoom y-axis control
• FFT Samples slider (64-2048)
• Loop Regions toggle
• Auto-play New Regions toggle
• Integrated spectrogram controls

5. Advanced Spectrogram Controls

Detailed configuration options:

• FFT Samples selection (512)
• Mel Bands adjustment (64)
• Spectrogram dB range (-80 to -10)
• Windowing Function selection (Blackman)
• Color Scheme selection (Viridis)
• View toggles for timeline, audio wave, and spectrogram

Rollout strategy

The feature is implemented with a progressive enhancement approach:

1. Feature Flag:

<AudioPlus name="audio"
          value="$audio"
          hotkey="space"
          sync="group_a"
          defaultscale="1"
          defaultzoom="2"
          zoom="true"
          spectrogram="true"
/>

2. Backward Compatibility:

• Existing audio components continue to work without changes
• Spectrogram can be enabled/disabled per instance
• All new parameters have sensible defaults

3. Performance Considerations:

• Lazy loading of FFT computation code
• Progressive rendering for large files
• Configurable quality settings

Testing

Comprehensive testing strategy:

1. Unit Tests:

• WindowFunctions.ts: Window function calculations
• ColorMapper.ts: Color scheme management
• FFT computation accuracy
• Parameter validation

2. Integration Tests:

• Audio loading and visualization
• Real-time parameter updates
• Color scheme switching
• Performance benchmarks

3. Manual Testing Scenarios:

• Various audio formats (WAV, MP3, OGG)
• Different file lengths (short clips to long recordings)
• Multiple sample rates and bit depths
• Browser compatibility (Chrome, Firefox, Safari)
• Performance with large files

Risks

1. Performance:

• FFT computation is CPU-intensive
• Mitigated through:
  • Chunked rendering
  • Yield scheduling
  • Canvas optimization
  • Caching mechanisms

2. Memory Usage:

• Large audio files require more memory for FFT
• Mitigated through:
  • Buffer management
  • Cleanup of unused resources
  • Progressive loading

Reviewer notes

Key areas to review:

1. Visualizer.ts: Spectrogram rendering logic
2. WindowFunctions.ts: Audio processing utilities
3. ColorMapper.ts: Color scheme management
4. Performance optimizations in rendering loops
5. Error handling and edge cases

General notes

The spectrogram visualization feature provides:

• Real-time frequency analysis
• Multiple color schemes for different use cases
• Configurable parameters for detailed analysis
• Smooth integration with existing audio tools
• Optimized performance for large files

[netlify]

👷 Deploy request for heartex-docs pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	ae15d4f

[netlify]

👷 Deploy request for label-studio-docs-new-theme pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	ae15d4f

[netlify]

✅ Deploy Preview for label-studio-storybook ready!

Name	Link
🔨 Latest commit	ae15d4f
🔍 Latest deploy log	https://app.netlify.com/sites/label-studio-storybook/deploys/680bee5ee59421000859e50f
😎 Deploy Preview	https://deploy-preview-7400--label-studio-storybook.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[cloudmark]

To help visualize the new spectrogram functionality implemented in this PR (#7400), I've recorded a short video demonstration:

Video Demonstration: Spectrogram Feature

What the video shows:

The video walks through the spectrogram feature within the Label Studio interface, highlighting:

• The spectrogram display integrated below the audio waveform.
• Synchronized playback tracking across both the waveform and spectrogram.
• How zooming affects both views simultaneously.
• Real-time updates as various configuration options are adjusted in the settings panel:
  • Changing FFT window sizes.
  • Applying different color schemes.
  • Selecting various windowing functions (Blackman, Hann, Hamming).
  • Toggling the Mel frequency scale.
  • Adjusting the amplitude (dB) range.

Hope this provides a helpful overview of the user experience!

[makseq]

Great PR! How well will it work with long audio files around 1-2 hours?

[cloudmark]

Hey @makseq,

TL; DR: Yes! it handles long files (1-2 hours) efficiently.

The core strategies implemented are:

1. On-Demand Processing: It only analyzes the audio needed for the currently visible portion.
2. Efficient Sampling (Zoomed Out): When zoomed out (e.g., 1 hour on a 1024px view), each pixel represents a time slice (~3.5s). For each pixel, I take one representative FFT (e.g., 512 samples covering ~11.6ms) within that slice, avoiding processing every single audio sample.
3. Non-Blocking Rendering: A generator function (renderSpectrogramSlice) renders pixel by pixel, yielding frequently (~16ms) to keep the UI responsive during interactions like scrolling/zooming.

This approach balances performance, memory, and visual overview. As you zoom in, the detail naturally increases as fewer samples are represented per pixel.

Separately, the chosen FFT window size affects the computation time per slice (larger FFTs = more detail but slower slice render). This characteristic is independent of total file length. For the most fluid feel, 512 is often a good balance.

To demonstrate this with varied audio content, the video uses a 1-hour file created by concatenating samples from the ESC-50 dataset (https://github.com/karolpiczak/ESC-50). This dataset contains 2000 short environmental sound recordings across 50 categories (like dogs barking, rain, helicopters, etc.), ensuring the test file has diverse spectral characteristics.

Video Demo: Spectrogram Performance & FFT Size Impact (1hr ESC-50 file)

(Video shows loading/panning the long, varied file & the visible speed difference when switching FFT sizes).

[farioas]

@cloudmark please rebase your branch on the latest changes from repo to include this commit 9b0487f. It will fix failing checks.

[cloudmark]

Done @farioas, I also updated #7376 to take from this upstream

[makseq]

<AudioPlus> was deprecated, could you please use <Audio> instead?

[cloudmark]

Thank you @makseq for the heads up. I think internally they should resolve to the same component so there are no further updates needed (I believe).