Model loading 2

[panv-kw]

No description provided.

[github-actions]

Single File Benchmark Results

Test File: ES2004a (1049.4s audio)
Overall Result: ✅

Accuracy Metrics

Metric	Value
DER (Diarization Error Rate)	18.7%
JER (Jaccard Error Rate)	22.6%
RTF (Real-Time Factor)	0.04x
Speakers Detected	4/4

⏱️ Performance Timing

Stage	Time (s)	% of Total
Model Download	0.947	2.0%
Model Compilation	0.744	1.6%
Audio Loading	0.079	0.2%
Segmentation	11.172	24.0%
Embedding Extraction	33.539	72.1%
Speaker Clustering	0.024	0.1%
Total Processing	46.504	100%

Inference Time: 44.734s (96.2% of total)
Setup Overhead: 1.691s (3.6% of total)

Research Comparison:

• Powerset BCE (2023): 18.5% DER
• EEND (2019): 25.3% DER
• x-vector clustering: 28.7% DER

[github-actions]

VAD Benchmark Results

Performance Comparison

Metric	FluidAudio VAD	Industry Standard	Status
Accuracy	98.0%	85-90%	✅
Precision	96.2%	85-95%	✅
Recall	100.0%	80-90%	✅
F1-Score	98.0%	85.9% (Sohn's VAD)	✅
Processing Time	497.2s (100 files)	~1ms per 30ms chunk	✅

Industry Leaders:

• Silero VAD: ~90-95% F1 (DNN-based, 1.8MB model)
• WebRTC VAD: ~75-80% F1 (GMM-based, fast but lower accuracy)
• Sohn's VAD: 77.5% F1 (traditional approach)
• Modern DNNs: 85-97% F1 (varies by SNR conditions)

[github-actions]

VAD Benchmark Results

Performance Comparison

Metric	FluidAudio VAD	Industry Standard	Status
Accuracy	98.0%	85-90%	✅
Precision	96.2%	85-95%	✅
Recall	100.0%	80-90%	✅
F1-Score	98.0%	85.9% (Sohn's VAD)	✅
Processing Time	479.9s (100 files)	~1ms per 30ms chunk	✅

Industry Leaders:

• Silero VAD: ~90-95% F1 (DNN-based, 1.8MB model)
• WebRTC VAD: ~75-80% F1 (GMM-based, fast but lower accuracy)
• Sohn's VAD: 77.5% F1 (traditional approach)
• Modern DNNs: 85-97% F1 (varies by SNR conditions)

[github-actions]

VAD Benchmark Results

Performance Comparison

Metric	FluidAudio VAD	Industry Standard	Status
Accuracy	98.0%	85-90%	✅
Precision	96.2%	85-95%	✅
Recall	100.0%	80-90%	✅
F1-Score	98.0%	85.9% (Sohn's VAD)	✅
Processing Time	421.8s (100 files)	~1ms per 30ms chunk	✅

Industry Leaders:

• Silero VAD: ~90-95% F1 (DNN-based, 1.8MB model)
• WebRTC VAD: ~75-80% F1 (GMM-based, fast but lower accuracy)
• Sohn's VAD: 77.5% F1 (traditional approach)
• Modern DNNs: 85-97% F1 (varies by SNR conditions)

[github-actions]

VAD Benchmark Results

❌ Test Failed

No benchmark results were generated. Check the logs for details.

[github-actions]

VAD Benchmark Results

❌ Test Failed

No benchmark results were generated. Check the logs for details.

[github-actions]

VAD Benchmark Results

Performance Comparison

Metric	FluidAudio VAD	Industry Standard	Status
Accuracy	98.0%	85-90%	✅
Precision	96.2%	85-95%	✅
Recall	100.0%	80-90%	✅
F1-Score	98.0%	85.9% (Sohn's VAD)	✅
Processing Time	414.4s (100 files)	~1ms per 30ms chunk	✅

Industry Leaders:

• Silero VAD: ~90-95% F1 (DNN-based, 1.8MB model)
• WebRTC VAD: ~75-80% F1 (GMM-based, fast but lower accuracy)
• Sohn's VAD: 77.5% F1 (traditional approach)
• Modern DNNs: 85-97% F1 (varies by SNR conditions)

[github-actions]

Diarization Benchmark Results

Metric	Value	Target	Status
DER	18.7%	<30%	✅
JER	22.6%	<25%	✅
RTF	0.05x	<1.0x	✅

Performance Timing

Stage	Time (s)	%
Model Download	1.266	2.6
Model Compile	0.977	2.0
Audio Load	0.083	0.2
Segmentation	12.541	25.3
Embedding	34.633	69.9
Clustering	0.025	0.0
Total	49.525	100

Research Comparison

Method	DER	Year
FluidAudio	18.7%	2025
Powerset BCE	18.5%	2023
EEND	25.3%	2019
x-vector clustering	28.7%	2018

_{ES2004a • 1049.4s audio • 47.2s inference • Test runtime: 1m 10s • 07/24/2025, 02:43 PM EST}

[github-actions]

VAD Benchmark Results

Performance Comparison

Metric	FluidAudio VAD	Industry Standard	Status
Accuracy	98.0%	85-90%	✅
Precision	96.2%	85-95%	✅
Recall	100.0%	80-90%	✅
F1-Score	98.0%	85.9% (Sohn's VAD)	✅
Processing Time	427.2s (100 files)	~1ms per 30ms chunk	✅

Industry Leaders:

• Silero VAD: ~90-95% F1 (DNN-based, 1.8MB model)
• WebRTC VAD: ~75-80% F1 (GMM-based, fast but lower accuracy)
• Sohn's VAD: 77.5% F1 (traditional approach)
• Modern DNNs: 85-97% F1 (varies by SNR conditions)

📊 Detailed Research Comparisons

Paper	Dataset	F1-Score	Method
Silero VAD (2021)	TEDx	88.1%	LSTM-based lightweight model
WebRTC VAD	MUSAN	64.4%	GMM-based (traditional)
pyannote.audio (2020)	AMI	85.9%	SincTDNN architecture
MarbleNet (2020)	AVA-Speech	87.8%	1D time-channel separable CNN
FluidAudio VAD	MUSAN-mini	98.0%	CoreML-optimized Silero

Note: Direct comparisons should consider dataset differences. MUSAN contains challenging noise conditions.

[github-actions]

ASR Benchmark Results

Dataset	WER Avg	WER Med	RTFx	Status
test-clean	5.42%	0.00%	1.22x	✅
test-other	9.60%	2.44%	0.92x	✅

_{100 files per dataset • Test runtime: 13m15s • 07/24/2025, 02:55 PM EST}

_{RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time
Processing time includes: Model inference on Apple Neural Engine, audio preprocessing, state resets between files, token-to-text conversion, and file I/O
Example: RTFx of 2.0x means 10 seconds of audio processed in 5 seconds (2x faster than real-time)}

_{Note: CI RTFx degraded by M1/M2 Mac virtualization. M1 Mac test: ~28x (clean), ~25x (other). Testing per HuggingFace Open ASR Leaderboard.}