Skip to content

feat: v8 factorized decoder cascade — Stage A acoustic model + Stage B conditioned vocoder - #13

Merged
stremtec merged 8 commits into
mainfrom
feat/v8-decoder-cascade
Jul 2, 2026
Merged

feat: v8 factorized decoder cascade — Stage A acoustic model + Stage B conditioned vocoder#13
stremtec merged 8 commits into
mainfrom
feat/v8-decoder-cascade

Conversation

@stremtec

@stremtec stremtec commented Jul 2, 2026

Copy link
Copy Markdown
Owner

Summary

Replaces the failed monolithic-GAN decoder with a two-stage streaming cascade split at the teacher's 50Hz waist. Stage A (AcousticModel) is a causal replica of the teacher's steps ①–⑤ plus supervised heads that regress mel/logF0/voicing/energy @150hz — this carries all the voice-conversion work. Stage B (ConditionedVocoder) is a causal Vocos-class conv vocoder conditioned on those 150Hz acoustic frames, trained adversarially (MPD + MRD with a mel anchor), with no speaker input. The factorization lets each stage be trained and gated independently (Stage B via copy-synthesis).

Motivation

The prior monolithic adversarial decoder produced non-speech output: 21.5 bits/frame of content conditioning is far below the densely-conditioned regime (mel @ ~10ms) where GAN vocoders are proven, so the generator had to adversarially invent all acoustics and the waveform at once. Splitting the problem — regression for "what should it sound like" (well-posed, stable) and a densely-conditioned GAN for "render it" (the proven regime) — removes that ill-posedness.

Type

  • feat — new feature
  • fix / refactor / perf / docs / ci / test / chore

Checklist

  • Streaming invariance holds — tests/test_streaming_invariant.py passes (encoder unaffected)
  • All tests pass — pytest not run this session (training job active; see Verification)
  • Lint clean — ruff not run
  • Type check clean — mypy not run
  • Imports are intact — all 8 astrape.* modules import clean
  • No look-ahead introduced — Stage A & B verified future → 0.00e+00
  • Architecture doc — design captured in DECODER_V8_DESIGN.md (gitignored per repo *.md policy)

Verification

  • Streaming invariance: tests/test_streaming_invariant.py — all passed (stem/encoder truncation/future-padding).
  • Strict causality: python -m astrape.decoder_v8 — Stage A mel[:240] and Stage B wav both max_diff = 0.00e+00 on future-perturbation.
  • Acoustic target causality: acoustics.py left-aligned framing — perturb-future test shows zero leakage on earlier frames (a centered STFT would silently add ~10ms look-ahead).
  • Stage B training (in progress): frame_buzz_val 19 → ~8 dB, mel_sor_cos_val 0.91+, skip=0, stable GAN (d≈3.5, no collapse).

Notes for reviewers

  • frame_buzz is a new required metric. HNR/flat_hi/centroid are broadband and blind to the 150Hz frame-rate comb + inter-harmonic valley-fill that made copy-synth "clean but off"; eval_perceptual.py adds harm_clarity + frame_buzz to catch it.
  • iSTFT n_fft=882/hop=294 (67% overlap) is a deliberate choice: it suppresses the frame-rate buzz to ~GT level, whereas 50% overlap (588) re-creates it and 75% (1176) — though ideal on paper — is numerically untrainable on MPS (NaN gradients within ~5 steps, resistant to init/lr/warmup/eps fixes).
  • Training stability: clamp(mag_log) before exp (avoids fp32 overflow → NaN grad) and --lr-warmup-steps are required; both added here.
  • wave_cos is retired as a gate (phase-sensitive → ~0 even for good audio); mel_cos with content-alignment is the similarity gate.

🤖 Generated with Claude Code

stremtec and others added 8 commits July 1, 2026 12:21
… ISTFT

Clean MCS-philosophy design mirroring MioCodec teacher's ISTFT head:
  content(768) → prenet 7L (causal RoPE, no speaker) @25hz
  → ConvTranspose ×2 + Snake → 50Hz, 512d
  → FiLM(speaker)
  → dilated TCN 4 blocks @50Hz
  → ConvTranspose ×3×3 + Snake → 450Hz
  → single Linear → mag+phase → iSTFT

Training: direct teacher ISTFT head distillation (MPS — no CPU STFT).
  Loss: L1(mag_log) + anti-wrap-phase·0.5
  All on MPS, no CPU spectral losses.

Params: 32.3M (7L), causal 0.00, latency 3.3ms.
Each frequency band gets its own Linear projection head.
4 bands × ~50 bins each → 4× fewer bins per head → easier
phase prediction. Low freqs get finer bands (32 bins each),
high freqs get wider band (69 bins).

Bands: 0-31, 32-63, 64-127, 128-196 (197 total bins).
Params: 23.5M (same as single head).
Smooths band boundaries and shares mag+phase information across
neighboring frequency bins via depthwise CausalConv1d(k=5).
Stacks (mag_log, phase) as 2-channel (B*T, 2, 197) input,
conv along freq axis, then splits back.

Fixes phase discontinuity at band boundaries while keeping
the multi-band head advantage. +20 params only.
…ssions)

Multi-band: no significant improvement over single head.
Freq_smooth: actively harmful — depthwise freq conv mixed mag+phase
channels, causing slower convergence (mel_cos 0.63 vs 0.70 at E07).

Single Linear(512→n_fft+2) is simplest and performs best.
Keep 10L prenet + phase-weight 3.0.
…oder)

Replace the failed monolithic-GAN decoder with a two-stage streaming cascade
split at the teacher's 50Hz waist:

- Stage A (AcousticModel): teacher steps 1-5 causal replica + supervised
  mel/logF0/voicing/energy heads @150hz. Pure regression vs ground truth;
  carries the voice-conversion work.
- Stage B (ConditionedVocoder): causal Vocos-class conv vocoder conditioned on
  the 150Hz acoustic frames, adversarial (MPD + MRD, mel anchor), no speaker
  input. Trainable on any 44.1kHz audio.

New: acoustics.py (causal left-aligned 150Hz feature extraction, zero future
leakage), decoder_v8.py, train_stage_a.py, train_stage_b.py, eval_perceptual.py.
Extended: cache.py (--what acoustics), data.py (AcousticDataset),
discriminators.py (MultiResolutionDiscriminator + VocoderDiscriminator).

Both stages strictly causal (0 look-ahead, verified future->0.00e+00).
Stage B iSTFT n_fft=882/hop=294 (67% overlap) suppresses the 150Hz frame-rate
buzz; n_fft=1176 is numerically untrainable on MPS. lr-warmup + clamp-before-exp
needed for training stability.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@stremtec
stremtec merged commit bfee55c into main Jul 2, 2026
4 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 789f2e9b12

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread astrape/decoder_mcs.py
Comment on lines +218 to +219
if return_spec:
return wav, mag, phase

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Return log magnitudes for the MCS distillation loss

train_decoder_mcs unpacks this return_spec=True value as p_mag_log and applies F.l1_loss(p_mag_log, t_mag_log) against the teacher's raw ISTFT-head log magnitudes. Returning the exponentiated/clamped mag here puts the student and teacher on different scales, so the advertised log-magnitude distillation target is wrong and will train the MCS decoder toward incompatible values; return mag_log here or take logs before the loss.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant