feat: v8 factorized decoder cascade — Stage A acoustic model + Stage B conditioned vocoder - #13
Conversation
… ISTFT Clean MCS-philosophy design mirroring MioCodec teacher's ISTFT head: content(768) → prenet 7L (causal RoPE, no speaker) @25hz → ConvTranspose ×2 + Snake → 50Hz, 512d → FiLM(speaker) → dilated TCN 4 blocks @50Hz → ConvTranspose ×3×3 + Snake → 450Hz → single Linear → mag+phase → iSTFT Training: direct teacher ISTFT head distillation (MPS — no CPU STFT). Loss: L1(mag_log) + anti-wrap-phase·0.5 All on MPS, no CPU spectral losses. Params: 32.3M (7L), causal 0.00, latency 3.3ms.
Each frequency band gets its own Linear projection head. 4 bands × ~50 bins each → 4× fewer bins per head → easier phase prediction. Low freqs get finer bands (32 bins each), high freqs get wider band (69 bins). Bands: 0-31, 32-63, 64-127, 128-196 (197 total bins). Params: 23.5M (same as single head).
Smooths band boundaries and shares mag+phase information across neighboring frequency bins via depthwise CausalConv1d(k=5). Stacks (mag_log, phase) as 2-channel (B*T, 2, 197) input, conv along freq axis, then splits back. Fixes phase discontinuity at band boundaries while keeping the multi-band head advantage. +20 params only.
…ssions) Multi-band: no significant improvement over single head. Freq_smooth: actively harmful — depthwise freq conv mixed mag+phase channels, causing slower convergence (mel_cos 0.63 vs 0.70 at E07). Single Linear(512→n_fft+2) is simplest and performs best. Keep 10L prenet + phase-weight 3.0.
…oder) Replace the failed monolithic-GAN decoder with a two-stage streaming cascade split at the teacher's 50Hz waist: - Stage A (AcousticModel): teacher steps 1-5 causal replica + supervised mel/logF0/voicing/energy heads @150hz. Pure regression vs ground truth; carries the voice-conversion work. - Stage B (ConditionedVocoder): causal Vocos-class conv vocoder conditioned on the 150Hz acoustic frames, adversarial (MPD + MRD, mel anchor), no speaker input. Trainable on any 44.1kHz audio. New: acoustics.py (causal left-aligned 150Hz feature extraction, zero future leakage), decoder_v8.py, train_stage_a.py, train_stage_b.py, eval_perceptual.py. Extended: cache.py (--what acoustics), data.py (AcousticDataset), discriminators.py (MultiResolutionDiscriminator + VocoderDiscriminator). Both stages strictly causal (0 look-ahead, verified future->0.00e+00). Stage B iSTFT n_fft=882/hop=294 (67% overlap) suppresses the 150Hz frame-rate buzz; n_fft=1176 is numerically untrainable on MPS. lr-warmup + clamp-before-exp needed for training stability. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 789f2e9b12
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if return_spec: | ||
| return wav, mag, phase |
There was a problem hiding this comment.
Return log magnitudes for the MCS distillation loss
train_decoder_mcs unpacks this return_spec=True value as p_mag_log and applies F.l1_loss(p_mag_log, t_mag_log) against the teacher's raw ISTFT-head log magnitudes. Returning the exponentiated/clamped mag here puts the student and teacher on different scales, so the advertised log-magnitude distillation target is wrong and will train the MCS decoder toward incompatible values; return mag_log here or take logs before the loss.
Useful? React with 👍 / 👎.
Summary
Replaces the failed monolithic-GAN decoder with a two-stage streaming cascade split at the teacher's 50Hz waist. Stage A (
AcousticModel) is a causal replica of the teacher's steps ①–⑤ plus supervised heads that regress mel/logF0/voicing/energy @150hz — this carries all the voice-conversion work. Stage B (ConditionedVocoder) is a causal Vocos-class conv vocoder conditioned on those 150Hz acoustic frames, trained adversarially (MPD + MRD with a mel anchor), with no speaker input. The factorization lets each stage be trained and gated independently (Stage B via copy-synthesis).Motivation
The prior monolithic adversarial decoder produced non-speech output: 21.5 bits/frame of content conditioning is far below the densely-conditioned regime (mel @ ~10ms) where GAN vocoders are proven, so the generator had to adversarially invent all acoustics and the waveform at once. Splitting the problem — regression for "what should it sound like" (well-posed, stable) and a densely-conditioned GAN for "render it" (the proven regime) — removes that ill-posedness.
Type
Checklist
tests/test_streaming_invariant.pypasses (encoder unaffected)pytestnot run this session (training job active; see Verification)ruffnot runmypynot runastrape.*modules import cleanfuture → 0.00e+00DECODER_V8_DESIGN.md(gitignored per repo*.mdpolicy)Verification
tests/test_streaming_invariant.py— all passed (stem/encoder truncation/future-padding).python -m astrape.decoder_v8— Stage Amel[:240]and Stage Bwavbothmax_diff = 0.00e+00on future-perturbation.acoustics.pyleft-aligned framing — perturb-future test shows zero leakage on earlier frames (a centered STFT would silently add ~10ms look-ahead).frame_buzz_val19 → ~8 dB,mel_sor_cos_val0.91+,skip=0, stable GAN (d≈3.5, no collapse).Notes for reviewers
frame_buzzis a new required metric. HNR/flat_hi/centroid are broadband and blind to the 150Hz frame-rate comb + inter-harmonic valley-fill that made copy-synth "clean but off";eval_perceptual.pyaddsharm_clarity+frame_buzzto catch it.n_fft=882/hop=294(67% overlap) is a deliberate choice: it suppresses the frame-rate buzz to ~GT level, whereas 50% overlap (588) re-creates it and 75% (1176) — though ideal on paper — is numerically untrainable on MPS (NaN gradients within ~5 steps, resistant to init/lr/warmup/eps fixes).clamp(mag_log)beforeexp(avoids fp32 overflow → NaN grad) and--lr-warmup-stepsare required; both added here.wave_cosis retired as a gate (phase-sensitive → ~0 even for good audio);mel_coswith content-alignment is the similarity gate.🤖 Generated with Claude Code