Hey Rui Shu,
I really learn lots from your post, recently I read a candidate paper for ICLR 2019:
https://openreview.net/pdf?id=rygkk305YQ
'HIERARCHICAL GENERATIVE MODELING FOR
CONTROLLABLE SPEECH SYNTHESIS
which use GMVAE too, I compared it to yours in gmvae.py, what's in the paper is quite confusing for get y from sampled z, have you ever read it too?