Speech Representation Demo

SphereVAE

Hyperspherical Latent Autoencoders for Robust Autoregressive Speech Representation Modeling

Code & Models Listen to Samples

Abstract

With the rapid development of speech generation technology, discrete codec representations have been widely used because they provide a stable prediction paradigm. In expressive speech generation, however, the quantization bottleneck of discrete codecs results in information gaps in fine-grained prosody, timbre, pronunciation, and frame-to-frame continuity. Continuous representations (e.g., VAE latents), by eliminating this constraint, have emerged as a more effective alternative for autoregressive modeling. Yet when continuous representations are used as autoregressive prediction targets, prediction errors can accumulate along the generation chain, causing latent drift and degrading long-form stability. To mitigate this problem, we propose SphereVAE, which constrains the VAE latent space to the unit hypersphere. SphereVAE defines a Power Spherical posterior on the hypersphere and regularizes the latent distribution toward a uniform prior, so that information is encoded mainly by directional variation, providing a bounded geometric target for autoregressive prediction and reducing the risk of norm drift. SphereVAE underperforms the standard VAE on reconstruction metrics due to reduced latent freedom. However, when integrated into VoxCPM for zero-shot TTS and long-text generation, it yields lower content error rates with comparable speaker similarity, and shows more stable long-range speaker consistency. These results indicate that an appropriate latent geometric constraint can effectively mitigate autoregressive error accumulation and drift in speech generation.

Method

SphereVAE Model Architecture

SphereVAE adopts an encoder-decoder architecture for learning speech representations. Instead of the Euclidean Gaussian posterior used in a standard VAE, it constrains the latent to the unit hypersphere and models it with a Power Spherical distribution.

SphereVAE model architecture
Overall architecture of SphereVAE. The encoder maps input speech into latent parameters; the direction branch is L2-normalized to lie on the unit hypersphere, and the Power Spherical posterior is regularized toward the uniform hyperspherical prior. The decoder reconstructs speech from the sampled latent representation. Takeaway. The sampled hyperspherical latent is deterministically rescaled by the square root of the latent dimension before decoding. Its norm is therefore fixed rather than free to vary, so speech information is organized primarily through bounded directional variation. This may reduce reconstruction freedom while providing a clearer geometric boundary for autoregressive prediction.

Evaluation

Objective Evaluation Results

We evaluate the representation at two levels: tokenizer reconstruction measures local fidelity, while zero-shot TTS tests whether each corresponding latent representation is easy to predict with VoxCPM as the autoregressive backbone. The comparison exposes the intended trade-off between reconstruction capacity and generation robustness.

Reconstruction results on LibriSpeech-PC test-clean

Model STOI ↑ PESQ-WB ↑ MCD ↓ UTMOS ↑ SIM ↑ WER ↓
VAE 0.9707 3.470 1.996 3.966 0.772 0.038
SphereVAE 0.9574 3.009 2.443 3.580 0.673 0.042
SemanticVAE 0.9689 3.411 2.043 3.913 0.780 0.038
σ-VAE 0.9656 3.370 2.111 3.847 0.752 0.040

Interpretation. VAE leads on the reconstruction-oriented metrics, while SemanticVAE gives the highest SIM score. By restricting latent vectors to the unit hypersphere, SphereVAE reduces part of the reconstruction freedom available to an unconstrained Euclidean VAE. Its main goal is not to maximize one-step reconstruction quality, but to provide a constrained continuous latent space for autoregressive generation.

Zero-shot TTS Model results on SeedTTS-eval

Model EN WER ↓ EN SIM ↑ ZH CER ↓ ZH SIM ↑
SphereVAE 5.305 0.654 1.141 0.733
VAE 5.591 0.655 1.327 0.731
SemanticVAE 7.702 0.654 1.241 0.731
σ-VAE 9.442 0.534 4.621 0.675

Interpretation. SphereVAE obtains the lowest English WER and Chinese CER among the four VAE representations. The SIM differences among SphereVAE, VAE, and SemanticVAE are small: VAE is slightly higher on English, while SphereVAE is slightly higher on Chinese; σ-VAE is clearly lower on both language subsets. The benefit of the hyperspherical constraint therefore lies mainly in autoregressive predictability and generation stability rather than directly optimizing every similarity metric.

Analysis

Long-Text and Latent-Space Analysis

The following analyses ask why the generation results improve. We first track speaker identity as generation proceeds, then inspect whether the hyperspherical constraint produces a more organized latent geometry.

Segment-wise speaker similarity curves on Long-TTS-Eval
Segment-wise speaker similarity curves on Long-TTS-Eval. The generated audio is segmented every 3 seconds, and each segment is compared with the prompt speech. Observation. Over the common 0–27 s analysis window, SphereVAE maintains higher similarity in most comparable segments, especially over the late 18–27 s range. This trend suggests that the hyperspherical latent space helps reduce speaker drift in long-horizon autoregressive generation.

Long-text speaker similarity retention on Long-TTS-Eval over the common 0–27 s range

Model F_SIM ↑ L_SIM ↑ M_SIM ↑ ΔSIM ↓ SIMlate ↑
VAE 0.653 0.410 0.567 0.243 0.494
SphereVAE 0.667 0.447 0.591 0.220 0.524
SemanticVAE 0.655 0.404 0.562 0.251 0.480
σ-VAE 0.571 0.392 0.533 0.179 0.478

All rows use the common 0–27 s range of nine 3-second segments. F_SIM and L_SIM are the first- and last-segment scores, M_SIM is the mean across all nine segments, ΔSIM is the first-to-last decrease, and SIMlate is the mean over the final three segments (18–27 s).

Interpretation. SphereVAE leads F_SIM, L_SIM, M_SIM, and SIMlate, while also reducing similarity decay relative to VAE and SemanticVAE. σ-VAE has the smallest first-to-last drop, but its first-segment and mean SIM are both lower than the other models. Considering mean SIM, late-segment SIM, and first-to-last drop together, SphereVAE preserves higher speaker similarity while maintaining a smaller decline than VAE and SemanticVAE.

Latent-space visualizations

SphereVAE speaker-colored t-SNE

(a) SphereVAE t-SNE

VAE speaker-colored t-SNE

(b) VAE t-SNE

Joint SphereVAE and VAE t-SNE overlay

(c) Joint t-SNE

Silhouette Score comparison

(d) Silhouette

Davies-Bouldin Index comparison

(e) Davies--Bouldin

Latent-space comparison between SphereVAE and standard VAE on LibriSpeech-PC test-clean. (a) SphereVAE speaker-colored t-SNE, (b) VAE speaker-colored t-SNE, (c) joint t-SNE overlay, (d) Silhouette Score, and (e) Davies--Bouldin Index. Observation. SphereVAE forms more compact and separable speaker-conditioned neighborhoods: its Silhouette Score increases from 0.2218 to 0.2550, while its Davies--Bouldin Index decreases from 1.6890 to 1.5729. The joint projection also shows that the constraint changes the global organization of the representation rather than merely rescaling standard VAE latents.
PCA cumulative explained variance comparison between SphereVAE and VAE
PCA cumulative explained variance comparison between SphereVAE and standard VAE latent features. Observation. The SphereVAE curve rises faster, so fewer principal components explain the same fraction of variance. We interpret this as a more concentrated and structured representation: the hyperspherical constraint suppresses redundant radial degrees of freedom while preserving directional variation useful to the speech model.

Audio Demo

Speech Samples

These samples provide the qualitative counterpart to the objective results. For each row, compare models under the same source or prompt, using headphones if possible. Listen for content accuracy, naturalness, speaker identity, and whether artifacts or identity drift become stronger over time.

VAE Reconstruction

Compare each reconstruction with Ground Truth. Focus on short-range fidelity: timbre, pronunciation, transient detail, background texture, and codec artifacts.

Sample Ground Truth VAE SphereVAE SemanticVAE σ-VAE
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
Sample 6
Sample 7

What this demonstrates. SphereVAE is not designed to win pure reconstruction. These samples make the cost of the geometric constraint audible and should be considered alongside the stronger autoregressive results below.

VoxCPM Zero-Shot TTS on SeedTTS-eval

The Prompt establishes the target voice. Compare five Chinese and five English cases, paying attention to whether each system preserves that voice while speaking the new content clearly and naturally.

Sample Prompt VAE SphereVAE SemanticVAE σ-VAE
ZH Sample 1
ZH Sample 2
ZH Sample 3
ZH Sample 4
ZH Sample 5
EN Sample 1
EN Sample 2
EN Sample 3
EN Sample 4
EN Sample 5

What this demonstrates. The zero-shot samples test the practical balance between intelligibility and speaker similarity. SphereVAE's bounded target is intended to reduce prediction errors without sacrificing the identity conveyed by the prompt.

VoxCPM Long-Text Inference

Listen beyond the opening seconds. The most revealing differences are late in each sample: changes in speaker identity, muffled or electronic artifacts, unstable volume, repeated content, and declining intelligibility.

Sample Prompt VAE SphereVAE SemanticVAE σ-VAE
Sample 1
Sample 2
Sample 3
Sample 4
Sample 5
Sample 6
Sample 7

What this demonstrates. Long-text inference amplifies small autoregressive errors. Across these matched examples, SphereVAE is intended to preserve a clearer and more consistent voice deeper into the sequence, complementing the segment-wise similarity analysis above.