enne2/qwentts.cpp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs

Browse Source
This commit is contained in:
Pascal
2026-05-31 14:09:28 +02:00
parent 1c62494faa
commit 127bde4eea
1 changed files with 25 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+25 -16
View File
@@ -24,8 +24,8 @@ Predictor MTP head that expands each semantic token into the 15
acoustic codes of that frame. The codes are turned into a waveform by a acoustic codes of that frame. The codes are turned into a waveform by a
separate audio tokenizer, the Qwen3-TTS-Tokenizer-12Hz (Mimi-style separate audio tokenizer, the Qwen3-TTS-Tokenizer-12Hz (Mimi-style
SEANet plus a transformer, residual vector quantiser, a ConvNeXt SEANet plus a transformer, residual vector quantiser, a ConvNeXt
upsampler and a DAC v2 decoder), running at 12.5 frames per second over upsampler and a DAC decoder, the Descript Audio Codec family), running
24 kHz mono audio. at 12.5 frames per second over 24 kHz mono audio.
Public checkpoints, two talker sizes : Public checkpoints, two talker sizes :
@@ -197,7 +197,7 @@ tensors
tok_dec.pre_conv.* conv_pre into the decoder transformer tok_dec.pre_conv.* conv_pre into the decoder transformer
tok_dec.pre_tfm.input_proj / output_proj / norm + transformer blocks tok_dec.pre_tfm.input_proj / output_proj / norm + transformer blocks
tok_dec.upsample.* ConvNeXt upsample (2 blocks, 4x) tok_dec.upsample.* ConvNeXt upsample (2 blocks, 4x)
tok_dec.<dac>.* DAC v2 decoder chain tok_dec.<dac>.* DAC decoder chain
tok_dec.vq_first.output_proj / vq_rest.output_proj tok_dec.vq_first.output_proj / vq_rest.output_proj
tok_{enc,dec}.{vq_*}.<idx>.codebook RVQ codebook entries tok_{enc,dec}.{vq_*}.<idx>.codebook RVQ codebook entries
``` ```
@@ -304,7 +304,7 @@ codes [T, 16] i32
-> ConvNeXt upsample : 2 blocks, each CausalTransConv k=2 stride=2 then a -> ConvNeXt upsample : 2 blocks, each CausalTransConv k=2 stride=2 then a
ConvNeXt block (depthwise causal conv k=7, pointwise 1024 -> 4096 -> 1024, ConvNeXt block (depthwise causal conv k=7, pointwise 1024 -> 4096 -> 1024,
LayerScale gamma); together 4x on the time axis at channels 1024 LayerScale gamma); together 4x on the time axis at channels 1024
-> DAC v2 : conv_pre k=7 (1024 -> 1536), 4 blocks strides 8/5/4/3 -> DAC : conv_pre k=7 (1024 -> 1536), 4 blocks strides 8/5/4/3
channels 1536 -> 768 -> 384 -> 192 -> 96, each block SnakeBeta then channels 1536 -> 768 -> 384 -> 192 -> 96, each block SnakeBeta then
CausalTransConv then 3 ResUnits (dilations 1/3/9), snake_post, CausalTransConv then 3 ResUnits (dilations 1/3/9), snake_post,
conv_post k=7 (96 -> 1) conv_post k=7 (96 -> 1)
@@ -357,18 +357,27 @@ text and the reference codes (mode B, ICL).
### Modes and the validation rules ### Modes and the validation rules
The synthesis mode is read from the talker `model_type` at load, not The synthesis mode is read from the talker `model_type` at load, not
from a CLI flag. `qt_synthesize` enforces seven rules and surfaces them from a CLI flag. `qt_synthesize` validates the params against that
as `QT_STATUS_MODE_INVALID` or `QT_STATUS_INVALID_PARAMS` with a `model_type` before any compute and emits a verbatim `qt_last_error()`.
descriptive `qt_last_error()` : The checks split across two return codes.
Five conditions mean the flag set does not match the loaded checkpoint
family, returning `QT_STATUS_MODE_INVALID` :
``` ```
--speaker valid only for custom_voice MODE_INVALID --speaker given but model_type != custom_voice
--instruct rejected for base MODE_INVALID --instruct given but model_type == base
custom_voice requires --speaker MODE_INVALID model_type == custom_voice but no --speaker
voice_design requires a non-empty --instruct MODE_INVALID model_type == voice_design but --instruct empty or missing
--ref-wav valid only for base MODE_INVALID --ref-wav given but model_type != base
--speaker and --ref-wav are mutually exclusive INVALID_PARAMS ```
--ref-text requires --ref-wav INVALID_PARAMS
Two conditions mean the flag combination is self-contradictory whatever
the model_type, returning `QT_STATUS_INVALID_PARAMS` :
```
--speaker and --ref-wav both given (mutually exclusive)
--ref-text given without --ref-wav
``` ```
### Frame loop ### Frame loop
@@ -568,7 +577,7 @@ src/
tokenizer-transformer.h 8-layer local-causal decoder transformer (sw 72) tokenizer-transformer.h 8-layer local-causal decoder transformer (sw 72)
convnext-block.h ConvNeXt upsample stage (2 blocks, 4x) convnext-block.h ConvNeXt upsample stage (2 blocks, 4x)
causal-trans-conv.h Causal ConvTranspose1d via col2im_1d causal-trans-conv.h Causal ConvTranspose1d via col2im_1d
dac-decoder-v2.h DAC v2 decoder (strides 8/5/4/3, SnakeBeta) dac-decoder-v2.h DAC decoder (Descript Audio Codec; strides 8/5/4/3, SnakeBeta)
codec-chunked-decode.h Bounded-VRAM decode with rolling left context codec-chunked-decode.h Bounded-VRAM decode with rolling left context
prompt-builder.h Talker prefix assembly, modes, ICL geometry prompt-builder.h Talker prefix assembly, modes, ICL geometry
@@ -686,7 +695,7 @@ exist precisely to bisect that forward path stage by stage.
strided convs for down/up sampling. strided convs for down/up sampling.
DAC Descript Audio Codec. Convolutional decoder over the DAC Descript Audio Codec. Convolutional decoder over the
quantised latent. The v2 decoder uses SnakeBeta. quantised latent, here with SnakeBeta activations.
SnakeBeta Periodic activation `x + (1/beta) * sin^2(alpha*x)` with SnakeBeta Periodic activation `x + (1/beta) * sin^2(alpha*x)` with
`exp()` applied to alpha and beta, folded at load. `exp()` applied to alpha and beta, folded at load.