qwentts.cpp/docs/ARCHITECTURE.md

Architecture

Technical reference for qwentts.cpp, the GGML port of Qwen3-TTS 12 Hz (Qwen team, Alibaba). This document covers the model, the conversion to GGUF, the inference pipeline, the GGML graph conventions, and the CLI tools.

Upstream model

Qwen3-TTS 12 Hz (Qwen team / Alibaba, Apache 2.0) is a multilingual zero shot text-to-speech system covering 11 languages with Mandarin dialect support. It targets three checkpoint families :

base plain synthesis with an auto-picked voice, plus zero shot voice cloning from a reference clip custom_voice named speakers selected by name, some carrying a dialect override voice_design a synthesised speaker driven by a free text attribute instruction

The system is autoregressive. Two language models run in series : a Talker that emits the semantic codebook one frame at a time, and a Code Predictor MTP head that expands each semantic token into the 15 acoustic codes of that frame. The codes are turned into a waveform by a separate audio tokenizer, the Qwen3-TTS-Tokenizer-12Hz (Mimi-style SEANet plus a transformer, residual vector quantiser, a ConvNeXt upsampler and a DAC decoder, the Descript Audio Codec family), running at 12.5 frames per second over 24 kHz mono audio.

Public checkpoints, two talker sizes :

Talker Qwen3 0.6B or 1.7B decoder, codec_head over 3072 Code Predictor 5-layer Qwen3, 15 acoustic codebooks of 2048 each Speaker encoder ECAPA-TDNN, base checkpoints only (x-vector cloning) Audio codebooks 16 residual (1 semantic + 15 acoustic), 2048 each Audio framerate 12.5 Hz Hop length 1920 samples Sample rate 24 kHz mono Semantic SR 24 kHz (SEANet input, no separate semantic rate)

Build

git clone --recurse-submodules https://github.com/ServeurpersoCom/qwentts.cpp.git
cd qwentts.cpp
./buildcuda.sh      # NVIDIA GPU
./buildvulkan.sh    # AMD/Intel GPU (Vulkan)
./buildcpu.sh       # CPU only
./buildall.sh       # all backends, runtime DL loading

The GGML submodule lives at https://github.com/ServeurpersoCom/ggml.git and provides the two custom ops the codec needs : GGML_OP_SNAKE and GGML_OP_COL2IM_1D. Both have CPU, CUDA, Metal, and Vulkan kernels.

Model conversion

./checkpoints.sh    # hf download Qwen/Qwen3-TTS-12Hz-* -> checkpoints/
./convert.py        # F32 GGUFs (one talker per mode/size + tokenizer) -> models/
./quantize.sh       # BF16 / Q8_0 / Q4_K_M derived from each F32 source

convert.py writes the F32 source of truth. Each talker checkpoint produces one qwen-talker-{size}-{mode}-{variant}.gguf, and the shared tokenizer produces one qwen-tokenizer-12hz-{variant}.gguf. The talker and the Code Predictor share the same Qwen3 layout so a single tensor renamer covers both.

Quantisation policy, centralised in tools/quantize.cpp should_quantize and mirrored in quantize.sh :

RVQ codebooks (quantizer.quantizers.*), the input_proj / output_proj that wrap them, and the speaker encoder stay at F32 in every variant. Nearest-neighbour lookup is sensitive to per-row quantisation noise ; even BF16 mantissa truncation drifts codes enough to break voice fidelity.

1D tensors (LayerScale gamma, biases, norms, snake alpha and beta) stay at F32.

Conv kernels with non-alignable rows (K = 7, 3, 1) never divide a K-quant block size, so the quantiser lands on F16 directly. F16 has no block size and matches the runtime target dtype on every backend.

The Talker LM (hidden divisible by 256) follows standard llama.cpp K-quant. The Code Predictor MTP head and the speaker encoder live in the talker GGUF and inherit its quantisation.

GGUF layout

qwen-talker-{size}-{mode}-{variant}.gguf (arch qwen3-tts, 404 tensors on the 1.7B voice_design build) :

metadata
  general.architecture                          qwen3-tts
  qwen3-tts.tokenizer_type                       qwen3_tts_tokenizer_12hz
  qwen3-tts.model_size                           0.6b | 1.7b
  qwen3-tts.model_type                           base | custom_voice | voice_design
  qwen3-tts.num_code_groups                      16

  qwen3-tts.talker.embedding_length              1024 (0.6B) | 2048 (1.7B)
  qwen3-tts.talker.feed_forward_length           3072 (0.6B) | 6144 (1.7B)
  qwen3-tts.talker.block_count                    28
  qwen3-tts.talker.attention.head_count           16
  qwen3-tts.talker.attention.head_count_kv         8     (GQA 2:1)
  qwen3-tts.talker.attention.key_length          128
  qwen3-tts.talker.vocab_size                    3072   (codec_head)
  qwen3-tts.talker.text_vocab_size               151936
  qwen3-tts.talker.text_hidden_size              2048   (text-embedding width, both sizes)
  qwen3-tts.talker.context_length                32768
  qwen3-tts.talker.rope.freq_base                1e6
  qwen3-tts.talker.attention.layer_norm_rms_epsilon  1e-6
  qwen3-tts.talker.position_id_per_seconds       13
  qwen3-tts.talker.rope.mrope_section            [24, 20, 20]
  qwen3-tts.talker.mrope_interleaved             false

  qwen3-tts.code_pred.embedding_length           1024   (both sizes)
  qwen3-tts.code_pred.feed_forward_length        3072
  qwen3-tts.code_pred.block_count                 5
  qwen3-tts.code_pred.attention.head_count        16
  qwen3-tts.code_pred.attention.head_count_kv      8
  qwen3-tts.code_pred.attention.key_length       128
  qwen3-tts.code_pred.vocab_size                 2048
  qwen3-tts.code_pred.context_length             65536
  qwen3-tts.code_pred.attention.layer_norm_rms_epsilon  1e-6
  qwen3-tts.code_pred.rope.freq_base             1e6

  qwen3-tts.spk_enc.embedding_length             2048   (base only)
  qwen3-tts.spk_enc.sample_rate                  16000  (base only)

  qwen3-tts.codec.{pad,bos,eos,think,nothink,think_bos,think_eos}_id
  qwen3-tts.codec.language_names / language_ids
  qwen3-tts.codec.speaker_names / speaker_ids / speaker_dialects  (custom_voice)
  qwen3-tts.text.{im_start,im_end,tts_pad,tts_bos,tts_eos}_id
  generation.*                                   sampling defaults
  tokenizer (Qwen2 BPE, 151676 vocab, 151291 merges, eos 151643)

tensors
  talker.text_embd.weight                        text token embedding
  talker.codec_embd.weight                       (3072, hidden) codec/code embedding
  talker.text_proj.fc1.{weight,bias}             text-embedding -> hidden, 2-layer
  talker.text_proj.fc2.{weight,bias}
  talker.codec_head.weight                       hidden -> 3072, codebook 0 logits
  talker.output_norm.weight                      final RMSNorm
  talker.blk.0..27.attn_q / attn_k / attn_v / attn_o          GQA, no bias
  talker.blk.0..27.attn_q.q_norm / attn_k.k_norm              per-head RMSNorm (128,)
  talker.blk.0..27.attn_norm / ffn_norm                       RMSNorm
  talker.blk.0..27.ffn.{gate,up,down}_proj                    SwiGLU, no bias
  code_pred.blk.0..4.*                           same layout, 5 layers
  code_pred.output_norm.weight
  code_pred.mtp_proj.{weight,bias}               talker_hidden -> code_pred hidden (1.7B only, identity on 0.6B)
  spk_enc.*                                      ECAPA-TDNN (base only)

qwen-tokenizer-12hz-{variant}.gguf (arch qwen3-tts-tokenizer, 398 tensors) :

metadata
  qwen3-tts-tokenizer.input_sample_rate          24000
  qwen3-tts-tokenizer.output_sample_rate         24000
  qwen3-tts-tokenizer.decode_upsample_rate       1920
  qwen3-tts-tokenizer.encode_downsample_rate     1920
  qwen3-tts-tokenizer.encoder_valid_num_quantizers  16

  qwen3-tts-tokenizer.decoder.latent_dim         1024
  qwen3-tts-tokenizer.decoder.codebook_size      2048
  qwen3-tts-tokenizer.decoder.codebook_dim_internal  256
  qwen3-tts-tokenizer.decoder.hidden_size        512
  qwen3-tts-tokenizer.decoder.intermediate_size  1024
  qwen3-tts-tokenizer.decoder.head_dim            64
  qwen3-tts-tokenizer.decoder.num_attention_heads 16
  qwen3-tts-tokenizer.decoder.num_key_value_heads 16    (no GQA)
  qwen3-tts-tokenizer.decoder.num_hidden_layers    8
  qwen3-tts-tokenizer.decoder.num_quantizers      16
  qwen3-tts-tokenizer.decoder.num_semantic_quantizers 1
  qwen3-tts-tokenizer.decoder.rope_theta         10000
  qwen3-tts-tokenizer.decoder.sliding_window      72
  qwen3-tts-tokenizer.decoder.upsampling_ratios  [2, 2]   (ConvNeXt upsample, DAC strides 8/5/4/3 are internal)
  qwen3-tts-tokenizer.decoder.layer_scale_initial_scale

  qwen3-tts-tokenizer.encoder.num_filters         64
  qwen3-tts-tokenizer.encoder.upsampling_ratios  [8, 6, 5, 4]   (SEANet, reversed at encode)
  qwen3-tts-tokenizer.encoder.hidden_size        512
  qwen3-tts-tokenizer.encoder.intermediate_size  2048
  qwen3-tts-tokenizer.encoder.num_attention_heads 8
  qwen3-tts-tokenizer.encoder.num_hidden_layers    8
  qwen3-tts-tokenizer.encoder.rope_theta         10000
  qwen3-tts-tokenizer.encoder.codebook_size      2048
  qwen3-tts-tokenizer.encoder.num_quantizers      16

tensors
  tok_enc.*                  SEANet conv stack, encoder transformer,
                             downsample conv, RVQ encode (vq_first / vq_rest)
  tok_dec.pre_conv.*         conv_pre into the decoder transformer
  tok_dec.pre_tfm.input_proj / output_proj / norm + transformer blocks
  tok_dec.upsample.*         ConvNeXt upsample (2 blocks, 4x)
  tok_dec.<dac>.*            DAC decoder chain
  tok_dec.vq_first.output_proj / vq_rest.output_proj
  tok_{enc,dec}.{vq_*}.<idx>.codebook            RVQ codebook entries

Component architecture

Talker LM

Standard Qwen3 decoder, KV cached. 28 layers, 16 query heads, 8 KV heads (GQA 2:1), head_dim 128, RoPE theta 1e6, per-head RMSNorm on Q and K before RoPE, SwiGLU MLP, no bias on Q/K/V/O/MLP, RMS eps 1e-6. The two sizes differ only in width : hidden 1024 / FFN 3072 on the 0.6B, hidden 2048 / FFN 6144 on the 1.7B. Context length 32768, text vocab 151936, text embedding width 2048.

Two input streams, pad-aligned and summed into one embedding sequence :

text  stream : text_proj(text_embd(text_ids))      text vocab -> hidden
codec stream : codec_embd(codec_ids)               3072 -> hidden
input        : text_stream + codec_stream          [T_ctx, hidden]

The reference multimodal RoPE carries three sections (mrope_section [24, 20, 20]), but a TTS prompt is a single text-plus-codec timeline, so the forward collapses the sections to a plain 1D NEOX rope.

The final hidden state is RMS-normalised through output_norm and projected through codec_head to the 3072-entry codebook 0 logits. The sampler masks the reserved top range [vocab - 1024, vocab) except the codec EOS before applying the sampling chain. There is no text lm_head on this path.

Code Predictor MTP head

A 5-layer Qwen3 stack, hidden 1024, heads 16/8, head_dim 128, FFN 3072, RoPE 1e6, context length 65536, with its own KV cache. The predictor hidden is 1024 on both talker sizes, so mtp_proj (which maps the talker hidden onto the predictor hidden) is an identity on the 0.6B (1024 == 1024) and a learned linear on the 1.7B (2048 -> 1024). The load log prints mtp_proj identity or mtp_proj linear accordingly.

input : talker_hidden_last [hidden] -- last position from the Talker, post final norm
        c0                  -- semantic code sampled from codec_head (codebook 0)
output: codes[16] = [c0, c1, ..., c15] -- full code set for one frame

The predictor cache is local to a single frame. It is reset every frame, prefilled with the first two positions (mtp_proj(talker_hidden) and embed(c0)), then decodes 14 single-token steps. That drops the inner work from O(sum_{g=0..14} (g+2)^2) to O(16) per frame, about 90x for the inner loop. Each of the 15 acoustic heads samples over its own 2048-entry codebook.

Speaker encoder (base checkpoints only)

An ECAPA-TDNN, present only in the base checkpoints, where it extracts a fixed x-vector from the reference clip for voice cloning. The custom_voice checkpoints carry no speaker encoder : their named speakers are precomputed codec-embedding rows. voice_design carries neither.

audio [T_pad] -> mel [128, T] -> conv0 TDNN k=5 + ReLU [512, T]
              -> SE-Res2Net dil 2 / 3 / 4
              -> cat blk[1..3] + MFA k=1 + ReLU [1536, T]
              -> ASP attentive pooling [3072, 1]
              -> FC k=1 [2048, 1] -> squeeze [2048]

The forward fuses mel extraction so the whole speaker path is one graph. Tensors live under spk_enc.* (conv0, blk.N, mfa, asp.tdnn, asp.conv, fc) and stay at F32 in every quant.

Audio tokenizer encoder

omnivoice-codec style round-trip, here qwen-codec. The encode path turns 24 kHz audio into 16 RVQ codes at 12.5 Hz :

audio 24 kHz mono
  -> SEANet : init Conv1d k=7 (1 -> 64), 4 stages ratios 4/5/6/8
             (cumulative 960x, 64 -> 512 ch), last Conv1d k=3
  -> encoder transformer : 8 layers, hidden 512, heads 8/8, head_dim 64,
     FFN 2048, RoPE 10000, LayerNorm with bias, plain GELU MLP, LayerScale,
     pure causal attention (no GQA, no sliding window applied)
  -> downsample conv k=4 stride=2 (512 -> 512), 25 Hz -> 12.5 Hz
  -> RVQ encode : input_proj 1x1 (512 -> 256), 16 codebooks of 2048 x 256,
     argmin over the residual, codebook 0 semantic + 1..15 acoustic
  -> codes [16, T] i32

Audio tokenizer decoder

The decode path is the inverse, bounded in VRAM by a chunked roll :

codes [T, 16] i32
  -> RVQ decode : F.embedding per codebook, per-split output_proj 1x1
     (256 -> 512), sum the semantic and acoustic splits -> hidden [T, 512]
  -> conv_pre -> decoder transformer : 8 layers, hidden 512, heads 16/16,
     head_dim 64, FFN 1024, RoPE 10000, RMSNorm, SwiGLU, sliding window 72
     causal, LayerScale, input_proj 1024 -> 512 / output_proj 512 -> 1024
  -> ConvNeXt upsample : 2 blocks, each CausalTransConv k=2 stride=2 then a
     ConvNeXt block (depthwise causal conv k=7, pointwise 1024 -> 4096 -> 1024,
     LayerScale gamma); together 4x on the time axis at channels 1024
  -> DAC : conv_pre k=7 (1024 -> 1536), 4 blocks strides 8/5/4/3
     channels 1536 -> 768 -> 384 -> 192 -> 96, each block SnakeBeta then
     CausalTransConv then 3 ResUnits (dilations 1/3/9), snake_post,
     conv_post k=7 (96 -> 1)
  -> audio [T * 1920, 1] @ 24 kHz mono

SnakeBeta applies exp() to alpha and beta on every forward in the reference ; both factors are precomputed CPU-side at load time so the graph multiplies plain F32 buffers. The whole DAC pipeline runs T-first (ne[0] = T, ne[1] = C) so the fused SNAKE op and ggml_conv_1d share one layout.

Chunked decode (buffered path)

A standalone codec decode of an isolated window shows edge artefacts at the chunk boundary, because the causal conv kernels and the sliding window attention have no left context. codec_chunked_decode prepends left_ctx_frames worth of previously decoded frames, decodes, then strips the samples that belong to the left context. The chunk width comes from qt_init_params.codec_chunk_sec (default 24.0, 300 frames at 12.5 Hz) and resolves to a frame count once at qt_init. The left context is not a caller knob : it derives from the decoder's own sliding window, 2 x sliding_window (144 frames on this codec), which is where the chunk output reaches the residual floor of the split. A shorter context leaves the decode audibly off, a longer one redecodes frames for nothing. The first chunk collapses its left context to whatever is available. This routine serves the buffered one-shot decode only.

A chunk covering the whole utterance decodes in a single pass and is bit exact against pipeline_codec_decode. Any split leaves a residual around -50 dB that no amount of left context removes, growing slowly with the pass count, so the chunk is a memory knob and not a quality one.

Streaming decode (stateful path)

Every module in the decoder is causal, so a stateful decode of one frame at a time reproduces the offline full decode exactly, with no re-decoded context and no chunk seams. Each stride-1 causal conv keeps its left context ((k-1)*d input rows) in a persistent backend tensor that the graph concats ahead of the fresh rows and refreshes in graph; each DAC transposed conv carries its col2im overlap tail (kernel - stride rows, bias free) into the next frame; the decoder transformer attends over a 128-slot sliding window KV ring written through set_rows, with the ring slot and the absolute RoPE position carried as input data. All state clears to zero at reset, which matches the offline zero left pads bit for bit.

The whole T=1 frame graph builds and allocates once (pipeline_codec_stream_ensure, lazy) and computes directly on the backend without the scheduler : a frame decode is four input uploads (16 codes, position, ring slot, window mask), one graph compute, and a 1920-sample readback. The constant topology and tensor addresses keep the CUDA graph cache in pure replay. ICL cloning primes the state by running the full reference codes through the same path with the readback skipped, matching the upstream reference-plus-generated decode; the transformer receptive field (8 layers x window 72) exceeds any reference length, so the full prime is the exact one.

The primed state feeds a per-reference snapshot LRU (CODEC_SNAP_SLOTS): after a fresh prime the conv contexts, KV ring, and position copy device to device into a slot keyed by the FNV-1a hash of the reference codes, and a later request with the same reference restores the exact state in one pass of tensor copies instead of re-decoding every reference frame. Slots allocate lazily and evict least recently used, so the priming cost amortizes across repeated cloned-voice requests without any registry coupling.

Inference pipeline

Prompt assembly

The talker prefix mirrors the upstream generate(). Two pad-aligned streams (text and codec) are summed at single-vector granularity on the CPU using the mmapped weight blocks, no backend allocation :

role          text(input_id[0:3])                          3 vecs
prefill_lhs   tts_pad x4 + tts_bos
              + codec_emb([think, think_bos, lang_id, think_eos, codec_pad])
trailing_lhs  text(input_id[3:-5]) + tts_eos
              + codec_emb([codec_pad x (N_text + 1)])
trailing_rhs  tts_pad + codec_emb([codec_bos])             1 vec

custom_voice inserts the speaker codec-embedding row between think_eos and codec_pad. voice_design and custom_voice may prepend an instruct segment built from text_proj(text_embd(<|im_start|>user\n{instruct}<|im_end|>\n)) before the role. Base voice cloning fills the speaker slot with the x-vector from the speaker encoder (mode A) or, when a reference transcript is supplied, builds an in-context prefix from the reference text and the reference codes (mode B, ICL).

Modes and the validation rules

The synthesis mode is read from the talker model_type at load, not from a CLI flag. qt_synthesize validates the params against that model_type before any compute and emits a verbatim qt_last_error(). The checks split across two return codes.

Five conditions mean the flag set does not match the loaded checkpoint family, returning QT_STATUS_MODE_INVALID :

--speaker  given but model_type != custom_voice
--instruct given but model_type == base
model_type == custom_voice but no --speaker
model_type == voice_design but --instruct empty or missing
--ref-wav  given but model_type != base

Two conditions mean the flag combination is self-contradictory whatever the model_type, returning QT_STATUS_INVALID_PARAMS :

--speaker and --ref-wav both given (mutually exclusive)
--ref-text given without --ref-wav

Frame loop

prefill the Talker on the prompt prefix                      writes T_ctx into talker_kv
for frame in 0..max_new_tokens-1 :
    poll cancel
    c0 = sample(codec_head(last logits))                     codebook 0, top-k/top-p
    codes[1..15] = code_predictor_step(hidden_bridge, c0)    reads the device bridge
    if c0 == codec_eos : break
    streaming : decode the frame through the stateful codec, emit 1920 samples
    talker_forward_decode(codes, overlay)                    gathers next_emb in graph
buffered : chunked codec decode of the gathered frames

The talker's last-position hidden never round-trips through the host on the hot path : the talker graph copies it into a persistent device tensor (the hidden bridge) that the code predictor prefill reads as a graph leaf. The only per-frame host traffic is the code ids and overlay row up, and the logits down for sampling.

Sampling matches the HuggingFace generate() chain in F32 : repetition_penalty -> temperature -> top_k -> top_p -> softmax -> multinomial, the uniform draw coming from philox_uniform_fill so a fixed seed replays byte for byte across runs.

Public API

Top-level public ABI : src/qwen.h

Single-header, plain C99, extern "C". The opaque qt_context handle aggregates the GGML backend pair, the Talker LM, the Code Predictor, the optional speaker encoder, the 12 Hz codec, the BPE tokenizer and the language / speaker tables. One init, one free, one synthesize call covers the full TTS path, consumable identically from C, C++, Python ctypes, Rust bindgen or Go cgo.

#include "qwen.h"

struct qt_init_params iparams;
qt_init_default_params(&iparams);
iparams.talker_path = "models/qwen-talker-1.7b-base-Q8_0.gguf";
iparams.codec_path  = "models/qwen-tokenizer-12hz-Q8_0.gguf";

struct qt_context * q = qt_init(&iparams);

struct qt_tts_params params;
qt_tts_default_params(&params);
params.text = "Hello world.";
params.lang = "English";

struct qt_audio audio = { 0 };
enum qt_status rc = qt_synthesize(q, &params, &audio);
if (rc == QT_STATUS_OK) {
    /* audio.samples : malloc'd mono float PCM, audio.n_samples,
       audio.sample_rate = 24000, audio.channels = 1 */
}
qt_audio_free(&audio);
qt_free(q);

Status codes :

QT_STATUS_OK               0
QT_STATUS_INVALID_PARAMS  -1
QT_STATUS_MODE_INVALID    -2   (the seven mode rules)
QT_STATUS_GENERATE_FAILED -3
QT_STATUS_OOM             -4
QT_STATUS_CANCELLED       -5

qt_tts_params exposes cancel (polled at the top of every Talker decode step, ~83 ms granularity) and on_chunk. With on_chunk set, synthesis runs in streaming mode : every generated frame emits its 1920 samples immediately through the stateful codec and out stays empty on success. qt_init_params.codec_chunk_sec drives the chunk framing of the buffered path only; the streaming path ignores it.

QT_ABI_VERSION and QT_ABI_MIN_VERSION bound the struct layouts this build addresses : callers set abi_version (or let the default-params helpers do it) and the lib rejects anything outside that closed range, a struct laid out for a newer header as well as one whose fields sit at offsets this build no longer maps. qt_version() returns the git short hash and commit date.

Low-level API : src/pipeline-tts.h, src/pipeline-codec.h

Direct access to pipeline_tts_load / pipeline_tts_synthesize, pipeline_codec_encode / pipeline_codec_decode, pipeline_codec_stream_reset / pipeline_codec_decode_stream, codec_chunked_decode, and the talker / predictor forwards. Used by the qwen-codec round-trip and the Python cossim harness through dump files. C++ types in the signatures, not part of the public ABI.

ABI guarantee

tests/abi-c.c is built on every build as the test-abi-c target with -std=c99 -Wall -Werror -pedantic. It includes the public header, calls every entry through its early-return path, and never loads a model. Any regression that breaks plain C consumability fails the main build.

The static libqwen-core.a is the default artefact and the one the CLI tools link. For binding consumers, configure with -DQWEN_SHARED=ON to add libqwen.so (or .dll / .dylib) exporting only the qt_* symbols ; every internal pipeline_* and backend_* stays hidden behind -fvisibility=hidden.

CLI tools

qwen-tts

Verbatim --help (the binary also prints a qwentts.cpp <hash> (<date>) banner line first) :

Usage: ./build/qwen-tts --model <gguf> --codec <gguf> [options] -o <out.wav> < text.txt

Required:
  --model <gguf>          Talker LM GGUF (qwen-talker-*.gguf)
  --codec <gguf>          Codec GGUF (qwen-tokenizer-*.gguf)
  -o <path>               Output WAV (24 kHz mono). '-' streams to stdout (pipe friendly).

Input:
  stdin                   Target text to synthesise. Read fully then synthesised in one
                          shot, or line by line with --stream-by-line.

Optional:
  --format <fmt>          WAV output format: wav16, wav24, wav32 (default: wav16)
  --lang <name>           Language label (default: auto)
  --instruct <str>        Style instruction. Required for VoiceDesign, optional for
                          CustomVoice, rejected for Base
  --speaker <name>        Speaker name (CustomVoice only)
  --ref-wav <path>        Reference WAV for voice cloning (Base only)
  --ref-spk <path>        Pre-extracted speaker embedding from qwen-codec --talker
                          (replaces --ref-wav, Base only)
  --ref-rvq <path>        Pre-encoded reference codes from qwen-codec (requires
                          --ref-spk and --ref-text, enables ICL clone mode)
  --ref-text <path>       Transcript file for the reference (enables ICL clone mode)
  --max-new <n>           Max new audio frames (default: 2048)
  --codec-chunk-dur <f>   Codec decode chunk duration in seconds (default: 24.0)
  --stream-by-line        Flush synthesis at each newline, one WAV header per line (-o '-')

Sampling:
  --seed <int>            Sampling seed (default: -1 for random)
  --greedy                Disable stochastic sampling on both stacks
  --temp <f>              Talker temperature (default: 0.9)
  --top-k <n>             Talker top-k (default: 50, 0 disables)
  --top-p <f>             Talker top-p (default: 1.0)
  --rep-pen <f>           Talker repetition penalty (default: 1.05)
  --sub-temp <f>          Sub-talker temperature (default: 0.9)
  --sub-top-k <n>         Sub-talker top-k (default: 50)
  --sub-top-p <f>         Sub-talker top-p (default: 1.0)

Debug:
  --no-fa                 Disable flash attention
  --clamp-fp16            Clamp hidden states to FP16 range
  --dump <dir>            Dump intermediate tensors (f32) to <dir>

qwen-codec

Verbatim --help :

Usage: ./build/qwen-codec --model <gguf> [-i <input>] [--talker <gguf>] [--format <fmt>]

Required:
  --model <gguf>          Codec GGUF (qwen-tokenizer-12hz-*.gguf)

Optional:
  -i <path>               Input. WAV -> encode, .rvq -> decode
  --talker <gguf>         Talker GGUF (Base only). Encode also extracts the speaker
                          embedding and writes it next to the .rvq as a .spk file
  --format <fmt>          WAV output format: wav16, wav24, wav32 (default: wav16)

Output is auto-named next to input : clip.wav -> clip.rvq, clip.rvq -> clip.wav.
Encode truncates to the hop boundary, conforming to the qwen-tts --ref-wav path:
the .rvq feeds qwen-tts --ref-rvq, the .spk feeds qwen-tts --ref-spk.
When -i is omitted, runs a load self-test of the codec GGUF.

The .rvq container packs the 16 codes per frame at 11 bits LSB-first.

tts-server

OpenAI-compatible HTTP server over the public ABI, one GPU-resident context, synthesis serialized FIFO across connections. The shared HTTP core lives in src/tts-server.h (also consumed by the sibling *.cpp ports); tools/tts-server.cpp wires the qt_* ABI into it. response_format selects the codec path : pcm drives the stateful streaming decode frame by frame for lowest latency, wav runs the buffered chunked decode (batch codec, talker uninterrupted) for best throughput. Verbatim --help :

Usage: ./build/tts-server --model <gguf> --codec <gguf> [options]

Required:
  --model <gguf>          Talker LM GGUF (qwen-talker-*.gguf)
  --codec <gguf>          Codec GGUF (qwen-tokenizer-*.gguf)

Optional:
  --alias <name>          Report this model id instead of the GGUF file name
  --host <ip>             Listen address (default: 127.0.0.1)
  --port <n>              Listen port (default: 8080)
  --lang <n>              Language label (default: auto)
  --no-fa                 Disable flash attention
  --clamp-fp16            Clamp hidden states to FP16 range

Endpoints :

POST   /v1/audio/speech         OAI text-to-speech; response_format "pcm"
                                streams s16le 24 kHz mono chunked as it is
                                generated, "wav" returns a one-shot RIFF file.
                                Optional sampling overrides ride in the same
                                body: seed, max_new_tokens, temperature,
                                top_k, top_p, repetition_penalty. Unset
                                fields keep the engine defaults, temperature
                                0 selects greedy decoding, the subtalker
                                mirrors the talker knobs
GET    /v1/models               single loaded model, using --alias when set
GET    /v1/audio/voices         model speakers plus registered cloned voices
POST   /v1/audio/voices         register a cloned voice: {name, ref_text,
                                wav_b64} extracts server side through
                                qt_extract_voice_ref, {name, ref_text,
                                spk_b64, rvq_b64} takes the pre-extracted
                                latents verbatim
DELETE /v1/audio/voices/{name}  drop a registered voice
GET    /health                  liveness probe

A registered voice wins over a model speaker of the same name and injects the reference latents into qt_tts_params : ref_text present selects ICL clone mode, absent selects the x-vector-only mode. The registry lives in process RAM and every access shares the synthesis mutex, so registration (which runs the extraction on the GPU) and lookups never race a running synthesis.

Module map

src/
  backend.h                 GGML backend init, scheduler factory, env override
  weight-ctx.h              Generic weight context for GGUF loaders
  gguf-weights.h            mmap GGUF, gf_load_tensor, gf_load_conv_f16, gf_get_*
  kv-cache.h                Persistent per-layer KV cache, fixed max len, rewind reset
  audio-io.h / wav.h        WAV read, mono write (S16 / S24 / F32)
  audio-resample.h          Kaiser polyphase resampler
  audio-mel.h               Mel spectrogram for the speaker encoder
  philox.h                  Philox4x32-10 counter-based PRNG
  sampling.h                Talker / CodePredictor sampling chain
  debug.h                   Tensor dumper for cossim tests
  bpe.h                     Qwen2 byte-level BPE, GGUF loader

  talker-weights.h          Talker GGUF weights
  talker-forward.h          Talker prefill + decode forwards, KV cached
  code-predictor-weights.h  Code Predictor MTP weights
  code-predictor-forward.h  Per-frame 16-code expansion, cache reset per frame
  speaker-encoder-weights.h ECAPA-TDNN weights (base only)
  speaker-encoder-forward.h ECAPA-TDNN forward, fused mel
  speaker-encoder-extract.h x-vector extraction entry

  seanet-encoder.h          SEANet conv stack (4 stages, 960x)
  encoder-transformer.h     8-layer Mimi-style encoder transformer
  encoder-downsample.h      25 Hz -> 12.5 Hz downsample conv
  quantizer-encode.h        RVQ encode (16 codebooks, split semantic/acoustic)
  quantizer-decode.h        RVQ decode, per-split output_proj
  tokenizer-transformer.h   8-layer local-causal decoder transformer (sw 72), KV ring stream variant
  convnext-block.h          ConvNeXt upsample stage (2 blocks, 4x), depthwise stream states
  causal-trans-conv.h       Causal Conv1d / ConvTranspose1d, offline and stateful stream variants
  dac-decoder-v2.h          DAC decoder (Descript Audio Codec; strides 8/5/4/3, SnakeBeta), stream states
  codec-chunked-decode.h    Buffered chunked decode plus the stateful frame-by-frame stream decoder
  rvq-file.h                Packed .rvq code stream IO, file and buffer readers

  prompt-builder.h          Talker prefix assembly, modes, ICL geometry
  pipeline-codec.{h,cpp}    Audio tokenizer end-to-end, persistent stream state, static frame graph
  pipeline-tts.{h,cpp}      Full TTS orchestration, prefill, frame loop, decode
  tts-server.h              Shared OAI HTTP core : routes, parsing, voice registry hooks
  qwen.{h,cpp}              Public ABI : opaque qt_context, plain C99 header

tools/
  qwen-tts.cpp              CLI : text to WAV
  qwen-codec.cpp            CLI : codes <-> WAV
  tts-server.cpp            OAI HTTP server : qt_* adapter, cloned voice registry
  quantize.cpp              GGUF requantizer with the codec-aware policy
  version.cmake             Embeds the git short hash into the binary

tests/
  debug-{tts,base,clone,customvoice}-cossim.py   Per-stage cossim vs PyTorch
  cossim_common.py          Shared comparison helpers
  abi-c.c                   Plain C99 smoke test for the public ABI

GGML conventions

Tensor shape and layout

PyTorch (out, in) for a Linear stores as ggml ne[0]=in, ne[1]=out. ggml_mul_mat(A, B) with A.ne[0]=K, A.ne[1]=M, B.ne[1]=N gives output (N, M), equal to A @ B^T in PyTorch terms.

The codec runs T-first : ne[0]=T, ne[1]=C. ggml_conv_1d and ggml_conv_1d_dw are T-first natively, the fused SNAKE op requires it, and the only mul_mat in the convtranspose primitive transposes internally.

For ConvTranspose1d weight (IC, OC, K), the convert-time permutation rearranges to ggml (IC, K*OC) with k varying faster than oc, so ggml_col2im_1d receives the correct column matrix. The fork folds the padding crop into the op via p0, removing a follow-up ggml_view.

Custom GGML ops

Provided by the ServeurpersoCom/ggml fork :

ggml_snake(ctx, x, a, inv_b) : y = x + sin^2(a*x) * inv_b. The SnakeBeta exp() on alpha and beta is folded CPU-side at load.

ggml_col2im_1d(ctx, a, s0, oc, p0) : scatter-add [K*OC, T_in] columns into [T_out, OC] with T_out = (T_in-1)*s0 + K - 2*p0. Used by every CausalTransConv in the ConvNeXt upsample and the DAC decoder.

Conv weight dtype

Conv kernels are cast to F16 at load by gf_load_conv_f16 (ARM im2col is strict on the kernel dtype), independent of the GGUF storage dtype.

Backend lifecycle

backend_init("MOD") then backend_sched_new(bp, max_nodes). Backend handles are shared across modules, refcounted. use_fa collapses to false on CPU-only backends. clamp_fp16 inserts ggml_clamp(-65504, 65504) on V before attention and on the residual stream between blocks to guard FP16 matmul accumulation on sub-Ampere CUDA targets.

Validation

The harness is tests/debug-{tts,base,clone,customvoice}-cossim.py. It runs the same input through the PyTorch reference (TF32 disabled, eager attention) and through the C++ binary, dumps each stage with --dump, and reports cosine similarity per stage. Latest run, voice_design 1.7B, Q8_0 on CUDA0, greedy :

Forward fidelity (single pass, the correctness signal)
  PromptIDs exact   100.00%
  Embed       cos   0.999988
  L0 .. L27   cos   >= 0.999893
  Final       cos   0.999401
  Logits      cos   0.999824
  NextEmbStep0 cos  0.999990

Free-running generation
  CodesFull exact   4.96%
  Audio       cos   0.095
  WAV stft    cos   0.360

Read this the way an autoregressive sampler demands. The forward graph matches the reference closely at every stage : embeddings, all 28 hidden taps, the final norm and the codebook 0 logits all sit at cosine 0.9994 and above. The large per-channel max-abs values at L27 are the usual pre-final-norm outlier channels ; cosine stays high because the direction is preserved.

The low CodesFull and Audio numbers are not a defect. A single argmax tie at the FP epsilon between the GGML and cuBLAS kernels flips one token, and because each frame conditions the next, the two runs walk different sampling trajectories from that point on. The MaskGIT path in omnivoice.cpp re-converges over its 32 refinement steps, so it can be checked bit for bit ; an AR sampler has no such contraction, and the two runs simply produce different but equally valid utterances of the same text (here 64 frames against the reference 63). End-to-end token or waveform equality is therefore the wrong metric. The meaningful check is the per-stage forward fidelity above, plus listening ; the --dump taps exist precisely to bisect that forward path stage by stage.

Glossary

MTP Multi-Token Prediction head. Here the Code Predictor that expands one semantic token into 15 acoustic codes per frame in a short cached inner loop.

RVQ Residual Vector Quantisation. Stack of codebooks, each quantising the residual of the previous reconstruction. 16 here : 1 semantic + 15 acoustic.

SEANet Convolutional audio encoder/decoder backbone (Mimi style), strided convs for down/up sampling.

DAC Descript Audio Codec. Convolutional decoder over the quantised latent, here with SnakeBeta activations.

SnakeBeta Periodic activation x + (1/beta) * sin^2(alpha*x) with exp() applied to alpha and beta, folded at load.

ConvNeXt Depthwise conv plus pointwise MLP block with a LayerScale residual, used here for the 4x temporal upsample.

ECAPA-TDNN Time-delay speaker embedding network with SE-Res2Net blocks and attentive statistics pooling. Base checkpoints only.

ASP Attentive Statistics Pooling. Mean and std over time weighted by a learned attention, the ECAPA pooling head.

GQA Grouped Query Attention. Fewer KV heads than query heads (16/8 on the Talker, 2:1).

mrope Multimodal RoPE with per-section position axes. Collapsed to 1D NEOX for the single TTS timeline.

ICL In-Context Learning. Voice clone mode B : the reference text and codes prefix the prompt so the model continues the speaker.

Philox Counter-based PRNG used by PyTorch CUDA. Skip-ahead friendly, aligns the multinomial draw across runs.