One static frame graph per batch width replaces the per step chain:
prefill and the 15 acoustic steps run in a single backend compute.
This is the target architecture for the llama.cpp Qwen3-TTS port and
serves as its working GGML reference while under test.
Sampling is a plain op chain batched over slots: temperature, argsort
top_k (descending order is guaranteed on every backend, unlike top_k),
softmax, cumsum, cdf crossing against a per step philox uniform.
Greedy draws with u = 0 and lands on the argmax. Faster than the
fused sampling op under CUDA graph capture, greedy codes stay exact
against the Python reference on CPU, CUDA and Vulkan.
Opt in single slot latency mode (--codec-fused on qwen-tts and
tts-server, codec_fused in qt_init_params): the codec stream tail
joins the frame graph at T=1, codes read through a device view, one
80 ms chunk per compute with no host round trip.
Predictor 3.34 -> 3.11 ms/frame on CUDA, end to end -4%.
The left context of the buffered chunked decode is no longer a caller
knob: it derives from the codec's own sliding window (2x144 frames),
placing the default decode at the residual floor of the split.
codec_chunk_sec moves from qt_tts_params to qt_init_params, resolved
once to frames at load. The mid-struct removal bumps the ABI to a
closed range [QT_ABI_MIN_VERSION, QT_ABI_VERSION] = [4, 4]; the probe
asserts both bounds reject through the range check.
One codec stream state set per lane plus a staging set, static graphs
per (chunk class, lane count) decoding every streaming lane in one
compute. The engine drops the per slot decoders and ownership
save/load for a shared 1 -> 2 -> 4 -> 8 ramp: lanes accumulate and
flush together, an admit drains and restarts at width 1, a retirement
drains then compacts the lane span device side. ICL references prime
through the staging set into per set snapshots. Depthwise and
transposed convs fold lanes into the channel axis; the dense convs
rely on the ggml conv_1d batched layout fix (submodule bump).
The talker, code predictor, and tokenizer transformer built their MLPs
as two mul_mats followed by a separate silu and mul. Replacing the
silu + mul pair with the GGML_OP_GLU node keeps the gate and up
mul_mats adjacent to it, which is the exact pattern the CUDA backend
fusion pass matches (ggml_cuda_should_fuse_mul_mat): gate, up, and
activation collapse into one kernel on the GEMV decode shapes.
Applies to all five MLP sites: talker prefill and batched decode,
code predictor layer, tokenizer transformer offline and stream paths.
Validated bit-exact against HEAD on greedy and seed 42 (identical
WAVs, F32 master). Around +3 percent on the F32 AR frame; neutral on
correctness, two lines simpler per site.
The codec encode path allocated its graph without first resetting the
shared scheduler, unlike every sibling path (decode, prompt-builder,
speaker_encoder_extract). After a synthesis call leaves the codec sched
allocated, the next RVQ encode — e.g. registering a second cloned voice
after a generation — trips GGML_ASSERT(!sched->is_alloc) in
ggml_backend_sched_alloc_graph and crashes the server.
Reset the sched before alloc to match the other call sites.
Repro: register a voice (base model) -> generate speech -> register
another voice. Second register crashes without this fix.
backend_init returned a refcounted global BackendPair shared by every
context in the process. That cache is load bearing in acestep.cpp where
each module inits its own backend, but here the pipeline already shares
one BackendPair explicitly, so the cache never hit and only made
independent contexts collide: one CUDA VMM pool is a strict LIFO stack,
so two contexts interleaving alloc/free abort on the pool assert.
Each backend_init call now returns a fresh backend pair with its own
device context and memory pool, backend_release frees it directly, and
the one-time ggml_log_set + ggml_backend_load_all setup moves under a
magic static so concurrent context creation stays safe.
Single-context binaries (CLI, server) are bit-identical. Multi-context
embedders (python bindings running parallel pipelines) no longer crash.
Keep the talker causal mask in each static attention window graph and
reuse positions, KV rows and sliding attention mask buffers in each codec
stream graph class.
This removes the remaining host allocations from the talker and codec
streaming hot loops without changing graph execution.
Install a ggml log callback that collapses exact consecutive
duplicates and reports the total count when the run ends. The CUDA
graph capture logs one reused line per replay step, flooding stderr
and stalling a reader that blocks on a full pipe.
The stream graphs come in width classes T in {1, 2, 4, 8} sharing the
state and KV ring tensors, built lazily per class. The decoder ramps
1 -> 2 -> 4 -> 8 so the first frame keeps its latency while the steady
state interleaves the codec 8x less often and the batch amortizes the
kernel count; drain flushes the tail at EOS and the reference priming
runs in max width chunks.
The decode flavor builds one static graph per 256 step kv window,
lazily on the first step entering the span, and replays directly on
the backend: ids, overlay, positions, kv row, and mask re-upload each
step since n_past moves. The prefill keeps the dynamic arena and sched
path, and the frame id assembly leaves the shared core.
The 15 predictor flavors build and allocate once at load, positions,
kv rows, and mask baked as never freed graph outputs, and replay
directly on the backend. The prefill slices the last position before
lm_head so every flavor reads one logits row at offset zero. Replaces
the per step graph rebuild, sched allocation, and debug prints.
The speech body accepts seed, max_new_tokens, temperature, top_k,
top_p, and repetition_penalty. Unset fields keep the engine defaults,
a temperature of zero selects greedy decoding, and the subtalker
mirrors the talker knobs. A fixed seed makes a request reproducible.
seed_reference hashes the ICL reference codes and restores the conv
contexts, KV ring, and position from a per reference snapshot slot on
a repeat, saving the primed state device to device after a fresh
prime. The reference priming cost amortizes across repeated cloned
voice requests.
qt_extract_voice_ref now pays the speaker encoder weight load on its
first call, mirroring the qt_synthesize ref_audio path. The server
extraction endpoint works without a prior ref wav synthesis.
POST /v1/voices registers a voice from a WAV extracted server side
through qt_extract_voice_ref or from pre extracted .spk and .rvq
latents taken verbatim, DELETE drops it and GET lists it alongside the
model speakers. A registered voice wins over a speaker of the same
name and injects the reference latents into qt_tts_params, ref_text
present selects ICL clone mode. The registry lives in process RAM
under the synthesis mutex, so registration and lookups never race a
running synthesis. The audio and rvq readers gain buffer variants
factored from the file paths. The README and the architecture
document catch up on the streaming decode, the hidden bridge, and the
server endpoints.
Every causal conv carries its left context in a persistent backend
tensor and every transposed conv its overlap tail, so a T=1 frame
decode reproduces the offline full decode exactly with zero re decoded
context. The tokenizer transformer attends over a sliding window KV
ring written through set_rows. The frame graph builds and allocates
once, then every frame is input uploads, one direct backend compute,
and one readback. The quantizer conts each codebook id view so the
Vulkan get_rows path accepts the direct compute. Each generated frame
emits its samples immediately and ICL priming feeds the full reference
through the same state. The buffered path keeps the chunked decode and
both codec framing knobs now apply to it alone.
KV writes go through set_rows with the destination rows carried as data
and the code predictor steps get one arena per sub step, so every
decode graph keeps a fixed topology and fixed tensor addresses step
after step. The talker last hidden stays resident on device in a
persistent bridge tensor, written by the talker graph and concatenated
as a leaf by the predictor prefill. The hot loop uploads sixteen code
ids and one overlay row and reads back the logits alone, host hidden
readbacks survive only under the dump path. Fewer nodes, transfers and
syncs pay on every backend, and the stable topology lets the CUDA
backend replay its captured graphs without an update.
pipeline_codec_decode builds its graph in a persistent arena instead of
a fresh ggml context per call. Constant size streaming slices rebuild
every node at the same address with identical shapes, which trims host
side graph churn on every backend and lets the CUDA backend replay its
captured graph executable without an update. The encode path keeps its
per call context.
The upstream pipeline decodes reference plus generated codes then trims,
giving the first generated frames causal context from the reference; the
generated only decode started the vocoder from an empty state and
colored the clone onset. Seed both decode paths with the last
min(ref_T, left_ctx_frames) reference frames: the streaming decoder
takes them below its emit cursor so they are never emitted, the buffered
path prepends them and strips their samples. Raising
codec_left_context_sec past the reference duration reproduces the
upstream full reference decode exactly.
Load the codec encoder half (seanet, enc_transformer, enc_downsample,
qenc) and the speaker encoder lazily on their first real use: synthesis
from a pre extracted reference (--ref-spk --ref-rvq) now brings up only
the talker and the codec decoder, matching the load profile of a preset
voice server. Assemble the AR inputs on device: the talker decode graph
gathers and sums the 16 frame code embeddings plus the trailing text or
pad overlay via get_rows, and the code predictor gathers c0 and each
sampled sub code from its group table in graph. Per frame host traffic
drops from 16 gguf row reads plus a CPU sum plus 15 synchronous backend
readbacks to 16 code ids and one overlay row uploaded. The next-emb
parity dump reproduces the in graph composition on host under --dump
only, staying byte comparable against the Python hook.
Rebuild each forward into a persistent arena per graph shape class (one
for the talker, two for the code predictor prefill and step flavors that
alternate within a frame) so nodes keep stable addresses and the CUDA
graph cache replays its executable instead of reinstantiating. Pad the
talker attention window to 256 and fix the predictor window to the frame
cache size so decode shapes hold across steps, with the causal mask
carrying neg inf over the padded tail. Drop the per step ggml context
churn and the trailing sched resets: one talker step plus 15 predictor
micro steps per frame no longer pay a full build/alloc/free cycle each.
Same gap as omnivoice.cpp: the UTF-8 boundary covers argv, console
output, and fopen but not the bytes arriving from stdin or text files.
Windows shells and editors hand those over as UTF-16 with BOM
(PowerShell redirection, Notepad, Out-File) or the ANSI codepage (cmd
pipes), and the raw bytes reach the tokenizer as garbage.
utf8_normalize() closes the gap: UTF-8 BOM stripped on every platform,
UTF-16 BOM losslessly recoded to UTF-8, bytes failing UTF-8 validation
decoded from the ANSI codepage. Valid UTF-8 passes through untouched.
Wired into read_stdin_text (binary mode stdin so CRLF translation
cannot eat UTF-16 0x0D bytes) and read_text_file, which also moves
from raw fopen to utf8_fopen so a non-ASCII --ref-text path opens.
qwen-codec --talker extracts the speaker embedding (.spk, raw f32)
and the ICL codes (.rvq) in one pass, encode truncated to the hop
boundary conforming to the --ref-wav path. qwen-tts loads them via
--ref-spk / --ref-rvq and skips the speaker encoder and codec encode
on every synthesis: TTFA 205 ms -> 89 ms. Extends qt_tts_params with
ABI v2 latent fields, adds qt_num_codebooks(), ships freeman.spk +
freeman.rvq and switches clone scripts to the latent path. Output is
bit-identical to the raw path at fixed seed.
The talker path already resets the scheduler before allocating its graph
(thanks Andi Marafioti, #4 of qwentts.cpp). The same shared scheduler is
also allocated elsewhere without a reset first.
The scheduler keeps split and tensor->backend assignments from the
previous graph. Allocating a different graph topology on a dirty
scheduler can reuse stale assignments, which is exactly the CPU
divergence Andi fixed for the talker. Resetting before alloc_graph is
the canonical GGML contract and is idempotent when the scheduler is
already clean.
Suggested-by: Andres Marafioti <andimarafioti@gmail.com>
With -o '-', stdin is read line by line and every line synthesises
immediately as its own utterance, model and speaker staying resident
across lines. Each utterance after the first opens with a fresh RIFF
header, armed at end of line and consumed lazily at the next audio,
so a client can split the stream into standalone WAV clips on the
RIFF magic. Port of the feature contributed to omnivoice.cpp in
ServeurpersoCom/omnivoice.cpp#11.
Co-authored-by: Jeffrey van Binsbergen <comgenie@comgenie.com>
What we gain: lower TTFA (time to first audio), the latency before the
first frame is emitted. The win is entirely in the one shot prompt build;
per frame inference throughput (talker + code predictor) is unchanged.
How:
- Fuse projection: non ICL projects [instruct ; role] in one pass, ICL
projects [ref_text ; utterance] in one pass. Trailing utterance keeps
its own pass.
- ICL codec stream on the backend: the per frame per codebook host
embed_row_to_f32 + vec_add sum becomes num_code_groups ggml_get_rows
summed on the GPU in one graph (codebook 0 from talker.codec_embedding,
rest from code_predictor.codec_embedding, any quant via get_rows).
- Drop the host text projection (linear_f32, silu, read_tensor_f32,
text_projection_*, PromptTextProjection). tts_bos/eos/pad are projected
once on the backend in prompt_cache_load, now run after backend_sched_new.
No regression: direct prompt embed outputs match the host path within 2e-6
cosine on all 32 cells, bit exact in BF16/F32. The per frame AR loop is
untouched (ms/frame flat).
TTFA gain, CUDA0 RTX PRO 6000, greedy, 64 frames (old -> new):
mode/quant BF16 F32 Q8_0 Q4_K_M
base -18% ~flat -27% -37%
clone -36% -7% -26% -41%
customvoice +11% -1% -26% -32%
tts -25% -11% -24% +5%
Add steady_clock Timer (backend agnostic) and emit [Perf] lines per
synthesis stage: prompt build, prefill, TTFA, talker decode, code
predictor, host compose, codec decode, total with RTF. omnivoice logs
generate, codec decode and total for the one chunk path. Spans end on a
device readback so GPU work is covered, no cudaEvent dependency.
Pascal
karlandPascal
RaduAdumitroaeiandPascal
Andres Marafioti
