get_rows.comp already applies all three misalign offsets; the assert
predates that. Apply the b and d offsets in the quant variant too.
Only reject a quantized src0 with a sub-block offset, which cannot
be expressed in elements.
One codec stream state set per lane plus a staging set, static graphs
per (chunk class, lane count) decoding every streaming lane in one
compute. The engine drops the per slot decoders and ownership
save/load for a shared 1 -> 2 -> 4 -> 8 ramp: lanes accumulate and
flush together, an admit drains and restarts at width 1, a retirement
drains then compacts the lane span device side. ICL references prime
through the staging set into per set snapshots. Depthwise and
transposed convs fold lanes into the channel axis; the dense convs
rely on the ggml conv_1d batched layout fix (submodule bump).
The talker, code predictor, and tokenizer transformer built their MLPs
as two mul_mats followed by a separate silu and mul. Replacing the
silu + mul pair with the GGML_OP_GLU node keeps the gate and up
mul_mats adjacent to it, which is the exact pattern the CUDA backend
fusion pass matches (ggml_cuda_should_fuse_mul_mat): gate, up, and
activation collapse into one kernel on the GEMV decode shapes.
Applies to all five MLP sites: talker prefill and batched decode,
code predictor layer, tokenizer transformer offline and stream paths.
Validated bit-exact against HEAD on greedy and seed 42 (identical
WAVs, F32 master). Around +3 percent on the F32 AR frame; neutral on
correctness, two lines simpler per site.
Device-side embedding and codebook lookups now cover q2_K to q6_K, so a
quantized token_embd no longer drops the graph out of the direct device
path. i-quants are left as a TODO.
backend_init returned a refcounted global BackendPair shared by every
context in the process. That cache is load bearing in acestep.cpp where
each module inits its own backend, but here the pipeline already shares
one BackendPair explicitly, so the cache never hit and only made
independent contexts collide: one CUDA VMM pool is a strict LIFO stack,
so two contexts interleaving alloc/free abort on the pool assert.
Each backend_init call now returns a fresh backend pair with its own
device context and memory pool, backend_release frees it directly, and
the one-time ggml_log_set + ggml_backend_load_all setup moves under a
magic static so concurrent context creation stays safe.
Single-context binaries (CLI, server) are bit-identical. Multi-context
embedders (python bindings running parallel pipelines) no longer crash.
Keep the talker causal mask in each static attention window graph and
reuse positions, KV rows and sliding attention mask buffers in each codec
stream graph class.
This removes the remaining host allocations from the talker and codec
streaming hot loops without changing graph execution.
Install a ggml log callback that collapses exact consecutive
duplicates and reports the total count when the run ends. The CUDA
graph capture logs one reused line per replay step, flooding stderr
and stalling a reader that blocks on a full pipe.
The stream graphs come in width classes T in {1, 2, 4, 8} sharing the
state and KV ring tensors, built lazily per class. The decoder ramps
1 -> 2 -> 4 -> 8 so the first frame keeps its latency while the steady
state interleaves the codec 8x less often and the batch amortizes the
kernel count; drain flushes the tail at EOS and the reference priming
runs in max width chunks.
The decode flavor builds one static graph per 256 step kv window,
lazily on the first step entering the span, and replays directly on
the backend: ids, overlay, positions, kv row, and mask re-upload each
step since n_past moves. The prefill keeps the dynamic arena and sched
path, and the frame id assembly leaves the shared core.
The 15 predictor flavors build and allocate once at load, positions,
kv rows, and mask baked as never freed graph outputs, and replay
directly on the backend. The prefill slices the last position before
lm_head so every flavor reads one logits row at offset zero. Replaces
the per step graph rebuild, sched allocation, and debug prints.
The speech body accepts seed, max_new_tokens, temperature, top_k,
top_p, and repetition_penalty. Unset fields keep the engine defaults,
a temperature of zero selects greedy decoding, and the subtalker
mirrors the talker knobs. A fixed seed makes a request reproducible.
seed_reference hashes the ICL reference codes and restores the conv
contexts, KV ring, and position from a per reference snapshot slot on
a repeat, saving the primed state device to device after a fresh
prime. The reference priming cost amortizes across repeated cloned
voice requests.
qt_extract_voice_ref now pays the speaker encoder weight load on its
first call, mirroring the qt_synthesize ref_audio path. The server
extraction endpoint works without a prior ref wav synthesis.
POST /v1/voices registers a voice from a WAV extracted server side
through qt_extract_voice_ref or from pre extracted .spk and .rvq
latents taken verbatim, DELETE drops it and GET lists it alongside the
model speakers. A registered voice wins over a speaker of the same
name and injects the reference latents into qt_tts_params, ref_text
present selects ICL clone mode. The registry lives in process RAM
under the synthesis mutex, so registration and lookups never race a
running synthesis. The audio and rvq readers gain buffer variants
factored from the file paths. The README and the architecture
document catch up on the streaming decode, the hidden bridge, and the
server endpoints.
Every causal conv carries its left context in a persistent backend
tensor and every transposed conv its overlap tail, so a T=1 frame
decode reproduces the offline full decode exactly with zero re decoded
context. The tokenizer transformer attends over a sliding window KV
ring written through set_rows. The frame graph builds and allocates
once, then every frame is input uploads, one direct backend compute,
and one readback. The quantizer conts each codebook id view so the
Vulkan get_rows path accepts the direct compute. Each generated frame
emits its samples immediately and ICL priming feeds the full reference
through the same state. The buffered path keeps the chunked decode and
both codec framing knobs now apply to it alone.
KV writes go through set_rows with the destination rows carried as data
and the code predictor steps get one arena per sub step, so every
decode graph keeps a fixed topology and fixed tensor addresses step
after step. The talker last hidden stays resident on device in a
persistent bridge tensor, written by the talker graph and concatenated
as a leaf by the predictor prefill. The hot loop uploads sixteen code
ids and one overlay row and reads back the logits alone, host hidden
readbacks survive only under the dump path. Fewer nodes, transfers and
syncs pay on every backend, and the stable topology lets the CUDA
backend replay its captured graphs without an update.
pipeline_codec_decode builds its graph in a persistent arena instead of
a fresh ggml context per call. Constant size streaming slices rebuild
every node at the same address with identical shapes, which trims host
side graph churn on every backend and lets the CUDA backend replay its
captured graph executable without an update. The encode path keeps its
per call context.
The upstream pipeline decodes reference plus generated codes then trims,
giving the first generated frames causal context from the reference; the
generated only decode started the vocoder from an empty state and
colored the clone onset. Seed both decode paths with the last
min(ref_T, left_ctx_frames) reference frames: the streaming decoder
takes them below its emit cursor so they are never emitted, the buffered
path prepends them and strips their samples. Raising
codec_left_context_sec past the reference duration reproduces the
upstream full reference decode exactly.
Load the codec encoder half (seanet, enc_transformer, enc_downsample,
qenc) and the speaker encoder lazily on their first real use: synthesis
from a pre extracted reference (--ref-spk --ref-rvq) now brings up only
the talker and the codec decoder, matching the load profile of a preset
voice server. Assemble the AR inputs on device: the talker decode graph
gathers and sums the 16 frame code embeddings plus the trailing text or
pad overlay via get_rows, and the code predictor gathers c0 and each
sampled sub code from its group table in graph. Per frame host traffic
drops from 16 gguf row reads plus a CPU sum plus 15 synchronous backend
readbacks to 16 code ids and one overlay row uploaded. The next-emb
parity dump reproduces the in graph composition on host under --dump
only, staying byte comparable against the Python hook.
Rebuild each forward into a persistent arena per graph shape class (one
for the talker, two for the code predictor prefill and step flavors that
alternate within a frame) so nodes keep stable addresses and the CUDA
graph cache replays its executable instead of reinstantiating. Pad the
talker attention window to 256 and fix the predictor window to the frame
cache size so decode shapes hold across steps, with the causal mask
carrying neg inf over the padded tail. Drop the per step ggml context
churn and the trailing sched resets: one talker step plus 15 predictor
micro steps per frame no longer pay a full build/alloc/free cycle each.
Standalone ggml ships CUDA graphs off. Capture/replay batches every
kernel launch of a graph into a single submission. Override with
-DGGML_CUDA_GRAPHS=OFF or at runtime with GGML_CUDA_DISABLE_GRAPHS=1
Same gap as omnivoice.cpp: the UTF-8 boundary covers argv, console
output, and fopen but not the bytes arriving from stdin or text files.
Windows shells and editors hand those over as UTF-16 with BOM
(PowerShell redirection, Notepad, Out-File) or the ANSI codepage (cmd
pipes), and the raw bytes reach the tokenizer as garbage.
utf8_normalize() closes the gap: UTF-8 BOM stripped on every platform,
UTF-16 BOM losslessly recoded to UTF-8, bytes failing UTF-8 validation
decoded from the ANSI codepage. Valid UTF-8 passes through untouched.
Wired into read_stdin_text (binary mode stdin so CRLF translation
cannot eat UTF-16 0x0D bytes) and read_text_file, which also moves
from raw fopen to utf8_fopen so a non-ASCII --ref-text path opens.
qwen-codec --talker extracts the speaker embedding (.spk, raw f32)
and the ICL codes (.rvq) in one pass, encode truncated to the hop
boundary conforming to the --ref-wav path. qwen-tts loads them via
--ref-spk / --ref-rvq and skips the speaker encoder and codec encode
on every synthesis: TTFA 205 ms -> 89 ms. Extends qt_tts_params with
ABI v2 latent fields, adds qt_num_codebooks(), ships freeman.spk +
freeman.rvq and switches clone scripts to the latent path. Output is
bit-identical to the raw path at fixed seed.
The talker path already resets the scheduler before allocating its graph
(thanks Andi Marafioti, #4 of qwentts.cpp). The same shared scheduler is
also allocated elsewhere without a reset first.
The scheduler keeps split and tensor->backend assignments from the
previous graph. Allocating a different graph topology on a dirty
scheduler can reuse stale assignments, which is exactly the CPU
divergence Andi fixed for the talker. Resetting before alloc_graph is
the canonical GGML contract and is idempotent when the scheduler is
already clean.
Suggested-by: Andres Marafioti <andimarafioti@gmail.com>
With -o '-', stdin is read line by line and every line synthesises
immediately as its own utterance, model and speaker staying resident
across lines. Each utterance after the first opens with a fresh RIFF
header, armed at end of line and consumed lazily at the next audio,
so a client can split the stream into standalone WAV clips on the
RIFF magic. Port of the feature contributed to omnivoice.cpp in
ServeurpersoCom/omnivoice.cpp#11.
Co-authored-by: Jeffrey van Binsbergen <comgenie@comgenie.com>
What we gain: lower TTFA (time to first audio), the latency before the
first frame is emitted. The win is entirely in the one shot prompt build;
per frame inference throughput (talker + code predictor) is unchanged.
How:
- Fuse projection: non ICL projects [instruct ; role] in one pass, ICL
projects [ref_text ; utterance] in one pass. Trailing utterance keeps
its own pass.
- ICL codec stream on the backend: the per frame per codebook host
embed_row_to_f32 + vec_add sum becomes num_code_groups ggml_get_rows
summed on the GPU in one graph (codebook 0 from talker.codec_embedding,
rest from code_predictor.codec_embedding, any quant via get_rows).
- Drop the host text projection (linear_f32, silu, read_tensor_f32,
text_projection_*, PromptTextProjection). tts_bos/eos/pad are projected
once on the backend in prompt_cache_load, now run after backend_sched_new.
No regression: direct prompt embed outputs match the host path within 2e-6
cosine on all 32 cells, bit exact in BF16/F32. The per frame AR loop is
untouched (ms/frame flat).
TTFA gain, CUDA0 RTX PRO 6000, greedy, 64 frames (old -> new):
mode/quant BF16 F32 Q8_0 Q4_K_M
base -18% ~flat -27% -37%
clone -36% -7% -26% -41%
customvoice +11% -1% -26% -32%
tts -25% -11% -24% +5%
Add steady_clock Timer (backend agnostic) and emit [Perf] lines per
synthesis stage: prompt build, prefill, TTFA, talker decode, code
predictor, host compose, codec decode, total with RTF. omnivoice logs
generate, codec decode and total for the one chunk path. Spans end on a
device readback so GPU work is covered, no cudaEvent dependency.
snake.comp is renamed to follow the ggml DATA_A_* / A_TYPE convention.
A_TYPE now applies to the activation tensor data_a instead of the
broadcast multiplier, and the bindings become data_a (A_TYPE), data_b
(float), data_c (float) and data_d (D_TYPE). A header at the top of
the shader maps each buffer to its role in y = x + sin(b * x)^2 * c.
On the C++ side, ggml_vk_can_fuse_snake reuses the existing snake_pattern
constant instead of duplicating the op list, sin_node is extracted as a
named local alongside the other chain nodes, and the broadcast operands
a and inv_b are now required to be GGML_TYPE_F32 to match the hardcoded
float bindings on data_b and data_c (the previous a->type == x->type
would silently reject any future BF16 or F16 chain once the supports_op
gate for SIN / SQR is lifted). ggml_vk_snake_dispatch_fused gets an
explicit GGML_TYPE_F32 case and GGML_ABORT on default in place of the
silent f32 fallback, and a stale comment about data_a[i1] / data_inv_b[i1]
is refreshed to match the new binding names.
Clone mode used to read the same WAV twice (speaker encoder then codec
encoder), which failed silently on Windows where pipeline_codec_encode
returned empty on the second pass. PipelineTTSSynthesizeParams now takes
ref_audio_24k + ref_n_samples instead of a path. The CLI calls
audio_read_mono once, holds the buffer in a unique_ptr until synthesis
returns, and feeds the same pointer to both encoders.
GGUF norm matches llama.cpp policy: F32 master stays F32, BF16
variant keeps source BF16, K-quants fall back to F16 when kernel
rows do not align. No conv override in pick_type.
Conv kernels widen to F16 at load through gf_load_conv (12 sites).
qwen_load_ctw_f32 accepts BF16 source.
TODO upstream GGML: ggml_conv_1d and ggml_conv_1d_dw force F16 on
their im2col output, while conv_2d picks the kernel dtype. This
crashes F32 and BF16 kernels on CPU (im2col only handles F16) and
BF16 on Vulkan (mul_mat refuses BF16 on the operand the kernel
ends up on). Aligning conv_1d on conv_2d removes the workaround.
Adds a per-layer K/V ring (kv-cache.h) backed by a dedicated backend
buffer, sized at init for max_seq_len positions. The talker holds a
4096 position cache (896 MB f32) for the LM context, the code
predictor a 16 position cache (~80 KB) reset every frame.
talker_forward splits into prefill (resets the cache and writes T
positions in one shot) and decode (appends one position, reads the
[0, cur_len+1) window). code_predictor_step does the same with a T=2
prefill plus 14 single token decodes.
Bit identical audio output, validated by sha256 against the pre KV
cache run on a 64 frame F32 reference seed=42. Walltime drops ~10%
on a single utterance ; the win scales with sequence length and
unlocks frame by frame streaming.
Pascal
