KV writes go through set_rows with the destination rows carried as data
and the code predictor steps get one arena per sub step, so every
decode graph keeps a fixed topology and fixed tensor addresses step
after step. The talker last hidden stays resident on device in a
persistent bridge tensor, written by the talker graph and concatenated
as a leaf by the predictor prefill. The hot loop uploads sixteen code
ids and one overlay row and reads back the logits alone, host hidden
readbacks survive only under the dump path. Fewer nodes, transfers and
syncs pay on every backend, and the stable topology lets the CUDA
backend replay its captured graphs without an update.
Load the codec encoder half (seanet, enc_transformer, enc_downsample,
qenc) and the speaker encoder lazily on their first real use: synthesis
from a pre extracted reference (--ref-spk --ref-rvq) now brings up only
the talker and the codec decoder, matching the load profile of a preset
voice server. Assemble the AR inputs on device: the talker decode graph
gathers and sums the 16 frame code embeddings plus the trailing text or
pad overlay via get_rows, and the code predictor gathers c0 and each
sampled sub code from its group table in graph. Per frame host traffic
drops from 16 gguf row reads plus a CPU sum plus 15 synchronous backend
readbacks to 16 code ids and one overlay row uploaded. The next-emb
parity dump reproduces the in graph composition on host under --dump
only, staying byte comparable against the Python hook.
Rebuild each forward into a persistent arena per graph shape class (one
for the talker, two for the code predictor prefill and step flavors that
alternate within a frame) so nodes keep stable addresses and the CUDA
graph cache replays its executable instead of reinstantiating. Pad the
talker attention window to 256 and fix the predictor window to the frame
cache size so decode shapes hold across steps, with the causal mask
carrying neg inf over the padded tail. Drop the per step ggml context
churn and the trailing sched resets: one talker step plus 15 predictor
micro steps per frame no longer pay a full build/alloc/free cycle each.
What we gain: lower TTFA (time to first audio), the latency before the
first frame is emitted. The win is entirely in the one shot prompt build;
per frame inference throughput (talker + code predictor) is unchanged.
How:
- Fuse projection: non ICL projects [instruct ; role] in one pass, ICL
projects [ref_text ; utterance] in one pass. Trailing utterance keeps
its own pass.
- ICL codec stream on the backend: the per frame per codebook host
embed_row_to_f32 + vec_add sum becomes num_code_groups ggml_get_rows
summed on the GPU in one graph (codebook 0 from talker.codec_embedding,
rest from code_predictor.codec_embedding, any quant via get_rows).
- Drop the host text projection (linear_f32, silu, read_tensor_f32,
text_projection_*, PromptTextProjection). tts_bos/eos/pad are projected
once on the backend in prompt_cache_load, now run after backend_sched_new.
No regression: direct prompt embed outputs match the host path within 2e-6
cosine on all 32 cells, bit exact in BF16/F32. The per frame AR loop is
untouched (ms/frame flat).
TTFA gain, CUDA0 RTX PRO 6000, greedy, 64 frames (old -> new):
mode/quant BF16 F32 Q8_0 Q4_K_M
base -18% ~flat -27% -37%
clone -36% -7% -26% -41%
customvoice +11% -1% -26% -32%
tts -25% -11% -24% +5%
Clone mode used to read the same WAV twice (speaker encoder then codec
encoder), which failed silently on Windows where pipeline_codec_encode
returned empty on the second pass. PipelineTTSSynthesizeParams now takes
ref_audio_24k + ref_n_samples instead of a path. The CLI calls
audio_read_mono once, holds the buffer in a unique_ptr until synthesis
returns, and feeds the same pointer to both encoders.
Adds a per-layer K/V ring (kv-cache.h) backed by a dedicated backend
buffer, sized at init for max_seq_len positions. The talker holds a
4096 position cache (896 MB f32) for the LM context, the code
predictor a 16 position cache (~80 KB) reset every frame.
talker_forward splits into prefill (resets the cache and writes T
positions in one shot) and decode (appends one position, reads the
[0, cur_len+1) window). code_predictor_step does the same with a T=2
prefill plus 14 single token decodes.
Bit identical audio output, validated by sha256 against the pre KV
cache run on a 64 frame F32 reference seed=42. Walltime drops ~10%
on a single utterance ; the win scales with sequence length and
unlocks frame by frame streaming.
Pascal
Andres Marafioti