With -o '-', stdin is read line by line and every line synthesises
immediately as its own utterance, model and speaker staying resident
across lines. Each utterance after the first opens with a fresh RIFF
header, armed at end of line and consumed lazily at the next audio,
so a client can split the stream into standalone WAV clips on the
RIFF magic. Port of the feature contributed to omnivoice.cpp in
ServeurpersoCom/omnivoice.cpp#11.
Co-authored-by: Jeffrey van Binsbergen <comgenie@comgenie.com>
What we gain: lower TTFA (time to first audio), the latency before the
first frame is emitted. The win is entirely in the one shot prompt build;
per frame inference throughput (talker + code predictor) is unchanged.
How:
- Fuse projection: non ICL projects [instruct ; role] in one pass, ICL
projects [ref_text ; utterance] in one pass. Trailing utterance keeps
its own pass.
- ICL codec stream on the backend: the per frame per codebook host
embed_row_to_f32 + vec_add sum becomes num_code_groups ggml_get_rows
summed on the GPU in one graph (codebook 0 from talker.codec_embedding,
rest from code_predictor.codec_embedding, any quant via get_rows).
- Drop the host text projection (linear_f32, silu, read_tensor_f32,
text_projection_*, PromptTextProjection). tts_bos/eos/pad are projected
once on the backend in prompt_cache_load, now run after backend_sched_new.
No regression: direct prompt embed outputs match the host path within 2e-6
cosine on all 32 cells, bit exact in BF16/F32. The per frame AR loop is
untouched (ms/frame flat).
TTFA gain, CUDA0 RTX PRO 6000, greedy, 64 frames (old -> new):
mode/quant BF16 F32 Q8_0 Q4_K_M
base -18% ~flat -27% -37%
clone -36% -7% -26% -41%
customvoice +11% -1% -26% -32%
tts -25% -11% -24% +5%
Add steady_clock Timer (backend agnostic) and emit [Perf] lines per
synthesis stage: prompt build, prefill, TTFA, talker decode, code
predictor, host compose, codec decode, total with RTF. omnivoice logs
generate, codec decode and total for the one chunk path. Spans end on a
device readback so GPU work is covered, no cudaEvent dependency.
- Add sycl to the backend link loop so ggml-sycl links when built statically
- Add -fsycl link option for consumer executables (ggml-sycl links the
SYCL runtime PRIVATE, so consumers need it on their own link line)
- Gate _FORTIFY_SOURCE=2 with NOT GGML_SYCL: the fortified __memcpy_chk
symbol is unresolvable in SPIR-V device code and aborts kernel compilation
- Add buildsycl.sh convenience script (icx/icpx compilers)
Tested on Intel Arc A310 (DG2) with oneAPI 2026.0 + Level Zero: full TTS
generation produces correct audio output on the GPU.
snake.comp is renamed to follow the ggml DATA_A_* / A_TYPE convention.
A_TYPE now applies to the activation tensor data_a instead of the
broadcast multiplier, and the bindings become data_a (A_TYPE), data_b
(float), data_c (float) and data_d (D_TYPE). A header at the top of
the shader maps each buffer to its role in y = x + sin(b * x)^2 * c.
On the C++ side, ggml_vk_can_fuse_snake reuses the existing snake_pattern
constant instead of duplicating the op list, sin_node is extracted as a
named local alongside the other chain nodes, and the broadcast operands
a and inv_b are now required to be GGML_TYPE_F32 to match the hardcoded
float bindings on data_b and data_c (the previous a->type == x->type
would silently reject any future BF16 or F16 chain once the supports_op
gate for SIN / SQR is lifted). ggml_vk_snake_dispatch_fused gets an
explicit GGML_TYPE_F32 case and GGML_ABORT on default in place of the
silent f32 fallback, and a stale comment about data_a[i1] / data_inv_b[i1]
is refreshed to match the new binding names.
Clone mode used to read the same WAV twice (speaker encoder then codec
encoder), which failed silently on Windows where pipeline_codec_encode
returned empty on the second pass. PipelineTTSSynthesizeParams now takes
ref_audio_24k + ref_n_samples instead of a path. The CLI calls
audio_read_mono once, holds the buffer in a unique_ptr until synthesis
returns, and feeds the same pointer to both encoders.
GGUF norm matches llama.cpp policy: F32 master stays F32, BF16
variant keeps source BF16, K-quants fall back to F16 when kernel
rows do not align. No conv override in pick_type.
Conv kernels widen to F16 at load through gf_load_conv (12 sites).
qwen_load_ctw_f32 accepts BF16 source.
TODO upstream GGML: ggml_conv_1d and ggml_conv_1d_dw force F16 on
their im2col output, while conv_2d picks the kernel dtype. This
crashes F32 and BF16 kernels on CPU (im2col only handles F16) and
BF16 on Vulkan (mul_mat refuses BF16 on the operand the kernel
ends up on). Aligning conv_1d on conv_2d removes the workaround.
Adds a per-layer K/V ring (kv-cache.h) backed by a dedicated backend
buffer, sized at init for max_seq_len positions. The talker holds a
4096 position cache (896 MB f32) for the LM context, the code
predictor a 16 position cache (~80 KB) reset every frame.
talker_forward splits into prefill (resets the cache and writes T
positions in one shot) and decode (appends one position, reads the
[0, cur_len+1) window). code_predictor_step does the same with a T=2
prefill plus 14 single token decodes.
Bit identical audio output, validated by sha256 against the pre KV
cache run on a 64 frame F32 reference seed=42. Walltime drops ~10%
on a single utterance ; the win scales with sequence length and
unlocks frame by frame streaming.
PascalandJeffrey van Binsbergen
Andres Marafioti
smashingtags