Same gap as omnivoice.cpp: the UTF-8 boundary covers argv, console
output, and fopen but not the bytes arriving from stdin or text files.
Windows shells and editors hand those over as UTF-16 with BOM
(PowerShell redirection, Notepad, Out-File) or the ANSI codepage (cmd
pipes), and the raw bytes reach the tokenizer as garbage.
utf8_normalize() closes the gap: UTF-8 BOM stripped on every platform,
UTF-16 BOM losslessly recoded to UTF-8, bytes failing UTF-8 validation
decoded from the ANSI codepage. Valid UTF-8 passes through untouched.
Wired into read_stdin_text (binary mode stdin so CRLF translation
cannot eat UTF-16 0x0D bytes) and read_text_file, which also moves
from raw fopen to utf8_fopen so a non-ASCII --ref-text path opens.
qwen-codec --talker extracts the speaker embedding (.spk, raw f32)
and the ICL codes (.rvq) in one pass, encode truncated to the hop
boundary conforming to the --ref-wav path. qwen-tts loads them via
--ref-spk / --ref-rvq and skips the speaker encoder and codec encode
on every synthesis: TTFA 205 ms -> 89 ms. Extends qt_tts_params with
ABI v2 latent fields, adds qt_num_codebooks(), ships freeman.spk +
freeman.rvq and switches clone scripts to the latent path. Output is
bit-identical to the raw path at fixed seed.
The talker path already resets the scheduler before allocating its graph
(thanks Andi Marafioti, #4 of qwentts.cpp). The same shared scheduler is
also allocated elsewhere without a reset first.
The scheduler keeps split and tensor->backend assignments from the
previous graph. Allocating a different graph topology on a dirty
scheduler can reuse stale assignments, which is exactly the CPU
divergence Andi fixed for the talker. Resetting before alloc_graph is
the canonical GGML contract and is idempotent when the scheduler is
already clean.
Suggested-by: Andres Marafioti <andimarafioti@gmail.com>
With -o '-', stdin is read line by line and every line synthesises
immediately as its own utterance, model and speaker staying resident
across lines. Each utterance after the first opens with a fresh RIFF
header, armed at end of line and consumed lazily at the next audio,
so a client can split the stream into standalone WAV clips on the
RIFF magic. Port of the feature contributed to omnivoice.cpp in
ServeurpersoCom/omnivoice.cpp#11.
Co-authored-by: Jeffrey van Binsbergen <comgenie@comgenie.com>
What we gain: lower TTFA (time to first audio), the latency before the
first frame is emitted. The win is entirely in the one shot prompt build;
per frame inference throughput (talker + code predictor) is unchanged.
How:
- Fuse projection: non ICL projects [instruct ; role] in one pass, ICL
projects [ref_text ; utterance] in one pass. Trailing utterance keeps
its own pass.
- ICL codec stream on the backend: the per frame per codebook host
embed_row_to_f32 + vec_add sum becomes num_code_groups ggml_get_rows
summed on the GPU in one graph (codebook 0 from talker.codec_embedding,
rest from code_predictor.codec_embedding, any quant via get_rows).
- Drop the host text projection (linear_f32, silu, read_tensor_f32,
text_projection_*, PromptTextProjection). tts_bos/eos/pad are projected
once on the backend in prompt_cache_load, now run after backend_sched_new.
No regression: direct prompt embed outputs match the host path within 2e-6
cosine on all 32 cells, bit exact in BF16/F32. The per frame AR loop is
untouched (ms/frame flat).
TTFA gain, CUDA0 RTX PRO 6000, greedy, 64 frames (old -> new):
mode/quant BF16 F32 Q8_0 Q4_K_M
base -18% ~flat -27% -37%
clone -36% -7% -26% -41%
customvoice +11% -1% -26% -32%
tts -25% -11% -24% +5%
Add steady_clock Timer (backend agnostic) and emit [Perf] lines per
synthesis stage: prompt build, prefill, TTFA, talker decode, code
predictor, host compose, codec decode, total with RTF. omnivoice logs
generate, codec decode and total for the one chunk path. Spans end on a
device readback so GPU work is covered, no cudaEvent dependency.
- Add sycl to the backend link loop so ggml-sycl links when built statically
- Add -fsycl link option for consumer executables (ggml-sycl links the
SYCL runtime PRIVATE, so consumers need it on their own link line)
- Gate _FORTIFY_SOURCE=2 with NOT GGML_SYCL: the fortified __memcpy_chk
symbol is unresolvable in SPIR-V device code and aborts kernel compilation
- Add buildsycl.sh convenience script (icx/icpx compilers)
Tested on Intel Arc A310 (DG2) with oneAPI 2026.0 + Level Zero: full TTS
generation produces correct audio output on the GPU.
snake.comp is renamed to follow the ggml DATA_A_* / A_TYPE convention.
A_TYPE now applies to the activation tensor data_a instead of the
broadcast multiplier, and the bindings become data_a (A_TYPE), data_b
(float), data_c (float) and data_d (D_TYPE). A header at the top of
the shader maps each buffer to its role in y = x + sin(b * x)^2 * c.
On the C++ side, ggml_vk_can_fuse_snake reuses the existing snake_pattern
constant instead of duplicating the op list, sin_node is extracted as a
named local alongside the other chain nodes, and the broadcast operands
a and inv_b are now required to be GGML_TYPE_F32 to match the hardcoded
float bindings on data_b and data_c (the previous a->type == x->type
would silently reject any future BF16 or F16 chain once the supports_op
gate for SIN / SQR is lifted). ggml_vk_snake_dispatch_fused gets an
explicit GGML_TYPE_F32 case and GGML_ABORT on default in place of the
silent f32 fallback, and a stale comment about data_a[i1] / data_inv_b[i1]
is refreshed to match the new binding names.
Pascal
Andres Marafioti
smashingtags