predictor: unroll the frame into one cgraph and sample in standard ops

One static frame graph per batch width replaces the per step chain:
prefill and the 15 acoustic steps run in a single backend compute.
This is the target architecture for the llama.cpp Qwen3-TTS port and
serves as its working GGML reference while under test.

Sampling is a plain op chain batched over slots: temperature, argsort
top_k (descending order is guaranteed on every backend, unlike top_k),
softmax, cumsum, cdf crossing against a per step philox uniform.
Greedy draws with u = 0 and lands on the argmax. Faster than the
fused sampling op under CUDA graph capture, greedy codes stay exact
against the Python reference on CPU, CUDA and Vulkan.

Opt in single slot latency mode (--codec-fused on qwen-tts and
tts-server, codec_fused in qt_init_params): the codec stream tail
joins the frame graph at T=1, codes read through a device view, one
80 ms chunk per compute with no host round trip.

Predictor 3.34 -> 3.11 ms/frame on CUDA, end to end -4%.
This commit is contained in:
Pascal
2026-07-30 22:53:58 +02:00
parent 35ebe5376b
commit 26dd8adbf0
11 changed files with 715 additions and 257 deletions
+2 -1
View File
@@ -225,6 +225,7 @@ void qt_init_default_params(struct qt_init_params * p) {
p->max_batch = 1;
p->codec_chunk_sec = QT_CODEC_CHUNK_SEC_DEFAULT;
p->codec_fused = false;
}
void qt_tts_default_params(struct qt_tts_params * p) {
@@ -362,7 +363,7 @@ struct qt_context * qt_init(const struct qt_init_params * params) {
}
if (!pipeline_tts_load(&q->pt, params->talker_path, params->codec_path, q->bp, params->use_fa,
params->clamp_fp16, max_batch, chunk_sec)) {
params->clamp_fp16, max_batch, chunk_sec, params->codec_fused)) {
qt_throw("qt_init: pipeline_tts_load failed for '%s' / '%s'", params->talker_path, params->codec_path);
}