predictor: unroll the frame into one cgraph and sample in standard ops
One static frame graph per batch width replaces the per step chain: prefill and the 15 acoustic steps run in a single backend compute. This is the target architecture for the llama.cpp Qwen3-TTS port and serves as its working GGML reference while under test. Sampling is a plain op chain batched over slots: temperature, argsort top_k (descending order is guaranteed on every backend, unlike top_k), softmax, cumsum, cdf crossing against a per step philox uniform. Greedy draws with u = 0 and lands on the argmax. Faster than the fused sampling op under CUDA graph capture, greedy codes stay exact against the Python reference on CPU, CUDA and Vulkan. Opt in single slot latency mode (--codec-fused on qwen-tts and tts-server, codec_fused in qt_init_params): the codec stream tail joins the frame graph at T=1, codes read through a device view, one 80 ms chunk per compute with no host round trip. Predictor 3.34 -> 3.11 ms/frame on CUDA, end to end -4%.
This commit is contained in:
+9
-1
@@ -157,11 +157,19 @@ struct qt_init_params {
|
||||
// frames at 12.5 Hz). The streaming path frames its own chunks
|
||||
// through the persistent codec stream state and reads none of this.
|
||||
float codec_chunk_sec;
|
||||
|
||||
// Fuse the codec streaming tail into the predictor frame graph:
|
||||
// one compute per frame delivers its 80 ms audio chunk with no
|
||||
// host round trip between the predictor and the decode. Single
|
||||
// slot latency mode: requires max_batch 1 and a streaming
|
||||
// synthesis (on_chunk); throughput drops against the buffered
|
||||
// flush, which stays the default.
|
||||
bool codec_fused;
|
||||
};
|
||||
|
||||
// Initialise to the standard defaults: both paths NULL (caller must set
|
||||
// them before calling qt_init), use_fa true, clamp_fp16 false,
|
||||
// max_batch 1, codec_chunk_sec 24.0.
|
||||
// max_batch 1, codec_chunk_sec 24.0, codec_fused false.
|
||||
QT_API void qt_init_default_params(struct qt_init_params * p);
|
||||
|
||||
// Allocate every module described by params. Returns NULL on any
|
||||
|
||||
Reference in New Issue
Block a user