Prevenzione allucinazioni/loop infiniti (Qwen3-TTS autoregressivo):
- block_repeated_ngrams: maschera i token che ricreerebbero un n-gram gia visto (n=4)
- has_repeating_cycle: ferma la generazione su cicli periodici (periodo 1-16, 4 ripetizioni)
- stuck detector: token dominante nella finestra recente (4 occorrenze in 8 token)
- fallback EOS quando tutti i logits sono mascherati (evita NaN)
- KV cache talker configurabile (--kv-cache, default 8192): 4096 overflowava con
reference lunghe + testi lunghi ("decode would overflow cache")
- parametri esposti via API (no_repeat_ngram_size, loop_max_period, loop_repeats,
loop_window) e CLI (--no-repeat-ngram, --loop-period, --loop-repeats, --loop-window)
- Docker: TTS_KV_CACHE env (default 8192)
Validato: testo 2788 char che prima degenerava in loop ora sintetizza pulito
(141s via API, nessuna ripetizione).
The fused mode appended the codec stream tail to the predictor frame
graph, so one compute produced both a frame's codes and its 80 ms of
audio with no host round trip. The experiment applied to max_batch 1
with a streaming synthesis only, it cost throughput against the
buffered flush that stays the default, and it kept a second frame
graph, its ring inputs and an init flag alive for that single case. It
is not worth keeping.
Remove the tail helpers, the fused graph of CodePredGraphSet, the
codec_fused field of qt_init_params, the --codec-fused flag of both
tools and the harness switch that exercised it. The predictor frame
unroll and the in graph sampler are untouched.
One static frame graph per batch width replaces the per step chain:
prefill and the 15 acoustic steps run in a single backend compute.
This is the target architecture for the llama.cpp Qwen3-TTS port and
serves as its working GGML reference while under test.
Sampling is a plain op chain batched over slots: temperature, argsort
top_k (descending order is guaranteed on every backend, unlike top_k),
softmax, cumsum, cdf crossing against a per step philox uniform.
Greedy draws with u = 0 and lands on the argmax. Faster than the
fused sampling op under CUDA graph capture, greedy codes stay exact
against the Python reference on CPU, CUDA and Vulkan.
Opt in single slot latency mode (--codec-fused on qwen-tts and
tts-server, codec_fused in qt_init_params): the codec stream tail
joins the frame graph at T=1, codes read through a device view, one
80 ms chunk per compute with no host round trip.
Predictor 3.34 -> 3.11 ms/frame on CUDA, end to end -4%.
The left context of the buffered chunked decode is no longer a caller
knob: it derives from the codec's own sliding window (2x144 frames),
placing the default decode at the residual floor of the split.
codec_chunk_sec moves from qt_tts_params to qt_init_params, resolved
once to frames at load. The mid-struct removal bumps the ABI to a
closed range [QT_ABI_MIN_VERSION, QT_ABI_VERSION] = [4, 4]; the probe
asserts both bounds reject through the range check.
The CLI tool already exposes these flags for controlling the vocoder's
chunked-decode window. tts-server always used the hardcoded 24.0s
chunk / 2.0s left-context defaults, with no way to override them at
the server binary's CLI, unlike qwen-tts.
This matters on memory-constrained GPUs: utterances shorter than the
chunk duration decode in a single pass, which can OOM on a small GPU
shared with other processes. Tightening these values (e.g. 4.0/1.5)
forces genuine chunked decode with bounded peak memory per chunk.
The 15 predictor flavors build and allocate once at load, positions,
kv rows, and mask baked as never freed graph outputs, and replay
directly on the backend. The prefill slices the last position before
lm_head so every flavor reads one logits row at offset zero. Replaces
the per step graph rebuild, sched allocation, and debug prints.
The speech body accepts seed, max_new_tokens, temperature, top_k,
top_p, and repetition_penalty. Unset fields keep the engine defaults,
a temperature of zero selects greedy decoding, and the subtalker
mirrors the talker knobs. A fixed seed makes a request reproducible.
POST /v1/voices registers a voice from a WAV extracted server side
through qt_extract_voice_ref or from pre extracted .spk and .rvq
latents taken verbatim, DELETE drops it and GET lists it alongside the
model speakers. A registered voice wins over a speaker of the same
name and injects the reference latents into qt_tts_params, ref_text
present selects ICL clone mode. The registry lives in process RAM
under the synthesis mutex, so registration and lookups never race a
running synthesis. The audio and rvq readers gain buffer variants
factored from the file paths. The README and the architecture
document catch up on the streaming decode, the hidden bridge, and the
server endpoints.
Same gap as omnivoice.cpp: the UTF-8 boundary covers argv, console
output, and fopen but not the bytes arriving from stdin or text files.
Windows shells and editors hand those over as UTF-16 with BOM
(PowerShell redirection, Notepad, Out-File) or the ANSI codepage (cmd
pipes), and the raw bytes reach the tokenizer as garbage.
utf8_normalize() closes the gap: UTF-8 BOM stripped on every platform,
UTF-16 BOM losslessly recoded to UTF-8, bytes failing UTF-8 validation
decoded from the ANSI codepage. Valid UTF-8 passes through untouched.
Wired into read_stdin_text (binary mode stdin so CRLF translation
cannot eat UTF-16 0x0D bytes) and read_text_file, which also moves
from raw fopen to utf8_fopen so a non-ASCII --ref-text path opens.
qwen-codec --talker extracts the speaker embedding (.spk, raw f32)
and the ICL codes (.rvq) in one pass, encode truncated to the hop
boundary conforming to the --ref-wav path. qwen-tts loads them via
--ref-spk / --ref-rvq and skips the speaker encoder and codec encode
on every synthesis: TTFA 205 ms -> 89 ms. Extends qt_tts_params with
ABI v2 latent fields, adds qt_num_codebooks(), ships freeman.spk +
freeman.rvq and switches clone scripts to the latent path. Output is
bit-identical to the raw path at fixed seed.
With -o '-', stdin is read line by line and every line synthesises
immediately as its own utterance, model and speaker staying resident
across lines. Each utterance after the first opens with a fresh RIFF
header, armed at end of line and consumed lazily at the next audio,
so a client can split the stream into standalone WAV clips on the
RIFF magic. Port of the feature contributed to omnivoice.cpp in
ServeurpersoCom/omnivoice.cpp#11.
Co-authored-by: Jeffrey van Binsbergen <comgenie@comgenie.com>
Clone mode used to read the same WAV twice (speaker encoder then codec
encoder), which failed silently on Windows where pipeline_codec_encode
returned empty on the second pass. PipelineTTSSynthesizeParams now takes
ref_audio_24k + ref_n_samples instead of a path. The CLI calls
audio_read_mono once, holds the buffer in a unique_ptr until synthesis
returns, and feeds the same pointer to both encoders.
GGUF norm matches llama.cpp policy: F32 master stays F32, BF16
variant keeps source BF16, K-quants fall back to F16 when kernel
rows do not align. No conv override in pick_type.
Conv kernels widen to F16 at load through gf_load_conv (12 sites).
qwen_load_ctw_f32 accepts BF16 source.
TODO upstream GGML: ggml_conv_1d and ggml_conv_1d_dw force F16 on
their im2col output, while conv_2d picks the kernel dtype. This
crashes F32 and BF16 kernels on CPU (im2col only handles F16) and
BF16 on Vulkan (mul_mat refuses BF16 on the operand the kernel
ends up on). Aligning conv_1d on conv_2d removes the workaround.
enne2
Pascal
Gary
karlandPascal
