Prevenzione allucinazioni/loop infiniti (Qwen3-TTS autoregressivo):
- block_repeated_ngrams: maschera i token che ricreerebbero un n-gram gia visto (n=4)
- has_repeating_cycle: ferma la generazione su cicli periodici (periodo 1-16, 4 ripetizioni)
- stuck detector: token dominante nella finestra recente (4 occorrenze in 8 token)
- fallback EOS quando tutti i logits sono mascherati (evita NaN)
- KV cache talker configurabile (--kv-cache, default 8192): 4096 overflowava con
reference lunghe + testi lunghi ("decode would overflow cache")
- parametri esposti via API (no_repeat_ngram_size, loop_max_period, loop_repeats,
loop_window) e CLI (--no-repeat-ngram, --loop-period, --loop-repeats, --loop-window)
- Docker: TTS_KV_CACHE env (default 8192)
Validato: testo 2788 char che prima degenerava in loop ora sintetizza pulito
(141s via API, nessuna ripetizione).
The CMakeLists.txt >=12.8 CUDA-architecture branch claimed 121a-real
(Blackwell Ultra) alongside 120a-real, but nvcc from CUDA 12.8.1
rejects compute_121 ("nvcc fatal: Unsupported gpu architecture
'compute_121'"). Confirmed against a real build: 12.8.1 compiles
cleanly with 120a-real but not 121a-real; 12.9.2 compiles both. Split
the >=12.8 branch into >=12.9 (full Blackwell + Ultra) and >=12.8
(Blackwell only, no Ultra).
Since no single CUDA toolkit spans the full arch range -- 12.9.x is
the newest 12.x that still emits Pascal (61-real) SASS, 13.x drops
Pascal entirely but is otherwise more current -- the single :cuda
image can no longer serve both audiences. Split it into two CI
variants, :cuda12 (12.9.2, Pascal through Blackwell Ultra) and
:cuda13 (13.3.1, Turing and newer). The Dockerfile's own --target
cuda stage is unchanged; the CUDA_BUILD_IMAGE/CUDA_RUNTIME_IMAGE ARGs
now default to 12.9.2 (was 12.4.1) and the CI matrix overrides them
per variant.
Both variants were build-tested against nvidia/cuda:12.8.1 and 12.9.2
locally and exercised with real end-to-end synthesis requests on an
actual sm_61 card (GTX 1070 Max-Q) -- RTF ~0.4 on both, no kernel
image / arch mismatch errors.
Co-authored-by: Gary <gitea@gerasch.dev>
github.repository preserves the owner's mixed-case login
(ServeurpersoCom), but GHCR/OCI repository names must be all
lowercase, so every docker/build-push-action tag failed with
'repository name must be lowercase'.
Co-authored-by: Gary <gitea@gerasch.dev>
CUDA 13 removed offline compilation for pre-Turing architectures,
so keeping 61-real in the default list makes nvcc fail with
"Unsupported gpu architecture compute_61" on 13+ toolkits.
Split the version check: 13+ drops 61-real, 12.8+ keeps it with
Blackwell, older 12.x keeps it without.
* Add Dockerfile (cpu/cuda), entrypoint, docs, and a GHCR release workflow
Multi-stage Dockerfile with cpu and cuda targets, built from the
existing build scripts' cmake invocations. The CUDA target documents
and applies the two docker-build-specific gotchas we hit running this
in production: CMAKE_CUDA_ARCHITECTURES needs an explicit override for
GPUs older than this project's own default arch list when building
without GPU device access, and the CUDA driver stub library needs an
explicit -L/-lcuda at link time since ggml's VMM pool allocator needs
driver-API symbols that aren't present without a real driver. Also
installs make/pkg-config/libopenblas-dev (CPU build) and libgomp1
(CUDA runtime), and copies the ggml shared libraries alongside the
binaries with LD_LIBRARY_PATH set, since the binaries' baked-in RPATH
points at the build-tree location that doesn't exist in the final
stage.
Adds a GitHub Actions workflow that builds both variants on every
push to master and version tag, publishing to
ghcr.io/serveurpersocom/qwentts.cpp, and validates the build (without
pushing) on PRs that touch the Docker files.
Verified end-to-end on a real GTX 1070 (Pascal): both cpu and cuda
targets build clean, run, and produce valid synthesized WAV output
through tts-server's HTTP API.
* Dockerfile: add a vulkan build target (AMD/Intel GPUs)
Uses the LunarG Vulkan SDK apt repo for glslc (ggml-vulkan's shader
compiler), which Ubuntu 22.04's own repos don't package. Runtime image
ships Mesa's Vulkan drivers; NVIDIA users should prefer :cuda instead,
since NVIDIA's Vulkan ICD isn't bundled.
* docker workflow: bump actions to latest majors (checkout v7, buildx v4, login v4, build-push v7)
* Support Pascal (sm_61) in the default CUDA architecture list
Adds 61-real to CMAKE_CUDA_ARCHITECTURES' default so Pascal cards
(GTX 10-series etc.) work without an explicit override -- including
in the Docker cuda target, where docker build has no GPU device to
autodetect against in the first place. Real-only (no virtual/PTX):
Pascal is now the oldest supported card, so it doesn't need to seed
forward JIT compatibility for anything older the way the 75-virtual
entry does for 7.5+.
Verified end-to-end on a real GTX 1070 (sm_61): built the cuda Docker
target with no --build-arg override, ran it with --gpus all, and
confirmed via container logs that the GPU loaded the model and served
a real synthesis request producing valid WAV output.
Adjusts the Dockerfile comments and docs/DOCKER.md accordingly.
* docker: pass through --max-prefill-tokens, add ref_text voice cloning
- entrypoint.sh: forward MAX_PREFILL_TOKENS to --max-prefill-tokens,
matching the existing optional-flag passthrough pattern
- entrypoint.sh: a same-stem .txt next to a voice's .wav now supplies
ref_text, enabling ICL clone mode instead of the x_vector_only
fallback used when no transcript is given
- switch voice registration's JSON construction from raw printf to jq
for safe escaping of arbitrary transcript text; base64 payloads go
through jq's --rawfile (not --arg) since large files blow past
ARG_MAX as a command-line argument
- add jq to all three runtime image stages (cpu/cuda/vulkan) for the
above
- docs/DOCKER.md: document both additions
Verified end-to-end on a CPU build: both the ref_text and no-ref_text
registration paths log correctly (ref_text=yes / ref_text=no) and
/health responds after voice registration completes.
* docker: opt-in fatal worst-case warmup synthesis
ggml_backend_sched grows its compute buffer to fit the largest graph
it has ever built and never shrinks it back, so VRAM use can ratchet
up the first time a long reply arrives on live traffic. This adds an
opt-in startup warmup: when WARMUP_VOICE is set, entrypoint.sh runs
one real synthesis capped at WARMUP_MAX_NEW_TOKENS (default 750)
frames after voice registration, forcing that worst-case decode/
codec-decode buffer growth to happen at startup instead of mid-request.
Complements --max-prefill-tokens, which only covers the input side.
Off by default (WARMUP_VOICE unset skips it entirely, matching every
other optional flag in this entrypoint), and fatal on failure: if the
warmup synthesis fails (most likely OOM), the container kills the
server and exits non-zero rather than come up healthy and fail
unpredictably later -- a deployment that can't afford its own
configured worst case should know that at startup, not on a live
request.
Verified on a CPU build: warmup runs after voice registration, hits
the configured frame cap exactly, and the container stays healthy.
---------
Co-authored-by: Gary <gitea@gerasch.dev>
The fused mode appended the codec stream tail to the predictor frame
graph, so one compute produced both a frame's codes and its 80 ms of
audio with no host round trip. The experiment applied to max_batch 1
with a streaming synthesis only, it cost throughput against the
buffered flush that stays the default, and it kept a second frame
graph, its ring inputs and an init flag alive for that single case. It
is not worth keeping.
Remove the tail helpers, the fused graph of CodePredGraphSet, the
codec_fused field of qt_init_params, the --codec-fused flag of both
tools and the harness switch that exercised it. The predictor frame
unroll and the in graph sampler are untouched.
Each harness re-runs the C++ side with --codec-fused after the
buffered run and gates on codes-full.bin equality between the two:
the decode path cannot change the predictor, so any divergence is a
bug. The fused run's perf lines land in the log as [Perf Fused] next
to the buffered ones, so every grid cell carries the buffered vs
fused comparison per backend and quant.
The fused audio itself is not scored here: greedy amplitudes vanish
in the streamed PCM_16 and the fused FP envelope has its own
validation against the stream path.
One static frame graph per batch width replaces the per step chain:
prefill and the 15 acoustic steps run in a single backend compute.
This is the target architecture for the llama.cpp Qwen3-TTS port and
serves as its working GGML reference while under test.
Sampling is a plain op chain batched over slots: temperature, argsort
top_k (descending order is guaranteed on every backend, unlike top_k),
softmax, cumsum, cdf crossing against a per step philox uniform.
Greedy draws with u = 0 and lands on the argmax. Faster than the
fused sampling op under CUDA graph capture, greedy codes stay exact
against the Python reference on CPU, CUDA and Vulkan.
Opt in single slot latency mode (--codec-fused on qwen-tts and
tts-server, codec_fused in qt_init_params): the codec stream tail
joins the frame graph at T=1, codes read through a device view, one
80 ms chunk per compute with no host round trip.
Predictor 3.34 -> 3.11 ms/frame on CUDA, end to end -4%.
The left context of the buffered chunked decode is no longer a caller
knob: it derives from the codec's own sliding window (2x144 frames),
placing the default decode at the residual floor of the split.
codec_chunk_sec moves from qt_tts_params to qt_init_params, resolved
once to frames at load. The mid-struct removal bumps the ABI to a
closed range [QT_ABI_MIN_VERSION, QT_ABI_VERSION] = [4, 4]; the probe
asserts both bounds reject through the range check.
The CLI tool already exposes these flags for controlling the vocoder's
chunked-decode window. tts-server always used the hardcoded 24.0s
chunk / 2.0s left-context defaults, with no way to override them at
the server binary's CLI, unlike qwen-tts.
This matters on memory-constrained GPUs: utterances shorter than the
chunk duration decode in a single pass, which can OOM on a small GPU
shared with other processes. Tightening these values (e.g. 4.0/1.5)
forces genuine chunked decode with bounded peak memory per chunk.
get_rows.comp already applies all three misalign offsets; the assert
predates that. Apply the b and d offsets in the quant variant too.
Only reject a quantized src0 with a sub-block offset, which cannot
be expressed in elements.
One codec stream state set per lane plus a staging set, static graphs
per (chunk class, lane count) decoding every streaming lane in one
compute. The engine drops the per slot decoders and ownership
save/load for a shared 1 -> 2 -> 4 -> 8 ramp: lanes accumulate and
flush together, an admit drains and restarts at width 1, a retirement
drains then compacts the lane span device side. ICL references prime
through the staging set into per set snapshots. Depthwise and
transposed convs fold lanes into the channel axis; the dense convs
rely on the ggml conv_1d batched layout fix (submodule bump).
The talker, code predictor, and tokenizer transformer built their MLPs
as two mul_mats followed by a separate silu and mul. Replacing the
silu + mul pair with the GGML_OP_GLU node keeps the gate and up
mul_mats adjacent to it, which is the exact pattern the CUDA backend
fusion pass matches (ggml_cuda_should_fuse_mul_mat): gate, up, and
activation collapse into one kernel on the GEMV decode shapes.
Applies to all five MLP sites: talker prefill and batched decode,
code predictor layer, tokenizer transformer offline and stream paths.
Validated bit-exact against HEAD on greedy and seed 42 (identical
WAVs, F32 master). Around +3 percent on the F32 AR frame; neutral on
correctness, two lines simpler per site.
Device-side embedding and codebook lookups now cover q2_K to q6_K, so a
quantized token_embd no longer drops the graph out of the direct device
path. i-quants are left as a TODO.
The codec encode path allocated its graph without first resetting the
shared scheduler, unlike every sibling path (decode, prompt-builder,
speaker_encoder_extract). After a synthesis call leaves the codec sched
allocated, the next RVQ encode — e.g. registering a second cloned voice
after a generation — trips GGML_ASSERT(!sched->is_alloc) in
ggml_backend_sched_alloc_graph and crashes the server.
Reset the sched before alloc to match the other call sites.
Repro: register a voice (base model) -> generate speech -> register
another voice. Second register crashes without this fix.
backend_init returned a refcounted global BackendPair shared by every
context in the process. That cache is load bearing in acestep.cpp where
each module inits its own backend, but here the pipeline already shares
one BackendPair explicitly, so the cache never hit and only made
independent contexts collide: one CUDA VMM pool is a strict LIFO stack,
so two contexts interleaving alloc/free abort on the pool assert.
Each backend_init call now returns a fresh backend pair with its own
device context and memory pool, backend_release frees it directly, and
the one-time ggml_log_set + ggml_backend_load_all setup moves under a
magic static so concurrent context creation stays safe.
Single-context binaries (CLI, server) are bit-identical. Multi-context
embedders (python bindings running parallel pipelines) no longer crash.
Keep the talker causal mask in each static attention window graph and
reuse positions, KV rows and sliding attention mask buffers in each codec
stream graph class.
This removes the remaining host allocations from the talker and codec
streaming hot loops without changing graph execution.
Install a ggml log callback that collapses exact consecutive
duplicates and reports the total count when the run ends. The CUDA
graph capture logs one reused line per replay step, flooding stderr
and stalling a reader that blocks on a full pipe.
The stream graphs come in width classes T in {1, 2, 4, 8} sharing the
state and KV ring tensors, built lazily per class. The decoder ramps
1 -> 2 -> 4 -> 8 so the first frame keeps its latency while the steady
state interleaves the codec 8x less often and the batch amortizes the
kernel count; drain flushes the tail at EOS and the reference priming
runs in max width chunks.
The decode flavor builds one static graph per 256 step kv window,
lazily on the first step entering the span, and replays directly on
the backend: ids, overlay, positions, kv row, and mask re-upload each
step since n_past moves. The prefill keeps the dynamic arena and sched
path, and the frame id assembly leaves the shared core.
The 15 predictor flavors build and allocate once at load, positions,
kv rows, and mask baked as never freed graph outputs, and replay
directly on the backend. The prefill slices the last position before
lm_head so every flavor reads one logits row at offset zero. Replaces
the per step graph rebuild, sched allocation, and debug prints.
The speech body accepts seed, max_new_tokens, temperature, top_k,
top_p, and repetition_penalty. Unset fields keep the engine defaults,
a temperature of zero selects greedy decoding, and the subtalker
mirrors the talker knobs. A fixed seed makes a request reproducible.
seed_reference hashes the ICL reference codes and restores the conv
contexts, KV ring, and position from a per reference snapshot slot on
a repeat, saving the primed state device to device after a fresh
prime. The reference priming cost amortizes across repeated cloned
voice requests.
qt_extract_voice_ref now pays the speaker encoder weight load on its
first call, mirroring the qt_synthesize ref_audio path. The server
extraction endpoint works without a prior ref wav synthesis.
POST /v1/voices registers a voice from a WAV extracted server side
through qt_extract_voice_ref or from pre extracted .spk and .rvq
latents taken verbatim, DELETE drops it and GET lists it alongside the
model speakers. A registered voice wins over a speaker of the same
name and injects the reference latents into qt_tts_params, ref_text
present selects ICL clone mode. The registry lives in process RAM
under the synthesis mutex, so registration and lookups never race a
running synthesis. The audio and rvq readers gain buffer variants
factored from the file paths. The README and the architecture
document catch up on the streaming decode, the hidden bridge, and the
server endpoints.
enne2
Pascal
karlandPascal
RaduAdumitroaeiandPascal