* Add Dockerfile (cpu/cuda), entrypoint, docs, and a GHCR release workflow
Multi-stage Dockerfile with cpu and cuda targets, built from the
existing build scripts' cmake invocations. The CUDA target documents
and applies the two docker-build-specific gotchas we hit running this
in production: CMAKE_CUDA_ARCHITECTURES needs an explicit override for
GPUs older than this project's own default arch list when building
without GPU device access, and the CUDA driver stub library needs an
explicit -L/-lcuda at link time since ggml's VMM pool allocator needs
driver-API symbols that aren't present without a real driver. Also
installs make/pkg-config/libopenblas-dev (CPU build) and libgomp1
(CUDA runtime), and copies the ggml shared libraries alongside the
binaries with LD_LIBRARY_PATH set, since the binaries' baked-in RPATH
points at the build-tree location that doesn't exist in the final
stage.
Adds a GitHub Actions workflow that builds both variants on every
push to master and version tag, publishing to
ghcr.io/serveurpersocom/qwentts.cpp, and validates the build (without
pushing) on PRs that touch the Docker files.
Verified end-to-end on a real GTX 1070 (Pascal): both cpu and cuda
targets build clean, run, and produce valid synthesized WAV output
through tts-server's HTTP API.
* Dockerfile: add a vulkan build target (AMD/Intel GPUs)
Uses the LunarG Vulkan SDK apt repo for glslc (ggml-vulkan's shader
compiler), which Ubuntu 22.04's own repos don't package. Runtime image
ships Mesa's Vulkan drivers; NVIDIA users should prefer :cuda instead,
since NVIDIA's Vulkan ICD isn't bundled.
* docker workflow: bump actions to latest majors (checkout v7, buildx v4, login v4, build-push v7)
* Support Pascal (sm_61) in the default CUDA architecture list
Adds 61-real to CMAKE_CUDA_ARCHITECTURES' default so Pascal cards
(GTX 10-series etc.) work without an explicit override -- including
in the Docker cuda target, where docker build has no GPU device to
autodetect against in the first place. Real-only (no virtual/PTX):
Pascal is now the oldest supported card, so it doesn't need to seed
forward JIT compatibility for anything older the way the 75-virtual
entry does for 7.5+.
Verified end-to-end on a real GTX 1070 (sm_61): built the cuda Docker
target with no --build-arg override, ran it with --gpus all, and
confirmed via container logs that the GPU loaded the model and served
a real synthesis request producing valid WAV output.
Adjusts the Dockerfile comments and docs/DOCKER.md accordingly.
* docker: pass through --max-prefill-tokens, add ref_text voice cloning
- entrypoint.sh: forward MAX_PREFILL_TOKENS to --max-prefill-tokens,
matching the existing optional-flag passthrough pattern
- entrypoint.sh: a same-stem .txt next to a voice's .wav now supplies
ref_text, enabling ICL clone mode instead of the x_vector_only
fallback used when no transcript is given
- switch voice registration's JSON construction from raw printf to jq
for safe escaping of arbitrary transcript text; base64 payloads go
through jq's --rawfile (not --arg) since large files blow past
ARG_MAX as a command-line argument
- add jq to all three runtime image stages (cpu/cuda/vulkan) for the
above
- docs/DOCKER.md: document both additions
Verified end-to-end on a CPU build: both the ref_text and no-ref_text
registration paths log correctly (ref_text=yes / ref_text=no) and
/health responds after voice registration completes.
* docker: opt-in fatal worst-case warmup synthesis
ggml_backend_sched grows its compute buffer to fit the largest graph
it has ever built and never shrinks it back, so VRAM use can ratchet
up the first time a long reply arrives on live traffic. This adds an
opt-in startup warmup: when WARMUP_VOICE is set, entrypoint.sh runs
one real synthesis capped at WARMUP_MAX_NEW_TOKENS (default 750)
frames after voice registration, forcing that worst-case decode/
codec-decode buffer growth to happen at startup instead of mid-request.
Complements --max-prefill-tokens, which only covers the input side.
Off by default (WARMUP_VOICE unset skips it entirely, matching every
other optional flag in this entrypoint), and fatal on failure: if the
warmup synthesis fails (most likely OOM), the container kills the
server and exits non-zero rather than come up healthy and fail
unpredictably later -- a deployment that can't afford its own
configured worst case should know that at startup, not on a live
request.
Verified on a CPU build: warmup runs after voice registration, hits
the configured frame cap exactly, and the container stays healthy.
---------
Co-authored-by: Gary <gitea@gerasch.dev>
The fused mode appended the codec stream tail to the predictor frame
graph, so one compute produced both a frame's codes and its 80 ms of
audio with no host round trip. The experiment applied to max_batch 1
with a streaming synthesis only, it cost throughput against the
buffered flush that stays the default, and it kept a second frame
graph, its ring inputs and an init flag alive for that single case. It
is not worth keeping.
Remove the tail helpers, the fused graph of CodePredGraphSet, the
codec_fused field of qt_init_params, the --codec-fused flag of both
tools and the harness switch that exercised it. The predictor frame
unroll and the in graph sampler are untouched.
Each harness re-runs the C++ side with --codec-fused after the
buffered run and gates on codes-full.bin equality between the two:
the decode path cannot change the predictor, so any divergence is a
bug. The fused run's perf lines land in the log as [Perf Fused] next
to the buffered ones, so every grid cell carries the buffered vs
fused comparison per backend and quant.
The fused audio itself is not scored here: greedy amplitudes vanish
in the streamed PCM_16 and the fused FP envelope has its own
validation against the stream path.
One static frame graph per batch width replaces the per step chain:
prefill and the 15 acoustic steps run in a single backend compute.
This is the target architecture for the llama.cpp Qwen3-TTS port and
serves as its working GGML reference while under test.
Sampling is a plain op chain batched over slots: temperature, argsort
top_k (descending order is guaranteed on every backend, unlike top_k),
softmax, cumsum, cdf crossing against a per step philox uniform.
Greedy draws with u = 0 and lands on the argmax. Faster than the
fused sampling op under CUDA graph capture, greedy codes stay exact
against the Python reference on CPU, CUDA and Vulkan.
Opt in single slot latency mode (--codec-fused on qwen-tts and
tts-server, codec_fused in qt_init_params): the codec stream tail
joins the frame graph at T=1, codes read through a device view, one
80 ms chunk per compute with no host round trip.
Predictor 3.34 -> 3.11 ms/frame on CUDA, end to end -4%.
The left context of the buffered chunked decode is no longer a caller
knob: it derives from the codec's own sliding window (2x144 frames),
placing the default decode at the residual floor of the split.
codec_chunk_sec moves from qt_tts_params to qt_init_params, resolved
once to frames at load. The mid-struct removal bumps the ABI to a
closed range [QT_ABI_MIN_VERSION, QT_ABI_VERSION] = [4, 4]; the probe
asserts both bounds reject through the range check.
The CLI tool already exposes these flags for controlling the vocoder's
chunked-decode window. tts-server always used the hardcoded 24.0s
chunk / 2.0s left-context defaults, with no way to override them at
the server binary's CLI, unlike qwen-tts.
This matters on memory-constrained GPUs: utterances shorter than the
chunk duration decode in a single pass, which can OOM on a small GPU
shared with other processes. Tightening these values (e.g. 4.0/1.5)
forces genuine chunked decode with bounded peak memory per chunk.
get_rows.comp already applies all three misalign offsets; the assert
predates that. Apply the b and d offsets in the quant variant too.
Only reject a quantized src0 with a sub-block offset, which cannot
be expressed in elements.
One codec stream state set per lane plus a staging set, static graphs
per (chunk class, lane count) decoding every streaming lane in one
compute. The engine drops the per slot decoders and ownership
save/load for a shared 1 -> 2 -> 4 -> 8 ramp: lanes accumulate and
flush together, an admit drains and restarts at width 1, a retirement
drains then compacts the lane span device side. ICL references prime
through the staging set into per set snapshots. Depthwise and
transposed convs fold lanes into the channel axis; the dense convs
rely on the ggml conv_1d batched layout fix (submodule bump).
The talker, code predictor, and tokenizer transformer built their MLPs
as two mul_mats followed by a separate silu and mul. Replacing the
silu + mul pair with the GGML_OP_GLU node keeps the gate and up
mul_mats adjacent to it, which is the exact pattern the CUDA backend
fusion pass matches (ggml_cuda_should_fuse_mul_mat): gate, up, and
activation collapse into one kernel on the GEMV decode shapes.
Applies to all five MLP sites: talker prefill and batched decode,
code predictor layer, tokenizer transformer offline and stream paths.
Validated bit-exact against HEAD on greedy and seed 42 (identical
WAVs, F32 master). Around +3 percent on the F32 AR frame; neutral on
correctness, two lines simpler per site.
Device-side embedding and codebook lookups now cover q2_K to q6_K, so a
quantized token_embd no longer drops the graph out of the direct device
path. i-quants are left as a TODO.
The codec encode path allocated its graph without first resetting the
shared scheduler, unlike every sibling path (decode, prompt-builder,
speaker_encoder_extract). After a synthesis call leaves the codec sched
allocated, the next RVQ encode — e.g. registering a second cloned voice
after a generation — trips GGML_ASSERT(!sched->is_alloc) in
ggml_backend_sched_alloc_graph and crashes the server.
Reset the sched before alloc to match the other call sites.
Repro: register a voice (base model) -> generate speech -> register
another voice. Second register crashes without this fix.
backend_init returned a refcounted global BackendPair shared by every
context in the process. That cache is load bearing in acestep.cpp where
each module inits its own backend, but here the pipeline already shares
one BackendPair explicitly, so the cache never hit and only made
independent contexts collide: one CUDA VMM pool is a strict LIFO stack,
so two contexts interleaving alloc/free abort on the pool assert.
Each backend_init call now returns a fresh backend pair with its own
device context and memory pool, backend_release frees it directly, and
the one-time ggml_log_set + ggml_backend_load_all setup moves under a
magic static so concurrent context creation stays safe.
Single-context binaries (CLI, server) are bit-identical. Multi-context
embedders (python bindings running parallel pipelines) no longer crash.
Keep the talker causal mask in each static attention window graph and
reuse positions, KV rows and sliding attention mask buffers in each codec
stream graph class.
This removes the remaining host allocations from the talker and codec
streaming hot loops without changing graph execution.
Install a ggml log callback that collapses exact consecutive
duplicates and reports the total count when the run ends. The CUDA
graph capture logs one reused line per replay step, flooding stderr
and stalling a reader that blocks on a full pipe.
The stream graphs come in width classes T in {1, 2, 4, 8} sharing the
state and KV ring tensors, built lazily per class. The decoder ramps
1 -> 2 -> 4 -> 8 so the first frame keeps its latency while the steady
state interleaves the codec 8x less often and the batch amortizes the
kernel count; drain flushes the tail at EOS and the reference priming
runs in max width chunks.
The decode flavor builds one static graph per 256 step kv window,
lazily on the first step entering the span, and replays directly on
the backend: ids, overlay, positions, kv row, and mask re-upload each
step since n_past moves. The prefill keeps the dynamic arena and sched
path, and the frame id assembly leaves the shared core.
The 15 predictor flavors build and allocate once at load, positions,
kv rows, and mask baked as never freed graph outputs, and replay
directly on the backend. The prefill slices the last position before
lm_head so every flavor reads one logits row at offset zero. Replaces
the per step graph rebuild, sched allocation, and debug prints.
The speech body accepts seed, max_new_tokens, temperature, top_k,
top_p, and repetition_penalty. Unset fields keep the engine defaults,
a temperature of zero selects greedy decoding, and the subtalker
mirrors the talker knobs. A fixed seed makes a request reproducible.
seed_reference hashes the ICL reference codes and restores the conv
contexts, KV ring, and position from a per reference snapshot slot on
a repeat, saving the primed state device to device after a fresh
prime. The reference priming cost amortizes across repeated cloned
voice requests.
qt_extract_voice_ref now pays the speaker encoder weight load on its
first call, mirroring the qt_synthesize ref_audio path. The server
extraction endpoint works without a prior ref wav synthesis.
POST /v1/voices registers a voice from a WAV extracted server side
through qt_extract_voice_ref or from pre extracted .spk and .rvq
latents taken verbatim, DELETE drops it and GET lists it alongside the
model speakers. A registered voice wins over a speaker of the same
name and injects the reference latents into qt_tts_params, ref_text
present selects ICL clone mode. The registry lives in process RAM
under the synthesis mutex, so registration and lookups never race a
running synthesis. The audio and rvq readers gain buffer variants
factored from the file paths. The README and the architecture
document catch up on the streaming decode, the hidden bridge, and the
server endpoints.
Every causal conv carries its left context in a persistent backend
tensor and every transposed conv its overlap tail, so a T=1 frame
decode reproduces the offline full decode exactly with zero re decoded
context. The tokenizer transformer attends over a sliding window KV
ring written through set_rows. The frame graph builds and allocates
once, then every frame is input uploads, one direct backend compute,
and one readback. The quantizer conts each codebook id view so the
Vulkan get_rows path accepts the direct compute. Each generated frame
emits its samples immediately and ICL priming feeds the full reference
through the same state. The buffered path keeps the chunked decode and
both codec framing knobs now apply to it alone.
KV writes go through set_rows with the destination rows carried as data
and the code predictor steps get one arena per sub step, so every
decode graph keeps a fixed topology and fixed tensor addresses step
after step. The talker last hidden stays resident on device in a
persistent bridge tensor, written by the talker graph and concatenated
as a leaf by the predictor prefill. The hot loop uploads sixteen code
ids and one overlay row and reads back the logits alone, host hidden
readbacks survive only under the dump path. Fewer nodes, transfers and
syncs pay on every backend, and the stable topology lets the CUDA
backend replay its captured graphs without an update.
pipeline_codec_decode builds its graph in a persistent arena instead of
a fresh ggml context per call. Constant size streaming slices rebuild
every node at the same address with identical shapes, which trims host
side graph churn on every backend and lets the CUDA backend replay its
captured graph executable without an update. The encode path keeps its
per call context.
The upstream pipeline decodes reference plus generated codes then trims,
giving the first generated frames causal context from the reference; the
generated only decode started the vocoder from an empty state and
colored the clone onset. Seed both decode paths with the last
min(ref_T, left_ctx_frames) reference frames: the streaming decoder
takes them below its emit cursor so they are never emitted, the buffered
path prepends them and strips their samples. Raising
codec_left_context_sec past the reference duration reproduces the
upstream full reference decode exactly.
Pascal
karlandPascal
RaduAdumitroaeiandPascal