enne2/qwentts.cpp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

predictor: static graphs replayed with a single id upload per step

Browse Source 
The 15 predictor flavors build and allocate once at load, positions,
kv rows, and mask baked as never freed graph outputs, and replay
directly on the backend. The prefill slices the last position before
lm_head so every flavor reads one logits row at offset zero. Replaces
the per step graph rebuild, sched allocation, and debug prints.
This commit is contained in:
Pascal
2026-07-06 21:46:08 +02:00
parent 66ebf32b2b
commit 2862be4152
6 changed files with 170 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+5 -2
View File
@@ -603,8 +603,11 @@ The `.rvq` container packs the 16 codes per frame at 11 bits LSB-first.
OpenAI-compatible HTTP server over the public ABI, one GPU-resident
context, synthesis serialized FIFO across connections. The shared HTTP
core lives in `src/tts-server.h` (also consumed by the sibling *.cpp
ports); `tools/tts-server.cpp` wires the `qt_*` ABI into it. Verbatim
`--help` :
ports); `tools/tts-server.cpp` wires the `qt_*` ABI into it.
response_format selects the codec path : pcm drives the stateful
streaming decode frame by frame for lowest latency, wav runs the
buffered chunked decode (batch codec, talker uninterrupted) for best
throughput. Verbatim `--help` :
```
Usage: ./build/tts-server --model <gguf> --codec <gguf> [options]