Add Dockerfile (cpu/cuda/vulkan), entrypoint, docs, and a GHCR build/release workflow (#15)
* Add Dockerfile (cpu/cuda), entrypoint, docs, and a GHCR release workflow Multi-stage Dockerfile with cpu and cuda targets, built from the existing build scripts' cmake invocations. The CUDA target documents and applies the two docker-build-specific gotchas we hit running this in production: CMAKE_CUDA_ARCHITECTURES needs an explicit override for GPUs older than this project's own default arch list when building without GPU device access, and the CUDA driver stub library needs an explicit -L/-lcuda at link time since ggml's VMM pool allocator needs driver-API symbols that aren't present without a real driver. Also installs make/pkg-config/libopenblas-dev (CPU build) and libgomp1 (CUDA runtime), and copies the ggml shared libraries alongside the binaries with LD_LIBRARY_PATH set, since the binaries' baked-in RPATH points at the build-tree location that doesn't exist in the final stage. Adds a GitHub Actions workflow that builds both variants on every push to master and version tag, publishing to ghcr.io/serveurpersocom/qwentts.cpp, and validates the build (without pushing) on PRs that touch the Docker files. Verified end-to-end on a real GTX 1070 (Pascal): both cpu and cuda targets build clean, run, and produce valid synthesized WAV output through tts-server's HTTP API. * Dockerfile: add a vulkan build target (AMD/Intel GPUs) Uses the LunarG Vulkan SDK apt repo for glslc (ggml-vulkan's shader compiler), which Ubuntu 22.04's own repos don't package. Runtime image ships Mesa's Vulkan drivers; NVIDIA users should prefer :cuda instead, since NVIDIA's Vulkan ICD isn't bundled. * docker workflow: bump actions to latest majors (checkout v7, buildx v4, login v4, build-push v7) * Support Pascal (sm_61) in the default CUDA architecture list Adds 61-real to CMAKE_CUDA_ARCHITECTURES' default so Pascal cards (GTX 10-series etc.) work without an explicit override -- including in the Docker cuda target, where docker build has no GPU device to autodetect against in the first place. Real-only (no virtual/PTX): Pascal is now the oldest supported card, so it doesn't need to seed forward JIT compatibility for anything older the way the 75-virtual entry does for 7.5+. Verified end-to-end on a real GTX 1070 (sm_61): built the cuda Docker target with no --build-arg override, ran it with --gpus all, and confirmed via container logs that the GPU loaded the model and served a real synthesis request producing valid WAV output. Adjusts the Dockerfile comments and docs/DOCKER.md accordingly. * docker: pass through --max-prefill-tokens, add ref_text voice cloning - entrypoint.sh: forward MAX_PREFILL_TOKENS to --max-prefill-tokens, matching the existing optional-flag passthrough pattern - entrypoint.sh: a same-stem .txt next to a voice's .wav now supplies ref_text, enabling ICL clone mode instead of the x_vector_only fallback used when no transcript is given - switch voice registration's JSON construction from raw printf to jq for safe escaping of arbitrary transcript text; base64 payloads go through jq's --rawfile (not --arg) since large files blow past ARG_MAX as a command-line argument - add jq to all three runtime image stages (cpu/cuda/vulkan) for the above - docs/DOCKER.md: document both additions Verified end-to-end on a CPU build: both the ref_text and no-ref_text registration paths log correctly (ref_text=yes / ref_text=no) and /health responds after voice registration completes. * docker: opt-in fatal worst-case warmup synthesis ggml_backend_sched grows its compute buffer to fit the largest graph it has ever built and never shrinks it back, so VRAM use can ratchet up the first time a long reply arrives on live traffic. This adds an opt-in startup warmup: when WARMUP_VOICE is set, entrypoint.sh runs one real synthesis capped at WARMUP_MAX_NEW_TOKENS (default 750) frames after voice registration, forcing that worst-case decode/ codec-decode buffer growth to happen at startup instead of mid-request. Complements --max-prefill-tokens, which only covers the input side. Off by default (WARMUP_VOICE unset skips it entirely, matching every other optional flag in this entrypoint), and fatal on failure: if the warmup synthesis fails (most likely OOM), the container kills the server and exits non-zero rather than come up healthy and fail unpredictably later -- a deployment that can't afford its own configured worst case should know that at startup, not on a live request. Verified on a CPU build: warmup runs after voice registration, hits the configured frame cap exactly, and the container stays healthy. --------- Co-authored-by: Gary <gitea@gerasch.dev>
This commit is contained in:
Stefan Gerasch
Gary
@@ -0,0 +1,7 @@
|
||||
.git
|
||||
build
|
||||
checkpoints
|
||||
models
|
||||
docs
|
||||
examples
|
||||
*.md
|
||||
@@ -0,0 +1,65 @@
|
||||
name: Docker
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
tags: ['v*']
|
||||
pull_request:
|
||||
paths:
|
||||
- 'Dockerfile'
|
||||
- 'docker/**'
|
||||
- '.dockerignore'
|
||||
- '.github/workflows/docker.yml'
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
variant: [cpu, cuda, vulkan]
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
submodules: recursive
|
||||
|
||||
- uses: docker/setup-buildx-action@v4
|
||||
|
||||
- name: Log in to GHCR
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Compute tags
|
||||
id: tags
|
||||
shell: bash
|
||||
run: |
|
||||
image="ghcr.io/${{ github.repository }}"
|
||||
variant="${{ matrix.variant }}"
|
||||
tags="${image}:${variant}-${{ github.sha }}"
|
||||
if [ "${{ github.ref }}" = "refs/heads/master" ]; then
|
||||
tags="${tags},${image}:${variant}"
|
||||
fi
|
||||
if [[ "${{ github.ref }}" == refs/tags/v* ]]; then
|
||||
version="${GITHUB_REF_NAME#v}"
|
||||
tags="${tags},${image}:${variant}-v${version}"
|
||||
fi
|
||||
echo "tags=${tags}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Build and push
|
||||
uses: docker/build-push-action@v7
|
||||
with:
|
||||
context: .
|
||||
target: ${{ matrix.variant }}
|
||||
push: ${{ github.event_name != 'pull_request' }}
|
||||
tags: ${{ steps.tags.outputs.tags }}
|
||||
cache-from: type=gha,scope=${{ matrix.variant }}
|
||||
cache-to: type=gha,mode=max,scope=${{ matrix.variant }}
|
||||
+6
-3
@@ -46,14 +46,17 @@ if(NOT MSVC AND NOT GGML_SYCL)
|
||||
add_compile_definitions(_FORTIFY_SOURCE=2)
|
||||
endif()
|
||||
|
||||
# CUDA architectures: cover Turing to Blackwell for distributed binaries.
|
||||
# CUDA architectures: cover Pascal to Blackwell for distributed binaries.
|
||||
# Pascal (61-real) is SASS-only, no virtual/PTX entry: it's the oldest
|
||||
# supported card and doesn't need to seed forward JIT compat for anything
|
||||
# older, unlike the 75-virtual baseline which does that for 7.5+.
|
||||
# Users can override with -DCMAKE_CUDA_ARCHITECTURES=native for local builds.
|
||||
if(NOT DEFINED CMAKE_CUDA_ARCHITECTURES)
|
||||
find_package(CUDAToolkit QUIET)
|
||||
if(CUDAToolkit_FOUND AND CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8")
|
||||
set(CMAKE_CUDA_ARCHITECTURES "75-virtual;80-virtual;86-real;89-real;120a-real;121a-real")
|
||||
set(CMAKE_CUDA_ARCHITECTURES "61-real;75-virtual;80-virtual;86-real;89-real;120a-real;121a-real")
|
||||
else()
|
||||
set(CMAKE_CUDA_ARCHITECTURES "75-virtual;80-virtual;86-real;89-real")
|
||||
set(CMAKE_CUDA_ARCHITECTURES "61-real;75-virtual;80-virtual;86-real;89-real")
|
||||
endif()
|
||||
endif()
|
||||
|
||||
|
||||
+108
@@ -0,0 +1,108 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
#
|
||||
# Build context must have the `ggml` submodule checked out already
|
||||
# (`git clone --recurse-submodules`, or `actions/checkout` with
|
||||
# `submodules: recursive` in CI) -- this Dockerfile does not fetch it.
|
||||
#
|
||||
# Usage:
|
||||
# docker build --target cpu -t qwentts.cpp:cpu .
|
||||
# docker build --target cuda -t qwentts.cpp:cuda .
|
||||
# docker build --target vulkan -t qwentts.cpp:vulkan .
|
||||
#
|
||||
# This project's default distributed arch list covers Pascal (sm_61) to
|
||||
# Blackwell. GPUs older than that need an explicit override, since
|
||||
# `docker build` has no GPU device to auto-detect against:
|
||||
# docker build --target cuda -t qwentts.cpp:cuda \
|
||||
# --build-arg CMAKE_CUDA_ARCHITECTURES=50 . # Maxwell
|
||||
# See docs/DOCKER.md for details.
|
||||
|
||||
ARG CUDA_BUILD_IMAGE=nvidia/cuda:12.4.1-devel-ubuntu22.04
|
||||
ARG CUDA_RUNTIME_IMAGE=nvidia/cuda:12.4.1-runtime-ubuntu22.04
|
||||
|
||||
# ---------------------------------------------------------------- CPU build
|
||||
FROM ubuntu:22.04 AS build-cpu
|
||||
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends \
|
||||
git ca-certificates cmake g++ make pkg-config libopenblas-dev \
|
||||
> /dev/null && rm -rf /var/lib/apt/lists/*
|
||||
WORKDIR /build
|
||||
COPY . .
|
||||
RUN cmake -B build -DGGML_BLAS=ON -DCMAKE_BUILD_TYPE=Release && \
|
||||
cmake --build build --config Release -j"$(nproc)"
|
||||
|
||||
FROM ubuntu:22.04 AS cpu
|
||||
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends \
|
||||
libgomp1 libopenblas0 curl ca-certificates jq \
|
||||
> /dev/null && rm -rf /var/lib/apt/lists/*
|
||||
WORKDIR /app
|
||||
COPY --from=build-cpu /build/build/tts-server /build/build/qwen-tts /build/build/qwen-codec /build/build/*.so* ./
|
||||
COPY docker/entrypoint.sh ./entrypoint.sh
|
||||
RUN chmod +x ./entrypoint.sh
|
||||
# Binaries are copied out of the build tree their RPATH points at, so the
|
||||
# ggml shared libraries (copied alongside, above) need an explicit search path.
|
||||
ENV LD_LIBRARY_PATH=/app
|
||||
ENTRYPOINT ["./entrypoint.sh"]
|
||||
|
||||
# --------------------------------------------------------------- CUDA build
|
||||
FROM ${CUDA_BUILD_IMAGE} AS build-cuda
|
||||
ARG CMAKE_CUDA_ARCHITECTURES
|
||||
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends \
|
||||
git ca-certificates cmake g++ make \
|
||||
> /dev/null && rm -rf /var/lib/apt/lists/*
|
||||
WORKDIR /build
|
||||
COPY . .
|
||||
# `docker build` never has GPU device access, unlike `docker run --gpus`, so:
|
||||
# - CMAKE_CUDA_ARCHITECTURES must be set explicitly when targeting a GPU
|
||||
# generation outside this project's own default arch list (see
|
||||
# docs/DOCKER.md); when unset here, CMake's own project default
|
||||
# (Pascal and newer) is used unchanged.
|
||||
# - ggml's CUDA VMM pool allocator needs driver-API symbols (cuMemCreate,
|
||||
# cuMemMap, ...) at link time. The real libcuda.so isn't present without
|
||||
# a GPU, but the devel image ships a link-time-only stub at
|
||||
# lib64/stubs/libcuda.so for exactly this case; it isn't on the default
|
||||
# linker search path so both -L and -lcuda are needed explicitly.
|
||||
RUN cmake -B build -DGGML_CUDA=ON \
|
||||
-DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
|
||||
${CMAKE_CUDA_ARCHITECTURES:+-DCMAKE_CUDA_ARCHITECTURES=${CMAKE_CUDA_ARCHITECTURES}} \
|
||||
-DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs -lcuda" \
|
||||
-DCMAKE_SHARED_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs -lcuda" \
|
||||
-DCMAKE_BUILD_TYPE=Release && \
|
||||
cmake --build build --config Release -j"$(nproc)"
|
||||
|
||||
FROM ${CUDA_RUNTIME_IMAGE} AS cuda
|
||||
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends \
|
||||
libgomp1 curl ca-certificates jq \
|
||||
> /dev/null && rm -rf /var/lib/apt/lists/*
|
||||
WORKDIR /app
|
||||
COPY --from=build-cuda /build/build/tts-server /build/build/qwen-tts /build/build/qwen-codec /build/build/*.so* ./
|
||||
COPY docker/entrypoint.sh ./entrypoint.sh
|
||||
RUN chmod +x ./entrypoint.sh
|
||||
ENV LD_LIBRARY_PATH=/app
|
||||
ENTRYPOINT ["./entrypoint.sh"]
|
||||
|
||||
# ------------------------------------------------------------- Vulkan build
|
||||
# AMD/Intel GPUs (and NVIDIA via its Vulkan ICD). glslc (shader compiler) is
|
||||
# only packaged by the LunarG SDK repo on Ubuntu 22.04, not apt's universe.
|
||||
FROM ubuntu:22.04 AS build-vulkan
|
||||
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends \
|
||||
git ca-certificates cmake g++ make wget gnupg \
|
||||
> /dev/null && rm -rf /var/lib/apt/lists/*
|
||||
RUN wget -qO- https://packages.lunarg.com/lunarg-signing-key-pub.asc | gpg --dearmor -o /usr/share/keyrings/lunarg.gpg && \
|
||||
echo "deb [signed-by=/usr/share/keyrings/lunarg.gpg] https://packages.lunarg.com/vulkan/1.3.296 jammy main" \
|
||||
> /etc/apt/sources.list.d/lunarg-vulkan.list && \
|
||||
apt-get update -qq && apt-get install -y -qq --no-install-recommends vulkan-sdk \
|
||||
> /dev/null && rm -rf /var/lib/apt/lists/*
|
||||
WORKDIR /build
|
||||
COPY . .
|
||||
RUN cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release && \
|
||||
cmake --build build --config Release -j"$(nproc)"
|
||||
|
||||
FROM ubuntu:22.04 AS vulkan
|
||||
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends \
|
||||
libgomp1 libvulkan1 mesa-vulkan-drivers curl ca-certificates jq \
|
||||
> /dev/null && rm -rf /var/lib/apt/lists/*
|
||||
WORKDIR /app
|
||||
COPY --from=build-vulkan /build/build/tts-server /build/build/qwen-tts /build/build/qwen-codec /build/build/*.so* ./
|
||||
COPY docker/entrypoint.sh ./entrypoint.sh
|
||||
RUN chmod +x ./entrypoint.sh
|
||||
ENV LD_LIBRARY_PATH=/app
|
||||
ENTRYPOINT ["./entrypoint.sh"]
|
||||
@@ -39,6 +39,9 @@ cd qwentts.cpp
|
||||
NVCC_CCBIN=g++-13 ./buildcuda.sh # rolling release distros (Arch w/ GCC 16, etc.)
|
||||
```
|
||||
|
||||
Docker images (CPU and CUDA, built and published on every release) are
|
||||
also available: see [docs/DOCKER.md](docs/DOCKER.md).
|
||||
|
||||
## Model conversion
|
||||
|
||||
Pre-converted GGUFs are available on Hugging Face :
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
#!/bin/bash
|
||||
# tts-server entrypoint: starts the server, then registers every reference
|
||||
# voice found in /voices (each *.wav registers under its filename stem,
|
||||
# optionally paired with a same-stem .txt for ref_text ICL cloning) once
|
||||
# the server is ready to accept requests.
|
||||
set -e
|
||||
|
||||
MODEL=${MODEL_PATH:-/models/qwen-talker-1.7b-base-Q8_0.gguf}
|
||||
CODEC=${CODEC_PATH:-/models/qwen-tokenizer-12hz-Q8_0.gguf}
|
||||
LANG=${TTS_LANG:-auto}
|
||||
HOST=${HOST:-0.0.0.0}
|
||||
PORT=${PORT:-8080}
|
||||
ALIAS=${MODEL_ALIAS:-}
|
||||
|
||||
extra_args=()
|
||||
[ -n "$ALIAS" ] && extra_args+=(--alias "$ALIAS")
|
||||
[ -n "$CODEC_CHUNK_DUR" ] && extra_args+=(--codec-chunk-dur "$CODEC_CHUNK_DUR")
|
||||
[ -n "$CODEC_LEFT_DUR" ] && extra_args+=(--codec-left-dur "$CODEC_LEFT_DUR")
|
||||
[ -n "$MAX_BATCH" ] && extra_args+=(--max-batch "$MAX_BATCH")
|
||||
[ -n "$MAX_PREFILL_TOKENS" ] && extra_args+=(--max-prefill-tokens "$MAX_PREFILL_TOKENS")
|
||||
[ "$NO_FA" = "1" ] && extra_args+=(--no-fa)
|
||||
[ "$CLAMP_FP16" = "1" ] && extra_args+=(--clamp-fp16)
|
||||
|
||||
/app/tts-server \
|
||||
--model "$MODEL" \
|
||||
--codec "$CODEC" \
|
||||
--lang "$LANG" \
|
||||
--host "$HOST" \
|
||||
--port "$PORT" \
|
||||
"${extra_args[@]}" &
|
||||
SERVER_PID=$!
|
||||
|
||||
until curl -sf "http://localhost:${PORT}/health" > /dev/null 2>&1; do
|
||||
kill -0 "$SERVER_PID" 2>/dev/null || { echo "tts-server exited before becoming healthy" >&2; wait "$SERVER_PID"; }
|
||||
sleep 1
|
||||
done
|
||||
|
||||
for wav in /voices/*.wav; do
|
||||
[ -f "$wav" ] || continue
|
||||
name=$(basename "$wav" .wav)
|
||||
# base64 payload can be multiple MB -- too large for an argv string
|
||||
# (ARG_MAX), so it's written to a temp file and read in via jq's
|
||||
# --rawfile rather than passed as a --arg.
|
||||
b64_file=$(mktemp)
|
||||
base64 -w0 "$wav" > "$b64_file"
|
||||
txt="${wav%.wav}.txt"
|
||||
# A same-stem .txt supplies the reference transcript, enabling ICL
|
||||
# clone mode (higher fidelity) instead of the x_vector_only fallback
|
||||
# used when no ref_text is sent. jq handles JSON-escaping arbitrary
|
||||
# transcript text safely (quotes, backslashes, ...).
|
||||
if [ -f "$txt" ]; then
|
||||
echo "Registering voice '$name' from $wav (with ref_text from $txt)"
|
||||
jq -n --arg name "$name" --rawfile wav_b64 "$b64_file" --rawfile ref_text "$txt" \
|
||||
'{name: $name, wav_b64: $wav_b64, ref_text: ($ref_text | sub("\n+$"; ""))}' \
|
||||
> /tmp/voice_payload.json
|
||||
else
|
||||
echo "Registering voice '$name' from $wav"
|
||||
jq -n --arg name "$name" --rawfile wav_b64 "$b64_file" \
|
||||
'{name: $name, wav_b64: $wav_b64}' \
|
||||
> /tmp/voice_payload.json
|
||||
fi
|
||||
curl -sf -X POST "http://localhost:${PORT}/v1/audio/voices" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d @/tmp/voice_payload.json \
|
||||
&& echo " -> ok" || echo " -> FAILED"
|
||||
rm -f /tmp/voice_payload.json "$b64_file"
|
||||
done
|
||||
|
||||
# Optional fatal worst-case warmup: exercises one real end-to-end synthesis
|
||||
# capped at WARMUP_MAX_NEW_TOKENS frames, so the decode and codec-decode
|
||||
# compute buffers (the part that scales with output AUDIO length) get
|
||||
# reserved up front -- complementing --max-prefill-tokens above, which
|
||||
# only reserves for input TEXT length. Off by default: only runs when
|
||||
# WARMUP_VOICE names an already-registered voice. Failure is fatal (kills
|
||||
# the server and exits non-zero) by design: a deployment that can't
|
||||
# afford its own configured worst case should refuse to come up healthy,
|
||||
# not fail unpredictably on a live request later. The exact wording of
|
||||
# WARMUP_TEXT doesn't matter -- only that it's long enough the model
|
||||
# doesn't hit EOS on its own before WARMUP_MAX_NEW_TOKENS does, so the
|
||||
# warmup actually exercises the same truncation path a real over-length
|
||||
# response would hit in production.
|
||||
if [ -n "$WARMUP_VOICE" ]; then
|
||||
WARMUP_MAX_NEW_TOKENS=${WARMUP_MAX_NEW_TOKENS:-750}
|
||||
WARMUP_TEXT=${WARMUP_TEXT:-"This is a warmup sentence used to reserve worst case memory usage before accepting real requests. This is a warmup sentence used to reserve worst case memory usage before accepting real requests. This is a warmup sentence used to reserve worst case memory usage before accepting real requests."}
|
||||
echo "Warming up with a ${WARMUP_MAX_NEW_TOKENS}-frame-capped synthesis to reserve worst-case VRAM..."
|
||||
if ! curl -sf -X POST "http://localhost:${PORT}/v1/audio/speech" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "$(jq -n --arg voice "$WARMUP_VOICE" --arg input "$WARMUP_TEXT" --argjson max_new_tokens "$WARMUP_MAX_NEW_TOKENS" \
|
||||
'{voice: $voice, input: $input, response_format: "pcm", max_new_tokens: $max_new_tokens}')" \
|
||||
-o /dev/null; then
|
||||
echo " -> FATAL: worst-case warmup synthesis failed (likely out of VRAM) -- refusing to come up healthy"
|
||||
kill "$SERVER_PID" 2>/dev/null
|
||||
exit 1
|
||||
fi
|
||||
echo " -> ok"
|
||||
fi
|
||||
|
||||
wait "$SERVER_PID"
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
# Docker
|
||||
|
||||
Pre-built images: `ghcr.io/serveurpersocom/qwentts.cpp:cpu`,
|
||||
`:cuda` and `:vulkan` (also tagged per release, e.g. `:cuda-v1.2.3`).
|
||||
All three run `tts-server`; `qwen-tts` and `qwen-codec` are included in
|
||||
the same image at `/app/`.
|
||||
|
||||
```
|
||||
docker run --rm -p 8080:8080 \
|
||||
-v /path/to/models:/models:ro \
|
||||
-e MODEL_PATH=/models/qwen-talker-1.7b-base-Q8_0.gguf \
|
||||
-e CODEC_PATH=/models/qwen-tokenizer-12hz-Q8_0.gguf \
|
||||
ghcr.io/serveurpersocom/qwentts.cpp:cpu
|
||||
```
|
||||
|
||||
CUDA image, with GPU access and a directory of reference WAVs to
|
||||
auto-register as cloned voices on startup:
|
||||
|
||||
```
|
||||
docker run --rm --gpus all -p 8080:8080 \
|
||||
-v /path/to/models:/models:ro \
|
||||
-v /path/to/voices:/voices:ro \
|
||||
-e MODEL_PATH=/models/qwen-talker-1.7b-base-Q8_0.gguf \
|
||||
-e CODEC_PATH=/models/qwen-tokenizer-12hz-Q8_0.gguf \
|
||||
ghcr.io/serveurpersocom/qwentts.cpp:cuda
|
||||
```
|
||||
|
||||
Vulkan image (AMD/Intel GPUs), passing through the DRI device node:
|
||||
|
||||
```
|
||||
docker run --rm --device /dev/dri -p 8080:8080 \
|
||||
-v /path/to/models:/models:ro \
|
||||
-e MODEL_PATH=/models/qwen-talker-1.7b-base-Q8_0.gguf \
|
||||
-e CODEC_PATH=/models/qwen-tokenizer-12hz-Q8_0.gguf \
|
||||
ghcr.io/serveurpersocom/qwentts.cpp:vulkan
|
||||
```
|
||||
|
||||
The `:vulkan` image bundles Mesa's Vulkan drivers (AMD/Intel). On an
|
||||
NVIDIA GPU, prefer `:cuda`; running `:vulkan` there would additionally
|
||||
need the host's proprietary NVIDIA Vulkan ICD mounted in, which the
|
||||
image does not provide.
|
||||
|
||||
## Entrypoint environment variables
|
||||
|
||||
| Variable | Default |
|
||||
|--------------------|---------------------------------------------|
|
||||
| `MODEL_PATH` | `/models/qwen-talker-1.7b-base-Q8_0.gguf` |
|
||||
| `CODEC_PATH` | `/models/qwen-tokenizer-12hz-Q8_0.gguf` |
|
||||
| `TTS_LANG` | `auto` |
|
||||
| `HOST` | `0.0.0.0` |
|
||||
| `PORT` | `8080` |
|
||||
| `MODEL_ALIAS` | unset (reports the GGUF file name) |
|
||||
| `CODEC_CHUNK_DUR` | unset (server default: `24.0`) |
|
||||
| `CODEC_LEFT_DUR` | unset (server default: `2.0`) |
|
||||
| `MAX_BATCH` | unset (server default: `1`) |
|
||||
| `MAX_PREFILL_TOKENS` | unset (server default: `0`, disabled) |
|
||||
| `NO_FA` | unset; set to `1` to disable flash attention |
|
||||
| `CLAMP_FP16` | unset; set to `1` to clamp hidden states |
|
||||
| `WARMUP_VOICE` | unset; set to a registered voice name to enable the startup warmup below |
|
||||
| `WARMUP_MAX_NEW_TOKENS` | `750` (only used when `WARMUP_VOICE` is set) |
|
||||
| `WARMUP_TEXT` | a generic filler sentence (only used when `WARMUP_VOICE` is set) |
|
||||
|
||||
Every `*.wav` placed in `/voices` is registered as a cloned voice
|
||||
under its filename stem (e.g. `/voices/freeman.wav` -> voice
|
||||
`freeman`) once `/health` responds. A same-stem `.txt` file (e.g.
|
||||
`/voices/freeman.txt`) supplies that voice's `ref_text` -- the
|
||||
transcript of the reference clip -- which enables higher-fidelity ICL
|
||||
clone mode instead of the x_vector_only fallback used when no
|
||||
transcript is given.
|
||||
|
||||
### Worst-case VRAM warmup
|
||||
|
||||
`ggml_backend_sched` grows its compute buffer to fit the largest graph
|
||||
it has ever built and never shrinks it back, so VRAM use can ratchet
|
||||
up the first time a long prompt or a long reply arrives on live
|
||||
traffic. Setting `WARMUP_VOICE` runs one real synthesis at container
|
||||
startup, capped at `WARMUP_MAX_NEW_TOKENS` frames, to force that
|
||||
worst-case buffer growth to happen up front instead of on a live
|
||||
request. Combine with `--max-prefill-tokens` (`MAX_PREFILL_TOKENS`
|
||||
above) for the input side of the same problem. If the warmup
|
||||
synthesis fails (most likely out of VRAM), the container exits
|
||||
non-zero rather than come up healthy and fail unpredictably later --
|
||||
by design, since a deployment that can't afford its own configured
|
||||
worst case should know that at startup.
|
||||
|
||||
## Building locally
|
||||
|
||||
```
|
||||
git clone --recurse-submodules https://github.com/ServeurpersoCom/qwentts.cpp.git
|
||||
cd qwentts.cpp
|
||||
docker build --target cpu -t qwentts.cpp:cpu .
|
||||
docker build --target cuda -t qwentts.cpp:cuda .
|
||||
docker build --target vulkan -t qwentts.cpp:vulkan .
|
||||
```
|
||||
|
||||
`--target` is required to pick a variant; without it, `docker build`
|
||||
uses the last stage in the `Dockerfile` (`cuda`).
|
||||
|
||||
### Older GPUs (pre-Pascal)
|
||||
|
||||
`docker build` never has GPU device access (unlike `docker run
|
||||
--gpus`), so CMake's CUDA-architecture autodetection has nothing to
|
||||
detect against. This project's own `CMakeLists.txt` already handles
|
||||
that by defaulting `CMAKE_CUDA_ARCHITECTURES` to a fixed Pascal-and-newer
|
||||
list (`61-real;75-virtual;80-virtual;86-real;89-real`, plus Blackwell
|
||||
with CUDA 12.8+) when the variable isn't set, so Pascal cards (sm_61,
|
||||
e.g. the GTX 10-series) work out of the box with no override. GPUs
|
||||
older than Pascal (Maxwell and earlier) still need the architecture
|
||||
passed explicitly:
|
||||
|
||||
```
|
||||
docker build --target cuda -t qwentts.cpp:cuda \
|
||||
--build-arg CMAKE_CUDA_ARCHITECTURES=50 . # Maxwell
|
||||
```
|
||||
|
||||
Find your GPU's compute capability at
|
||||
https://developer.nvidia.com/cuda-gpus.
|
||||
|
||||
### CUDA driver stub at link time
|
||||
|
||||
The CUDA build links against `libcuda.so` (the driver API, used by
|
||||
ggml's VMM pool allocator) at build time even though no driver is
|
||||
present. The `Dockerfile` already points the linker at the devel
|
||||
image's `lib64/stubs/libcuda.so` for this; it's mentioned here only in
|
||||
case you customize `CUDA_BUILD_IMAGE` to a base that ships that stub
|
||||
somewhere else.
|
||||
Reference in New Issue
Block a user