R36SHack/docs/development-status.md

# Development Status

## Current Milestone

Milestone 3 — SDL Video Viewport, HUD, and Wayland Compatibility

## Current Architecture Decisions

- **Language / UI**: Python 3.9+ with PySDL2 (ctypes wrapper around system SDL2), with UI layout metrics now computed from the runtime window/display size instead of being fixed to 640x480
- **DLNA Discovery**: Custom SSDP M-SEARCH implementation using asyncio datagrams + aiohttp for device description XML
- **Content Browsing**: Direct SOAP/XML ContentDirectory client with DIDL-Lite parser (no dependency on async-upnp-client browsing at runtime — only aiohttp)
- **Playback**: integrated GStreamer backend via `PyGObject` / `GstPlayBin`, decoding video into `GstAppSink` frames that are uploaded to SDL textures and rendered in the main SDL renderer
- **Playback viewport**: SDL scales decoded video into a dedicated playback viewport in the same render pass as the HUD, with full-width video bounds and a black playback backdrop outside the viewport
- **Concurrency**: Dedicated asyncio event loop in a daemon thread; thread-safe queues bridge it to the SDL2 main loop
- **Input**: Keyboard mapping for desktop testing + SDL2 GameController for R36S D-pad/buttons
- **Font**: Bundled package font preferred first; system font fallback kept only for development hosts
- **UI icons**: prefer bundled monochrome glyphs or a bundled icon font subset instead of depending on OS emoji fonts
- **Playback HUD**: SDL-rendered overlay is now simplified and compact, uses bundled playback icons, uses a smaller dedicated playback font with title ellipsis for 640x480 readability, supports auto-hide or fixed visibility, stays visible while playback is paused, and remains in the same SDL render pass as video
- **Wayland / DRM strategy**: playback no longer depends on native overlay sinks or X11 window handles; R36S-class targets continue to prefer `kmsdrm` when no display server is present
- **Deploy packaging**: a `conda-forge`-oriented `environment.yml` now defines a reproducible Miniforge/Miniconda environment for local development and release preparation
- **Python packaging**: direct runtime dependency declarations now explicitly include `aiohttp` instead of relying on transitive installation through other packages
- **ArkOS deploy layout**: on-device installs should place Miniforge and the git checkout under `/home/ark` to avoid the full `/roms` partition, while EmulationStation integration should stay lightweight under `/roms/ports`

## Completed Tasks

- Phase 1: Project bootstrap (`pyproject.toml`, `requirements.txt`, `README.md`, package layout under `src/`)
- Phase 2: DLNA discovery (`dlna/discovery.py` — SSDP M-SEARCH, friendly-name fetch) and browsing (`dlna/client.py` — SOAP Browse, DIDL-Lite parser with relative-URL resolution + `dlna/models.py` domain models + `dlna/browser_state.py` navigation stack/cache)
- Phase 3: SDL2 UI (`ui/sdl_app.py` — window, event loop, input dispatch; `ui/screens.py` — server list, browse list, playback, error screens; `ui/theme.py` — runtime-scaled layout helpers for 640x480 and 720x720-class displays)
- Phase 4: Playback (`player/backend.py` abstract interface + `player/gstreamer_backend.py` integrated GStreamer backend)
- Phase 5: Device integration (`platform/controls.py` — keyboard + gamecontroller mapping; `platform/runtime.py` — logging, R36S heuristic, SDL env hints)
- Phase 7: Tests — 75 tests across 7 test files all passing (DIDL mapping, SOAP/XML parser, navigation state, playback backend, SDL redraw policy, input controls, runtime environment setup)
- Desktop runtime verification completed: fixed SSDP discovery socket setup for IPv4 and removed pending-task shutdown noise from the async worker thread
- Packaging hardening: bundled a local UI font asset and configured setuptools to ship it with the package
- Real LAN regression fixed: Browse SOAP parser now handles `Result` elements both with and without a namespace, matching responses from the discovered MiniDLNA/Jellyfin servers
- Playback backend pivoted to GStreamer because libmpv continued to create a separate native window on the desktop host instead of remaining embedded in the SDL UI
- Milestone 2 is now implemented with GStreamer: playback uses `GstPlayBin` plus `GstAppSink` instead of an external player, libmpv, or native overlay sinks
- SDL playback flow updated: decoded GStreamer frames are uploaded into SDL textures, playback end-of-stream returns automatically to the browser, and playback controls support pause/resume, relative seek, and volume
- Milestone 3 implemented in code: SDL scales video into a dedicated viewport inside the SDL window, with reserved HUD margins instead of using the whole window area for video
- Playback HUD expanded: progress bar, elapsed/duration, volume, buffer, resolution, and control legends are rendered around the video area and updated from GStreamer bus/pipeline queries
- Playback-page flashing root cause addressed by removing native overlay composition entirely: video and HUD are now rendered together by SDL in one pass, with redraws driven by decoded frame availability and HUD state changes
- Playback HUD simplified: the border around the video area was removed, playback control/status icons were added as bundled SVG+PNG assets, the title/timer top bar no longer overlaps, and playback now supports `auto / fixed / hidden` HUD modes through a dedicated command while staying visible when paused
- UI scaling hardened for mixed small-display targets: list rows, HUD bands, icon sizes, viewport margins, and font sizes are now derived from the actual SDL window/display size so the app remains readable on both 640x480 and 720x720 screens
- Deployment assets added: `.gitignore`, `environment.yml`, and a real `LICENSE` file so the project can be initialized and published as a clean git repository
- Conda environment refreshed for current playback needs: runtime now includes GStreamer codec/plugin packages plus explicit Python build/test tooling, while editable install keeps the package code sourced from the repo checkout
- Packaging fix: `pyproject.toml` now uses a valid TOML `[project.urls]` table so editable installs work with modern `pip` / `tomllib`
- Copilot instructions and this status file
- Device deployment reconnaissance completed on a real ArkOS-derived R36S over SSH: `/roms` is full, `/home/ark` has free space, required download tools are present, and `/roms/ports` plus `gamelist.xml` are the least invasive integration points for launchers

## Tasks In Progress

- Verify that the SDL-texture playback path is smooth enough on real host playback and on R36S hardware
- Measure whether BGRA frame upload is acceptable on RK3326 or whether a future YUV texture path is needed
- Device deployment on the physical R36S is now wired through ArkOS `Ports -> MatHacks`, with the heavy runtime under `/home/ark` and only a lightweight stub launcher under `/roms/ports`
- Device env bootstrap on the physical R36S reaches a clean `from r36s_dlna_browser.app import Application` inside `/home/ark/miniconda3/envs/r36s-dlna-browser`
- ArkOS launcher asset added at `deploy/arkos/MatHacks.sh`; current launcher uses the `/home/ark/R36SHack` checkout plus verified `LD_LIBRARY_PATH`, `GST_PLUGIN_PATH`, and `LD_PRELOAD` exports needed to load the system `gstreamer1.0-libav` plugins from the conda runtime
- **Rockchip MPP hardware decode now deployed**: `librockchip-mpp` and `gst-mpp` compiled from source via Docker QEMU (arm64v8/ubuntu:focal), installed on device, `mppvideodec` confirmed visible to GStreamer. The `gstreamer_backend.py` probe auto-boosts `mppvideodec` rank if `/dev/vpu_service` is accessible.
- **Pre-built .so files bundled** in `deploy/arkos/mpp-libs/`; `setup_hw_decode.sh` installs them automatically without network access.

## NV12 Render Path Benchmark Log

All runs performed on the physical R36S (RK3326, 4× A35 @ 1.3 GHz, 1 GB RAM) over SSH.
Stream: 1920×1080 H.264 MKV @ 24fps via MiniDLNA over LAN. Frame budget: 41.7 ms.

| Commit | Copy strategy | Copy mean | Copy % budget | FPS | Dropped | A/V drift |
|--------|--------------|-----------|---------------|-----|---------|-----------|
| `a201594` | `extract_dup` → bytes + `from_buffer_copy` → ctypes (2 copies, 6 MB/frame) | 36,499 µs | 87.6% | 24.01 | 1 | −42.8 ms |
| `da02e74` | `buffer.map(READ)` + `memmove` into reusable ctypes array (1 copy, 3 MB/frame) | 33,551 µs | 80.5% | 23.98 | 0 | −38.0 ms |

**Key observations (`da02e74`):**
- 1 dropped frame eliminated (0 vs 1)
- Jitter improved: stdev 2.6 ms vs 3.6 ms
- A/V drift tighter: −38 ms vs −43 ms
- Copy cost still 80.5% of frame budget — the 3.1 MB `memmove` on each frame is the remaining bottleneck
- Further reduction requires DMA-buf zero-copy (kernel VPU→SDL import without CPU memcpy), which depends on device driver support not currently available through gst-mpp's appsink path

## Blockers Or Open Questions

- `SDL2_ttf` system library needed for text rendering (`sudo dnf install SDL2_ttf` on Fedora, `sudo apt install libsdl2-ttf-2.0-0` on Debian/Ubuntu). The app handles its absence gracefully but will show no text.
- Integrated playback requires system GStreamer plus Python GI bindings (for Fedora: `python3-gobject gstreamer1 gstreamer1-plugins-base gstreamer1-plugins-good`; add codec/plugin packages as needed for target media).
- Root browse verified against two real DLNA servers on the LAN.
- On-device testing on R36S hardware is pending.
- The current SDL-texture path avoids window-manager dependencies but may still need optimization on low-end hardware if BGRA upload cost is too high.
- The first Miniforge install attempt on the physical R36S failed because the downloaded installer was corrupt and crashed during extraction.
- The physical R36S now has Miniconda installed at `/home/ark/miniconda3`; the dedicated app env exists at `/home/ark/miniconda3/envs/r36s-dlna-browser`, but package solves can hang on-device and are being handled incrementally.
- The dedicated R36S conda env requires `LD_LIBRARY_PATH=/home/ark/miniconda3/envs/r36s-dlna-browser/lib` for GI and GStreamer shared libraries to resolve correctly.
- GStreamer imports now succeed in the dedicated env (`GLib`, `GObject`, `Gst`, `GstApp`, `GstVideo`), and `Application` imports cleanly.
- ArkOS menu launch works on the physical device, and DLNA browsing reaches real MiniDLNA content.
- Real playback is currently blocked by missing decoder elements in the device env: direct probing of a MiniDLNA `.mkv` URL showed missing H.264 High Profile and MPEG-4 AAC decoders, while the user-facing "can't play a text file" message is a misleading fallback caused by an additional text stream in the container.
- **RESOLVED**: `gst-libav` conda package on `linux-aarch64` has an unfixable ABI mismatch: `libavcodec.so` links `libdav1d.so.6` (from dav1d <1.3) but only dav1d 1.4.x (`.so.7`) is available, and via `libxml2-16` it also pulls `libicuuc.so.78` which is not packaged for linux-aarch64 on conda-forge. **Solution**: install system `gstreamer1.0-libav` (v1.16.1) via apt and use `GST_PLUGIN_PATH` + `LD_PRELOAD` to expose its plugins to the conda Python runtime.
- On the physical R36S, `avdec_h264` and `avdec_aac` now register and resolve when launched with:
  `LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1 GST_PLUGIN_PATH=/usr/lib/aarch64-linux-gnu/gstreamer-1.0`
  The `LD_PRELOAD` is required to avoid "cannot allocate memory in static TLS block" from the conda `libgomp.so` being loaded late by dlopen.
- These variables are now persisted in `/home/ark/miniconda3/envs/r36s-dlna-browser/etc/conda/activate.d/gst-env.sh` and explicitly set in `deploy/run.sh`.

## Next Recommended Actions

1. Consider profiling what the remaining 19.5% of frame budget (≈8 ms) consists of — likely SDL_UpdateNVTexture upload + render call overhead + Python GIL churn. If SDL upload is the bottleneck, try `SDL_LockTexture` for direct write instead.
2. Investigate DMA-buf / dmabuf fd import as a future zero-copy path: gst-mpp may expose DRM DMA-buf fds that SDL's KMSDRM backend can import directly, eliminating the CPU memmove entirely.
3. Run a visual playback smoke test on device directly via the app launcher (MatHacks.sh) to confirm HUD and video render correctly together under KMSDRM at the current 80.5% copy load.
4. If 80.5% copy cost causes visible stutter under load (UI overhead competing for A35 cycles), the next option is to reduce resolution at the appsink by inserting a `videoscale` element to 1280×720 before the appsink, cutting memmove to ~1.3 MB/frame (≈35% budget).
5. `avdec_hevc` is still missing (HEVC decoders not in system apt `gstreamer1.0-libav 1.16.1`); `mppvideodec` covers H.264/H.265/VP8/VP9 via HW so this is less critical now.