# Release plan: command-stream viewer Status notes for taking the command-stream pipeline to production as the primary viewer, with the existing AWS video-streaming pipeline as fallback. Updated 2026-08-20. ## London CPU server (LIVE) Box: 82.39.133.153 = london-cpu-worker.war3replays.com (Cloudflare DNS-only A record; unproxied on purpose - WebRTC UDP cannot ride the CF proxy). Ubuntu 24.04, 4 cores / 8 GB / 192 GB, KVM (CRIU-capable). Deployed and verified streaming a live 1.28.5 session end to end on 2026-08-20. - Access: ssh danny@... (key only, passwordless sudo). Root login and password auth disabled. ufw: 22, 80, 443, 8145, 8150/tcp + 40000-40199/udp. fail2ban (sshd), unattended-upgrades, 5 GB swap total. - Service user `ubuntu` (uid 1001, no sudo, linger on) runs everything; paths mirror the AWS lab exactly (/home/ubuntu/w3cs-lab, /opt/war3-runtime). NOTE for future boxes: create `ubuntu` BEFORE any admin user, or rely on the now-parameterized SERVICE_UID in classic-criu-worker.sh. - Stack: wine-staging 11.13 (pinned + apt-mark hold, matches lab), full runtime (war3-1285/1140/rk, classic-131 CASC, prefix), engine profiles, 12 staged replays with sidecars, map corpus. - systemd: w3cs-live (relay + warm seat, Restart=always), w3cs-web (8150), w3cs-watchdog.timer (30 s Xvfb/relay liveness -> restart; fixes the lab's recurring dead-Xvfb-behind-live-relay failure), nginx, certbot.timer. - TLS: Let's Encrypt on nginx, single origin routing /signal|/reliable| /frame|/control|/resource-bundle -> relay :8145, everything else -> no-store page server :8150. Secure context is REQUIRED off-box (WebGPU and crypto.randomUUID do not exist on plain-HTTP non-localhost origins). interactive-live.html now derives ws/http bases from the page origin. - Test page: https://london-cpu-worker.war3replays.com/poc/cpu-test-index.html (replays.json generated at deploy from replay sidecars, grouped by patch). - Deploy: experiments/d3d9-command-stream/deploy/deploy-cpu-server.sh (rsync app + infra, remote build with -Werror, units, nginx, cert paths). - Replay source: staged files only. R2 on-demand fetch is the next work item (with catalog metadata mapping sha -> engineProfile + map, like the staged sidecars). No AWS credentials live on this box. ### Second London worker + worker pool (2026-08-21) Box 2: 82.39.133.12 = london-cpu-worker-2.war3replays.com (Cloudflare DNS-only A record, same rules as box 1). Ryzen 9 7900 (box 1 is a 9950X), 4 cores / 8 GB / 192 GB, Ubuntu 24.04, KVM. Provisioned as an exact sibling of box 1 by `deploy/provision-cpu-server.sh root@NEW_IP` (phases seed, base, sync, sync-finish, harden; each idempotent): - seed: box 1's config files (WineHQ key + sources, sshd hardening drop-in, fail2ban jail.local, sysctl 99-war3.conf, 20auto-upgrades, danny's authorized_keys) copied through the developer machine. - base: timezone, apt full-upgrade, i386 + WineHQ repo, box 1's manual package set, wine-staging 11.13~noble-1 pinned and held, users with box 1's uids (danny 1000 admin NOPASSWD sudo, ubuntu 1001 service + linger), 4 GiB swapfile, sysctl, fail2ban, unattended-upgrades, ufw with box 1's rule set. Ends by verifying the admin login + sudo from the developer machine. - sync: root-to-root rsync box 1 -> box 2 over a temporary ed25519 key (created for the transfer, deleted by sync-finish on both sides): /opt/war3-runtime (24 GB; the CRIU claim images are EXCLUDED because they are CPU-specific - box 2 builds its own matrix at deploy), /home/ubuntu (lab, replays + sidecars, engine profiles, bundle spool, war3-repo, pulse config), /usr/local/go, /usr/local/libexec/war3 (criu), /etc/w3cs (rclone R2 keys). Same-DC transfer ran at ~450 MB/s. - harden: 00-war3-hardening.conf (PermitRootLogin no, no password/kbd auth, MaxAuthTries 4, no X11) installed through the admin login, then a root login is attempted and must fail. - Then: `W3CS_CPU_HOST=danny@82.39.133.12 W3CS_CPU_DOMAIN=london-cpu-worker-2.war3replays.com deploy/deploy-cpu-server.sh`. The deploy is now per-box: DOMAIN drives nginx's server_name (sed over the repo config) and a first-deploy certbot run (webroot, no contact email like box 1, deploy hook reloads nginx). The per-IP viewer gate allowlist was created by hand with the developer address, as on box 1. Lessons baked into the scripts: ssh joins its arguments into one command string that the remote shell re-parses (quote with printf %q, flatten lists to one line); a script started by nohup needs a shebang or dash runs it; create destination parents before rsync. Worker pool (web/worker-pool.js, unit-tested): every box serves /workers.json (written by the deploy from W3CS_WORKER_HOSTS, same list on every box). The relay's /seatN/status carries Access-Control-Allow-Origin: *, so a page on any worker counts the free seats of all workers. - Policy: a new viewer goes to the worker with the MOST free seats; ties break at random. With equal pools this alternates arrivals and keeps the boxes within one viewer of each other with no shared state. A full or unreachable worker never wins. - cpu-test-index.html: links are re-pointed to the best worker every 8 s (new-tab/middle-click use) and re-probed on a plain click (viewers who arrive together must not follow one stale count). The status line shows per-worker free seats. - interactive-live.html: a deep link that lands on a full worker moves once (location.replace, ?hop=1) to the worker with the most free seats; the "All seats are taken" waiter polls its own origin first, then the other workers, and moves towards the first seat it sees free. Signaling, the WebTransport bridge and the per-IP gate are all same-origin, so the move is a full navigation, never a cross-origin connect. - PROVEN LIVE (2026-08-21, both boxes, 5 seats each): six viewers arriving one by one from the index landed box1, box2, box2, box1, box2, box1 - the seat map after each arrival was 1/0, 1/1, 1/2, 2/2, 2/3, 3/3 (never more than one apart), all six games rendering. With box 2 made full (five held signaling sockets) the index on box 2 itself pointed every link at box 1 ("london-cpu-worker-2 0/5 free") and a deep link aimed at box 2 moved itself to box 1 (?hop=1) and played. Box 2's first real session also proved the CRIU claim path there (40/40 images built on the box after the missing /usr/local/lib/war3-* gate libraries were copied; that path is now in the provisioning sync list). - The per-IP gate stays per worker (nginx on each box): one visitor can hold one seat per worker. Fleet-wide "one replay per IP" is the catalog's view lease once the catalog routes here. ### Viewer gate: one live replay per public IP (2026-08-21, live) Same policy as the AWS video workers (`W3_MAX_PER_IP=1` in streaming/server.py), applied on the London box so one visitor cannot take every seat by opening five tabs. - Where: nginx, not the relay. A seat is busy exactly while its relay holds an open `/signal` WebSocket (that socket IS the seat claim; the WebTransport and standby planes cannot hold a seat without it, and the relay's idle reaper closes it when a viewer vanishes). Every seat is its own relay process and the page picks its own seat, so nginx is the one place that sees all seats. `deploy/nginx-w3cs.conf`: `geo $w3cs_viewer_key` + `limit_conn_zone w3cs_viewer` + `limit_conn w3cs_viewer 1` on `/seatN/signal`, the legacy `/signal`, and the `/viewer-gate` probe only. Status probes, data planes and bundles stay unlimited. Refused sockets get HTTP 429 (`limit_conn_status`). - Page: `/viewer-gate` is a plain request accounted in the same zone, so 429 means "this address already holds a seat". interactive-live.html asks it before the seat search (with one 1.5 s retry, so a reload's dying socket does not count against its own successor) and again after a signaling socket fails to open, then shows "Please watch one replay at a time" and polls every 5 s, reloading when the gate clears. Lab http origins skip the probe. - Allowlist: `/etc/nginx/w3cs-trusted-ips.conf` (box-local, not in the repo; `deploy/w3cs-trusted-ips.conf.example` documents the format). Each listed address or CIDR maps to an empty key, which limit_conn does not account, so developers can open several tabs/browsers. The deploy script only guarantees the file exists. Loopback is always exempt. The developer connection was added on 2026-08-21; edit + `nginx -t` + reload to change. - Gotcha (found live): a `return 204;` probe answers in the REWRITE phase, before the PREACCESS phase where limit_conn counts, so it passed beside a held socket. The probe uses `empty_gif` (a content handler) for that reason. Do not "simplify" it back to `return`. - Verified live from an untrusted address: second socket on another seat 429, legacy `/signal` 429, gate 429 over h1.1 and h2, seat status/page 200 throughout, gate clears within ~1 s of the socket closing; two browser tabs: tab 2 shows the busy card and takes no seat, starts on its own after tab 1 closes. With the address trusted: gate 200 beside a held socket, three concurrent handshakes all 101. Tests: test_viewer_gate.py. - Open release steps for the gate: 1. Close the bypass. ufw still allows 8145/tcp and 8150/tcp from anywhere and the relays bind 0.0.0.0, so `ws://:8145/signal` reaches seat 1 without nginx. Either drop both ufw rules (nothing public needs them since nginx/TLS went in; 8146-8149 are already blocked) or make the relays and page server bind 127.0.0.1. The wt-bridge on 4443/udp must stay open; it cannot claim a seat by itself. 2. Decide the public copy/UX once the catalog routes real visitors here (the catalog already has its own per-IP view lease for the AWS path; one visitor should see one consistent message across both). 3. IPv6 visitors are keyed per address, as on AWS. A /64-keyed variant is a one-line change in the `geo` block if privacy-extension rotation ever becomes a problem. ### Multi-seat (2026-08-20, live) Three independent seats on the London box, each a full stack (Xvfb :1N + relay :814N + own wine prefixes + pulse sink + capture/control files), run as systemd template instances `w3cs-live@{1,2,3}` via deploy/w3cs-seat-launch.sh. Any seat runs any replay/profile: profiles resolve from replay metadata at session start; seats are homogeneous. - Allocation: the live page probes same-origin `/seatN/status` (nginx maps /seatN/* -> 814N; no extra ports, no CORS), connects to the first free seat, and retries on a lost race. The relay REFUSES a second viewer (ws close 4001 "seat-busy") - no session takeover. All seats busy -> the page shows "All seats are taken" and polls every 8 s, starting automatically when a seat frees (verified live end to end). - Seat freeing: viewer disconnect parks the engine (SIGSTOP, zero CPU) and frees the seat; replay end (page finishReplay) parks AND drops the viewer connection (close 4002 "replay-ended") so an open end-card tab cannot hold the seat. The page suppresses its auto-restart machinery once replayEnded is set. - Old-engine resolution SOLVED: pre-1.28 engines size their outer window to exactly 0.75 x desktop and use its client area (frame 8x34) as the D3D backbuffer; they ignore reswidth/resheight in windowed mode, and unknown -width/-height args break their -loadfile parsing (strands at the menu). Seat Xvfb roots are therefore 1376x1070, which makes the old engines pick 1024x768 themselves (verified: 1.14 source 1024x768, bandwidth 10.2 -> 5.5 Mbps). 1.28.5 uses the registry stamp as before. 1.31 becomes 1032x802 on this root (from 1440x1080); exact 1024x768 for it is a follow-up (it may accept explicit size args - untested). - Throughput observed: 3 concurrent games ~= load 2.0 on 4 cores, ~1.8 GiB used; 3 seats is conservative for this box. - Concurrency watch item: one simultaneous triple cold-start showed 1.28.5's "User Data Migration Required" dialog on one seat (shared-install race); staggered starts have not reproduced it. ### Client render worker (2026-08-20, live) The whole client stack (zstd frame decode, command state, texture uploads, WebGPU submit) now runs in a dedicated worker with an OffscreenCanvas, and the reliable/frame/recovery RTCDataChannels are TRANSFERRED INTO the worker directly, so the stream path (DataChannel -> zstd -> parse -> WebGPU) never touches the main thread at all. Main keeps UI, input, signaling, and the control channel only. Mechanics that matter for maintenance: - Chrome only allows transferring an RTCDataChannel synchronously inside its ondatachannel event, before any traffic. The worker is therefore spawned BEFORE connect and must pass a WebGPU probe first (a broken worker would otherwise strand the transferred channels with no fallback). Local sentinels mirror the worker-reported channel state. - Tiered fallback, per plane: direct transfer -> zero-copy ArrayBuffer byte forwarding (used automatically where transfer is unsupported, and for the WebSocket standby plane) -> full in-page path (?worker=0, ?debugDraws=1, ?parity=1, spawn/GPU-probe failure). - Also fixed alongside: WebGPU device loss now feeds the auto-restart machinery instead of dying as an unhandled rejection (the observed stuck-at-1-FPS state), and the render target is capped at the canvas's displayed device pixels (a 2x-DPR screen rendered 2560x1920 into a 2048x1536 display - +56% GPU work for nothing). - PROFILED (2026-08-20, live 90 s steady-state, instrumentation now permanent in transport stats: decodeMs/parseMs/consumeMs): zstd decode 0.83 ms/frame + parse 0.13 ms/frame vs consume (state apply + draw encode + WebGPU submit) 6.5 ms/frame. WebGPU CPU dominates decode by ~7x, so the two-pipelined-workers split would save at most ~1 ms/frame - NOT worth it (the advisor's own condition, decode rivaling WebGPU time, is clearly false). SharedArrayBuffer ring likewise unnecessary. If more client headroom is ever needed, the target is the 6.5 ms consume path (draw encoding), not the decode side. - Hotspot pass (2026-08-20) on that consume path, all fidelity-identical and live-verified: copy-on-write state snapshots replaced four full Map clones per draw (SnapshotMap in command-state), and snapshot identity now drives renderer memoization (pipeline/sampler/fidelity-check/MVP skip on unchanged state), fvfLayout is memoized, the encode loop skips redundant setPipeline/setVertexBuffer/setScissor/setViewport/setIndexBuffer calls and reuses a scratch dynamic-offset array, and updateTexture no longer wipes the whole bind-group cache (keys embed the texture generation, so stale entries are simply never hit; growth is bounded instead). Result: frame consume 4.97 -> ~4.0 ms/frame (-15-20%, phase-dependent) and thousands of per-frame allocations removed (less GC jitter). The remaining ~8 us/draw is spread across record dispatch, ~90 uniform slot writes, and the irreducible per-draw setBindGroup+draw calls; next levers if ever needed: a per-snapshot uniform template (one 376-byte copy instead of ~60 Map gets per draw), and a real worker flame graph via chrome://inspect before any further blind tuning. - Inner frame-path profile (permanent sub-buckets rendererDrawMs/BufferMs/ EndFrameMs in transport stats): within frame consume, record dispatch + state snapshot building in command-state BEFORE the renderer is ~62%, renderer.draw JS (uniform slot fills) ~27%, the deduped encode loop ~7%, new-geometry buffer uploads ~3%, queue.submit ~0.2%. So the next target is the command-state apply loop (per-record u32s/viewOf allocations and the _draw object build), not more WebGPU-call tuning. - zstd decode is NOT render-worker CPU: it already runs as WASM in dedicated per-plane sub-workers (web/vendor/zstd-wasm) - the measured decodeMs is await time. Known latency waste there for a future pass: every frame decode copies AND re-transfers its ~30KB dictionary to the zstd worker (protocol.js decompressZstd) plus a defensive input copy; caching decoded outputs worker-side by frame number would remove both. - Resource apply is mostly the texture path; on real GPUs BC ("texture- compression-bc", present on the user's metal-3 adapter) uploads DXT compressed with zero JS decoding - the JS decodeTexture loop only runs for 16-bit formats. - Second optimization pass (2026-08-20, live-verified, 21/21 tests): 1. Dispatch path: one WeakMap-cached full-buffer DataView per message replaces a DataView allocation per record, and every hot record branch reads fields directly (no u32s() array per record). 2. Uniform template: all state-derived uniform slots build once per state change into a cached Float32Array and copy wholesale into each draw's slot; per-draw work is the template copy + matrices + texture flag. Also removed beginFrame's whole-arena uniformData.fill(0). Renderer draw JS: 1.92 -> 1.22 ms/frame (-36%) like-for-like. 3. zstd dictionary-by-reference: the zstd sub-worker caches decoded recovery/anchor outputs by frame id (retainId) and chained decodes name their dictionary instead of re-shipping the decoded-anchor bytes (~60-130 KB) every frame; worker restart or eviction self-heals via a retained-set on the protocol side plus a one-shot resend on "dictionary-miss". Decode await: 0.77-4.3 -> ~0.6 ms/frame. Aggregate on a like-for-like heavy window: total client CPU ~8.3 -> ~5.7 ms/frame (-31%). ### End-to-end FPS pipeline instrumentation + the low-FPS root cause (2026-08-20) Question answered: "the server sends N FPS — where do frames disappear?" Instrumentation (permanent): - Relay logs one `native pipeline` line per 5 s while a viewer is live: capture/s (game Presents entering the relay), sent/s, dropped/s (app-queue latest-wins), sendFail, governor target + effective fps, frame buffer limit, bufPeak (max bufferedAmount between ticks — 5 s averages hid the spikes that trigger backpressure), wire Mbps, capacity/peak estimates, encoded frame KiB, and the client's reported recv/arrive/draw fps. One line shows every stage. - Client counts `framesArrived` (completed wire reassembly, in transport stats) and `framesDrawn` (renderer consumed, on the client), reports `arrivedFps`/`drawnFps` per second in the network report, and shows "net N / draw N fps" in the HUD. Root cause of the user-visible lag (10-20 fps, freezes, 1-fps wedge), measured live on both a 1285 and a 1311 replay from the real viewer path: - The client was NOT the bottleneck: draw == arrive in every window. - A normal frame whose reliable-channel resources were >500 ms late set awaitingRecovery AND requested a full epoch recovery — but the reliable channel is ordered and lossless, and the relay only sends a frame after its resources were already sent, so that lag is transient in-flight delay (bursty texture updates share the SCTP association; one session measured 109.6 MiB texture updates vs 76 MiB geometry on reliable). Each recovery re-sends the epoch (MBs) on the lagging channel: self-amplifying. Measured: 15 recoveries per ~150 s, all reason=dependency; repeated 5-25 s windows of capture=19-37/s vs sent=0.2-0.4/s with an EMPTY channel (awaiting_recovery drops all normal frames); one live 1-fps wedge (arrive 37.9/s, draw 1/s, dictionary drops 76) that self-healed after ~40 s. - Second cause: single-burst bufferedAmount spikes (one 60-90 KiB anchor+frame while usrsctp fragments it) crossed the 3/4 backpressure line for ~100 ms and triggered multiplicative cuts (36 -> 18.8 fps) while the client was drawing every frame that arrived. Fixes (deployed + verified 2026-08-20): - live-transport: dependency timeout on a normal frame now drops only that disposable frame ("dependency-wait", stats.dependencyWaits) and waits for the sequence to catch up; escalates to a recovery only when the reliable channel makes NO progress across 8 consecutive timeouts (real stall, e.g. SCTP zero-window) or when the blocked frame is itself a recovery. - Page: dependencyTimeoutMs 500 -> 1500 (both worker and in-page paths). - Relay: backpressure pressure_hold max(0.10, rtt*0.75) -> max(0.30, rtt*2.0) so one-frame fragmentation spikes stop cutting the rate; real congestion still sustains pressure and cuts within ~0.5 s. Verified on the same 1285 replay that wedged: dependency recoveries 15 -> 0 (3.5 min), zero-send windows gone (sent tracks capture every tick, drops <=0.6/s), dependencyWaits=0 (all previous "dependency" events were sub-1.5 s transients), draw == arrive throughout, governor climbing 18 -> 28+ fps with a healthy capacity estimate instead of sawtoothing. Remaining, expected: rare dictionary drops from anchor loss on the unreliable plane (~250 ms GOP blip, self-heals at the next anchor; escalation now needs 8 consecutive no-progress events). Next levers if more FPS is wanted on heavy replays: effective SCTP throughput (~9-15 Mbps observed on a 40 Mbps-clean path) caps 30-44 KiB frames below 40 fps; texture-update volume on the reliable channel is the burst source (109.6 MiB/session). ### Research: the road to 40 fps in large fights, then 60-100 fps (2026-08-20) Measured on the real seat-1 capture (native-1311 heavy replay, 4,995 frames, 782 MB): plain frame mean 117 KiB, p95 142 KiB, max 421 KiB. Post-zstd ablation (chained, matching the wire scheme) over a 400-frame heavy window: - GEOMETRY DELTAS ARE 91.3% OF THE WIRE. Transforms 1.0%, draws 8.2%, render/sampler/texture state 1.7%. Transforms repeat the previous frame exactly 95.9% of the time and states 93.7% — chained zstd already eliminates them. Optimizing anything but geometry is pointless. - Frames already ship at zstd level 7 with previous-frame dictionaries. - FLOAT16_DELTA internals (194 MiB plain): raw exact bytes 37.2% (positions are bit-exact by design — the old half-precision rounding made buildings vibrate), half values 32.4%, RAW MASK 22.7% (a fixed 4 bits/word regardless of how much changed), float mask 5.7%. Delta ratio vs referenced blobs is only 55% — weak, because disposable frames delta against the epoch/anchor base (up to 5 frames old), not the previous pose. XOR fallback adds 330k records / 111 MiB (small buffers: UI quads, glyphs, index data). Ranked plan (nothing implemented yet): 1. Tier 1 — shrink geometry deltas ~40-60% with existing machinery: (a) dependents delta against the PREVIOUS pose (1-frame base) instead of the anchor — smooth animation makes 1-frame residuals ~5x smaller; GOP loss behavior worsens only to "rest of the 5-frame GOP" (=today's anchor-loss cost). (b) error-feedback quantized position deltas (16-bit, sub-pixel at map scale): the encoder already interns its own RECONSTRUCTED blob as the next base, so encoder and client stay bit-identical and the vibration bug cannot return. (c) dense-change mask escape ("whole buffer changed" flag) to reclaim the 22.7% rmask. Expected heavy frame: ~37 -> 18-22 KiB => 40 fps in fights at 6-8 Mbps, inside today's SCTP ceiling. 2. Tier 2 — felt 60-120 fps at unchanged bandwidth: client-side pose morphing. Consecutive poses of a slice are already linked on the wire (delta base_id); bind both vertex buffers and lerp in the vertex shader, interpolate world transforms + camera between states. Display Hz decouples from state Hz (ship 30-40 state fps, render at display rate). Costs one state interval of latency (~25 ms at 40 fps) — fine for replays; for live gameplay use extrapolation on camera/cursor. 3. Tier 3 — transport for real 60-100 STATE fps: WebTransport (QUIC). One unidirectional stream per frame, cancelled on supersede = latest-wins with zero head-of-line blocking, BBR-class congestion control, no usrsctp ~10-15 Mbps practical ceiling. Chrome/Edge/ Firefox stable; certs already provisioned. Replaces the whole ICE/DTLS/SCTP data-plane stack for data (audio can stay WebRTC). 4. Tier 4 (endgame, 10x+): semantic capture — intercept bone palettes instead of post-skinned vertices (a unit = ~30 bones vs ~2,000 verts). Requires reverse-engineering WC3's CPU skinning per patch; only if Tiers 1-3 are not enough. Also required for 100 state fps: the game must Present at that rate (~1.2 cores/seat estimated vs ~0.5 today — seat-density tradeoff), which is why Tier 2 (display-rate interpolation) is the pragmatic 100 fps. ### Tier 1 trial outcome (2026-08-20): implemented, regressed live, now flag-gated OFF Per-frame chaining + quantized position deltas were fully implemented (proxy encoder opcode 35, relay chain-integrity queue rules + per-frame zstd dictionaries, client decoder + retention; 21/21 tests). The live trial on the heavy 1311 replay sustained 31-35 fps (vs 19-28 before) in a healthy session, BUT the user's session regressed hard: UI elements visibly jumping and laggy mouse feel — GOP-granular chain drops turn isolated frame losses into bursty multi-frame gaps, which reads far worse than evenly-paced lower fps — and a session that connected during game boot fell into a recovery-watchdog storm (30 recoveries sent, 3 acked, reliable queue pinned full at 4.4 MiB, zero frames for minutes). Everything is now gated behind W3CS_TIER1=1 (both proxy and relay read it; default OFF everywhere). The client decoder and retention changes are compatible supersets and stay enabled. Before re-enabling, in order: (1) reproduce and fix the boot-connect recovery storm on a lab seat; (2) smooth the chain-drop failure mode (e.g. shorter GOP under loss, or dependents that can also decode against the anchor as a degraded fallback); (3) verify felt smoothness side-by-side, not just fps counters; (4) the relay blob-ownership invariant checker needs chain awareness (it false-alarms under tier 1). ### Quant-only lab result (2026-08-20): -47% encoded frames, pixel-stable, on seat 3 W3CS_QUANT=1 enables ONLY the quantized-position encoder (opcode 35) — no per-frame chaining, so dependents delta against the anchor exactly as the classic build and drop behavior is unchanged. Live A/B on the same 1311 replay window (fresh sessions, seat 1 classic vs seat 3 quant): frame plain 122.5 -> 79.3 KiB (-35%), encoded 38.6 -> 20.6 KiB (-47%), geometry plain 66.5 -> 32.4 KiB. Zero recovery storms, zero invariant failures, screenshots pixel-correct and static scenes stable across 15 s (the fixed 1/64 absolute step bounds temporal jitter at ~1/300 px, unlike the old relative-error float16 vibration). Caveat: in the replay's extreme fight phase (~min 4+) BOTH builds collapse to ~8 fps with ~90-100 KiB frames and a texture-update flood (~100 KiB per animated-texture update; 124-230 MiB per session) that congests the shared association — quant does not help there because the bytes are not position deltas. Texture-update delta/compression is the next byte lever; transport (Tier 3) the other. HUMAN FEEL-TEST VERDICT (2026-08-20, same night): REVERTED. Screenshots and counters passed, but the user found real artifacts within minutes: - WC3's drawn cursor reappeared as the cyan targeting reticle: the client's cursor-atlas suppressor EXACT-MATCHES the cursor quad's bytes, and quantization shifts them sub-pixel, so the match fails. - UI elements visibly shifted (a construction progress-bar fill sat several pixels off its frame; text clipped) — paired UI quads are RHW screen-space content where downstream consumers assume exactness. - Movement felt spiky/slower. SECOND TRIAL (same night): world-only via FVF (XYZRHW -> exact classic record) FAILED IDENTICALLY — cyan reticle, shifted menu elements, minimap drag square off. Root cause finally understood: WC3 does not use XYZRHW for its UI at all. The ENTIRE UI — menus, tooltips, minimap square, and the cursor quad — is XYZ geometry on a tiny 0..0.8 coordinate plane (ortho projection), while the world uses coordinates in the thousands. An FVF-based world/screen split therefore does not exist, and the absolute 1/64 step is sub-pixel for the world but snaps UI to a ~2%-of-screen grid. Both flags are now OFF on every seat. Retry preconditions (in addition to the earlier ones): - Discriminate by PER-SLICE COORDINATE MAGNITUDE, not FVF: quantize only slices whose position span indicates world scale (e.g. max|xyz| >= 16); UI-plane and cursor content stays bit-exact. - Validate with offline pixel-diff (parity-style ground-truth compare) including UI-heavy scenes and the cursor path, before any live seat. Lessons: screenshots of static scenes do not catch exact-match breakage or motion feel; the user is the feel oracle; and verify engine-specific rendering assumptions (UI coordinate space) from the captured data before building on them. THIRD PASS (same night): magnitude gate implemented and validated live on seat 3. The encoder scans each slice's position words; peak |xyz| < 16 (WC3's whole UI plane is 0..0.8) or non-finite -> byte-identical classic float16 record; only world-scale slices use opcode 35. Gated capture: 72,901 quant records, delta ratio 38.8% (vs 55% classic), plain frames -20% on the same window, UI pixel-aligned in screenshots. Seat 3 runs the gated encoder awaiting the user's feel verdict; seats 1-2 classic. SEPARATE BUG FOUND AND FIXED during validation: the "cyan targeting cursor" was NOT a quant artifact — the classic build showed it too on the native-1311-d3d9 lane. Root cause: cursor-atlas DETECTION was checksum-based against 1.28 atlas bytes; 1.31 decodes different bytes, so suppression never armed on 1311 and the new hover-sync exposed the drawn reticle. Fix (client-only, all seats): registered the observed 1.31 undead atlas checksums AND added a structural fallback — a 256x128 sheet with >=30% magenta-colorkey/transparent background and >=40 pure-cyan reticle pixels is a cursor atlas regardless of patch or race; a structural match registers its checksums and rescans earlier textures so every copy suppresses. Verified live on 1311: reticle gone (the very first structural match fired on a texture id and checksum different from the registered one, proving the fallback necessary). ### Texture-update sparse XOR deltas (2026-08-20): -85% recurring texture wire, live on lab seat 1 Research first (analyze_textures.py over real captures): the recurring reliable-channel texture flood is fog-style content whose consecutive uploads are nearly identical (one 256x256 texture measured 99.97% unchanged: ~124 KB zstd'd full vs 108 BYTES as a sparse delta). The bootstrap snapshot is separate (one-time, blob/bundle-cached). Implementation (W3CS_TEXDELTA=1, lab seat 1): - W3CS_OP_UPDATE_TEXTURE_XOR (36): full-level sparse XOR (4-bit/word mask + changed bytes) against the browser-held content. The proxy keeps a per-level sent_shadow mirror, refreshed after every emitted update (after any emission the browser level equals mip_shadow, so the mirror is a plain copy). Mirrors drop at snapshot boundaries. - NO-DRAWBACKS DESIGN: XOR is lossless; the record carries base and result CRC-32s. The client verifies the base CRC before applying (XOR is not idempotent -- replay/desync is rejected, tested), applies into a copy, verifies the result CRC, only then commits. Any mismatch raises the existing repairable state error; every 32nd update ships full as a self-healing keyframe. Delta emitted only when smaller. - Bug found in trial 1: comparing the mirror's stored generation to the texture generation disabled the path entirely (generation advances on EVERY update by design); the CRC gate already carries all the safety, so the check was removed. Measured (same replay, ~3.5 min session): recurring texture records 142 MiB -> 21 MiB (-85%) BEFORE the wire zstd; 807 deltas, remaining fulls are 4.7 KB rects/keyframes (was 76 KB avg). Session: 24-31 fps, 2.0 Mbps at scenes that previously needed 5-9, zero recoveries, zero CRC rejections, screenshots pixel-perfect (fog/terrain correct). This also shrinks the reliable-channel bursts that cause the dependency-stall spikiness. PROMOTED to all seats 2026-08-20 after the user's feel-tests (their live texdelta session: sustained 40 fps target at 4.6 Mbps, p95 gap 46-69 ms, zero recoveries, zero CRC rejections), together with the gated quant encoder. The structural follow-up shipped the same day: the QUIC/WebTransport command planes (see the dedicated section below) remove reliable-channel head-of-line stalls as a category; WebRTC stays for audio. ### Input under high APM + start-of-game smoothness + capacity-fit pacing (2026-08-20) Input hardening (verified with a synthetic 600 APM same-spot click storm plus a 125 Hz minimap drag sweep): - Settle elision: a press or wheel whose position already rests under the pointer (same-spot bursts, the release of a still click, zoom spam) skips the warp + 17 ms settle. Before, every button message serialized 34 ms in the injection queue and a burst backed up hundreds of ms. - Standalone pointer moves coalesce in the injection queue (only the newest position matters between button transitions), so drag positions can never pile up behind a click's settle wait. - Drag position streaming 40 -> 16 ms (60 Hz) for native minimap drags and selection bands. - Permanent metric: the pipeline log's injectPeak (worst enqueue->execute wait). Measured under the storm: 18-37 ms (was: unbounded backlog). - Input boost softened: raise cadence by at most 1.5x (cap 30) instead of jumping a struggling 8-20 fps stream straight to 30 — that jump multiplied the exact burst that made it struggle. Start-of-game / churn cascade (measured live by running the storm during early game): camera jumps define new geometry with every captured Present; those bytes queued faster than the association shipped them, so every frame's resource dependency chased a receding target and the frame plane starved with an EMPTY sender (~25 s of sent<0.5/s), while frame-channel backpressure fired 13 useless cuts to the 8 fps floor. Fixes: - Resource-backlog backpressure: when reliable_ holds >1.5 MiB during gameplay, cut the CAPTURE rate (x0.80/2 s) — fewer Presents during churn produce proportionally fewer new resource bytes; the only lever that shrinks this backlog. Pipeline log shows backlog=..MiB. - Backpressure cuts land at the measured-capacity fit point (capacity*0.8/frameBytes) instead of walking down in x0.85 steps. - Proactive step-down: with a real measured ceiling and growing frames, step to the fitting cadence before the queue fills (1.10 hysteresis). - Delivery gate uses the smoothed render-fps EWMA, not the raw 1 s sample (raw samples alias 14->37 around a steady 23 and flapped the climb). Same-storm before/after: 13 consecutive cuts + target pinned at 8.0 + ~25 s arrive=0 -> 2 cuts landing at 23 fps, stream flowing (arrive/draw 25-31) THROUGH the storm, injectPeak 37 ms worst, dictionary drops 2. Tried and REVERTED (2026-08-20): routing GEOMETRY_ANCHOR frames over the reliable recovery channel to stop anchor loss from killing five-frame GOPs. Strictly worse: the channel is ordered, so each anchor head-of-line-blocked behind 164 KiB recovery keyframes/retransmits and arrived AFTER its unordered dependents, beyond the client's 20-80 ms reorder window — 449 dictionary drops + 17 recoveries in 3 minutes (vs 52 GOP blips with lossy anchors). Precondition for a retry: a client-side dictionary-wait (hold a dependent briefly when its dictionary frame is newer than the active recovery, since a reliable anchor is then guaranteed to arrive). The client-side gap-skip preference for buffered anchors (live-transport _scheduleFrame) was kept — it helps in both designs. ### CRIU fast-start: PROVEN on the London box (2026-08-20) Full round trip verified for native-1285: boot engine gated at replay-open (classic-replay-open-gate114 preload, 32+64-bit builds) -> criu dump of the whole namespaced tree (Xvfb + wineserver + game; 131 MiB images, sub-second) -> tree killed -> criu restore in **142 ms** -> place a DIFFERENT replay at the gated path -> touch release -> game loads and PLAYS it (screenshot-verified in-game at 13 s elapsed). One generic checkpoint per profile serves any replay of that profile: the gate fires before the replay header is read, so map load happens after restore. Key facts learned: - criu check: "Looks good." on kernel 6.8.0-106 (built 4.2.1 + war3-procfs patch via infra/runtime/build-criu-war3.sh; /usr/local/libexec/war3/criu). - classic-criu-worker.sh launch/dump/restore works almost unmodified; the hardcoded uid 1000 is now SERVICE_UID=$(id -u ubuntu) (London: 1001). - Xvfb lives INSIDE the pid+net namespace, so X sockets never cross the checkpoint boundary; /tmp is shared, so the host relay can still reach the namespaced display socket for XTest input. - restore needs a war3.json manifest (wineServerDirectory - derived from prefix dev/ino hex as /tmp/.wine-/server-- - plus xvfb/worker/war3 log paths and sizes at dump). The builder flow should write it automatically; the proof wrote it by hand. - Proof assets on the box: /opt/war3-runtime/w3cs-checkpoints/native-1285/ {images,war3.json}, prefix clone /opt/war3-runtime/classic/prefix-criu-slot0, gates at /usr/local/lib/war3-replay-open-gate{32,64}.so, shims at /usr/local/lib/war3-shims, seccomp wrapper in /usr/local/libexec/war3. Remaining CRIU -> production work: 1. d3d9 proxy inside the checkpoint: launch with W3_D3D9_* env + d3d9=n,b overrides so the capture stream exists in the image; define the capture file lifecycle across restore (proxy keeps its saved fd offset - decide truncate-and-resync vs fresh-file handshake with the relay). 2. Relay integration: a checkpoint-aware session command (restore + place replay + release instead of cold boot) per profile, falling back to lab-game-session.sh when no checkpoint exists. 3. Build checkpoints for native-124cde / native-1140 (same flow) and native-1311-d3d9 (win64 prefix + CASC - needs its own launch args). 4. Embedded audio mode (pulse inside the namespace) for streaming parity; the proof ran audio-disabled. ### QUIC/WebTransport command planes: LIVE (2026-08-20) The structural fix for the dependency-stall spikiness class: SCTP's ordered reliable channel head-of-line blocked everything behind one lost resource packet. QUIC streams are independent, so the failure mode is gone by construction, not tuned around. Architecture (details in WIRE-PROTOCOL.md "WebTransport planes"): - wt-bridge sidecar (Go, native/wt-bridge, systemd w3cs-wt-bridge, UDP 4443, one process for all seats): terminates browser WebTransport and consumes the relay's LOCAL WebSocket planes (loopback TCP is lossless; the lossy leg is exactly what QUIC replaced). Relay message bytes are untouched; the relay itself only gained the /input endpoint and the frameTransport mode "webtransport" (= websocket preference). - Frames: one QUIC uni stream PER FRAME; a superseded dependent frame still blocked by flow control is RESET (keyframes/anchors never). Resources/recovery: one ordered stream each (texture XOR deltas need per-texture ordering). Input: pointer moves as datagrams, buttons/keys on the reliable control stream; every input still races the signaling lane and the relay dedupes by inputId, so a click carries its own coordinates and cannot land stale. - Browser: the WebTransport session lives INSIDE the render worker (web/wt-planes.js): QUIC -> zstd -> parse -> WebGPU with zero main-thread hops. The planes surface as DataChannel-shaped shims, so LiveCommandTransport runs unchanged. WebRTC stays for audio, and the DataChannels stay connected as the automatic fallback (?wt=0 forces the old path; both verified live the same afternoon). - Page telemetry counts WT plane bytes into the network report, so the relay's capacity model keeps seeing the real receive rate. Verified live on seat 3 (Chrome, real internet path, 48-53 ms RTT): governor ramped 20 -> 40 fps in ~40 s; sustained sent 39/s with client arrive 39 / draw 39 (drawn == arrived == sent, zero dependency stalls); 13-15 Mbps wire; 1,864 per-frame streams, 15 superseded-frame resets absorbed without a single decode error; input click acked in 50.9 ms (= path RTT); zero recoveries over the soak; bridge cost 4.6% of one core. Pitfalls burned into code comments: quic-go does not add the h3 ALPN to a custom TLSConfig, and the default webtransport-go origin check rejects cross-port page origins. Live feel-test findings (same evening, user driving): - The eager cancel-on-supersede was real-world harmful: relay sent 27-29 fps while 3-17 fps completed arrival (transmitted-then-reset frames are pure wire waste; under congestion it self-amplified). Reworked: in-flight frames always finish, latest-wins applies to the UNSTARTED queue (costs no wire), reset only for a frame stuck >300 ms with newer data waiting. After the fix: thousands of streams, zero cancels, arrive == sent at 40 fps, clean shedding in heavy scenes. - The "engine freezes" the feel-test hit were NOT the transport and NOT the engines: the proxy's emergency blob-cache clear ran on EVERY vertex record once the working set stayed above BLOB_CACHE_LIMIT (64 MiB) - a livelock at ~0.2 fps inside clear_blob_cache_locked (caught live via gdb on the spinning 1.24 engine), with per-record chain resets exploding encoded frames to ~110 KiB right before each collapse. Old engines batch more draw positions and W3CS_QUANT interns a reconstruction per animated slice per frame, which is what pushed 1.14/1.24 over the old limit. Fixed: limit 64 -> 160 MiB and the emergency clear is spaced (>= 4 frames apart); snapshot and reconnect resets stay unconditional. Structural follow-ups noted below. Shipped the same evening after the livelock fix: - Chain-preserving cache eviction: a boundary mark-and-sweep (W3CS_BLOB_SWEEP, default on) frees unreferenced cache blobs 240 frames after their last touch, every 16 Presents. The cache tracks the live working set, chains never reset mid-scene, and per-seat memory stays low for multi-seat scaling. Safe because slice pointers are the only raw references that outlive a frame (marked first) and frame_wire_blob is only consulted within its own frame; content- derived blob ids make re-mint after eviction byte-identical. - Background tabs park their game: the page reports visibilitychange over signaling; the relay SIGSTOPs the engine group (replay position kept, zero CPU/encode/bandwidth) and resumes transparently - no recovery needed since nothing is dropped. Ten minutes hidden releases the seat (ws close 4004, honest end screen). Verified live by the user's test. Extended the same evening to app switches: window blur (tab stays "visible") parks after a 10 s grace delay, focus resumes instantly; ?blurPark=0 keeps a passive second-monitor session playing. - Engine-stall detection: zero queued Presents across two 30 s ticks with an unparked engine and a connected viewer -> "engineStalled" control message; the page shows the ended screen and the VIEWER chooses to restart (an automatic engine restart would silently rewind the replay to 0). No viewer -> nothing to do; the next session start replaces the engine anyway. - Audio-silence root cause (user report "no audio in a while"): seat restart storms orphan pulseaudio ("Unit process pulseaudio remains running after unit stopped") - the game then plays into one daemon's sink while the relay captures a dead daemon's monitor: a live but silent track. A full-stack restart (one fresh daemon) restored audio. Durable fix TODO: make pulse a systemd unit the seats BindsTo=, so pulse and seats can never split across daemons. The patch24proof retest CLOSED the livelock verification: with the spaced clear + sweep DLL the user ran the previously-fatal replay clean end to end ("went well"), through the heavy scenes that twice collapsed the old build to 0.2 fps. ### CPU hotspot pass (2026-08-20, perf on live seats) Attribution first, then fixes. Measured with perf on a live 40 fps QUIC session and on idle seats: - The game process is the game: 70% its own simulation code, 16% kernel/wine. Our proxy (capture + quant + texdelta + hashing) is 0.54% of the game's CPU - nothing material to optimize there. - Active relay: 45% of CPU was zstd - and half of that was waste. A fresh CCtx per frame plus re-digesting the dictionary on EVERY compress (dependents share their anchor's dictionary across the whole GOP). Fixed with a cached ZSTD_CDict + reused CCtx: identical bytes out, dictionary digested once per anchor instead of four times. zstd share dropped to 35% on a heavier scene. - Idle seats burned ~8% of a core EACH on pure timer churn (74% kernel + 19% glib dispatch: the 1 ms drain tick and 2 ms capture tail poll, ~1500 wakeups/s doing nothing, 24/7, x3 seats). Both now back off to 50 ms after 2 s / 0.5 s of no viewer and no work, and re-arm on the first sign of activity (at most one 50 ms delay at session start). Idle relay measured after: ~1.2% of a core. - Frame compression level stays 7 (bandwidth is the viewer's scarce resource, and zstd is no longer the bottleneck); the remaining relay profile is libc memcpy (~26%) and kernel I/O (~21%) - the capture file round-trip (game writes ~30 Mbps to /tmp, relay reads it back, ~1 GB per session) is the next structural target: a shared-memory ring would remove both sides' I/O. Roadmap, not urgent. The capture shared-memory ring SHIPPED the same evening (see WIRE-PROTOCOL.md "Capture transport"): a tmpfs SPSC byte ring carrying the byte-identical envelope stream replaces the append file - no disk writeback, no read syscalls, no torn tails, per-session reset by the relay, CRIU-safe (producer never caches write_pos; the file path stays as the checkpoint-compatibility fallback and the relay reads both sources). Verified live end to end: 39/39 fps over ring + QUIC with no capture file on disk; all seats promoted. Event-driven drain dispatch SHIPPED: enqueues, plane acks, channel opens, and the input boost wake the sender immediately (thread-safe coalescing g_main_context_invoke); a one-shot timer fires exactly at the pending frame's pacing deadline; a 5 ms retry runs only while a send is blocked AND senders are attached (without the sender gate, a viewerless seat's queued warm-boot resources spun the retry at 200/s); a 100 ms tick covers time-based housekeeping. Measured: idle relay 10 wakeups/s and ~0% CPU (was ~8% this morning); live session at 40 fps paces identically (39/39 net/draw, RTT unchanged) with ~300 wakeups/s total versus ~1000+ before. pulse-as-systemd-unit SHIPPED the same evening (w3cs-pulse.service, own cgroup, seats Wants/After it): the audio-silence class - seat restarts SIGKILLing the shared daemon out from under the other seats' games and the relays' pulsesrc monitors - is structurally gone. The user's "no audio on 1.22" report was this, not the engine; confirmed audible after the fix. Resource-vs-frame QUIC scheduling SHIPPED (the "first minute" fix; see WIRE-PROTOCOL.md "Resource-vs-frame scheduling"). Root cause of the observed 5-15 fps first minute: resources pushed into a fixed 4 MiB in-flight window at a fixed 64 Mbps token rate, and quic-go round-robins the congestion window between streams, so on a ~10 Mbps path several SECONDS of resource bytes crowded every frame stream; the SCTP-era yields keyed off buffered amounts that stay near zero in WebTransport mode. Now the resource plane runs behind frames: capacity-budgeted rate, BDP-scaled in-flight window, delivery-report yields, recovery-in-flight yields, dependency-byte bypass, and a time-based (not byte-based) capture backpressure. The rearm scheduler learned exact waits for the new blockers (token due time; plane acks and buffered-amount-low signals resume window-blocked work), so pacing added no polling - idle seats still measure 0-1 CPU ticks/5 s. Hard lesson from the first live attempt, preserved in the code comments: NEVER learn a pacing ceiling from ordinary receive samples. Receive is app-limited - a paced sender bounds what the client can receive, so the estimate can never rise. The first build pinned a fast path at the 2 Mbps floor for 90 s (reliable queue at its 256 cap, capture stalled, render 2-7 fps) before a recovery flush revealed the real path. The shipped design engages a ceiling ONLY from path-limited evidence (governor capacity, or receive measured during client delivery congestion), fades it 15%/s on healthy reports, and releases it after a 12 s hold. Verified live after the fix: bootstrap uncapped, target climbed 20->40 fps in ~10 s with render 22-35 fps throughout, zero resource-backpressure cuts, reliableQ=0. Second hard lesson, caught live the same evening: the ceiling engaged DURING BOOTSTRAP from loading-screen delivery hiccups (the congested predicate fired on loading-screen cadence while receive was app-limited) and pinned the resource snapshot at the 2 Mbps floor - delivery could never turn healthy because the throttled snapshot never finished, so the fade never fired; the client sat on dependency-wait until its stall detector killed the session ("reliable sequence stuck"). FIXED: all resource-pacing congestion evidence (ceiling engagement AND the delivery-congested yield) is gated on gameplay_epoch_ready_, and the gameplay-epoch history wipe also clears any engaged ceiling. Verified: fresh sessions boot straight to 36-37 fps render, and a mid-run recovery rotation dips one report before climbing back to 40. Engine render cap 40 SHIPPED (W3_D3D9_MAX_FPS 60 -> 40, all seats). Measured A/B (same replay, same 3-4 min segment, war3.exe alone): 85% of a core at 60 -> 80% at 40 - only ~5%, because 8x-speed SIMULATION dominates the JIT profile and is decoupled from render fps. The cap's real win is cadence: at 40/40 every present is captured (no 2-of-3 sampling beat; user-confirmed smoother), and the saving grows at lower replay speeds where the render share is larger. The relay's input-settle waits now derive from the same env (engine_poll_ms: 60 fps -> 17 ms, 40 fps -> 26 ms) instead of a hardcoded 17 ms, so the cap cannot desync the injection timing. Session-identity plane handoff SHIPPED: the relay records the page's session UUID from /signal?session=, a new viewer's signaling accept detaches every plane the previous session left behind, and a plane dialing in with a stale id is refused (4003). A page restart can no longer lose its first resource sequences into a dead bridge session's lingering sockets. (The full mid-session-death handoff - handing live planes back to the standby DataChannels without a restart - remains open.) Wire-size program, measurement phase COMPLETE (2026-08-20 evening). Full-game capture (native-1220, 9,896 frames, 1,135 MiB of frame records) via the new capture tee: W3CS_RING_DISABLE=1 drop-in reverts one seat to the append file (the tee also REMOVES the seat's tmpfs ring: a stale ring with valid magic swallowed the first capture while nothing read it). Findings, per-class chained-zstd attribution at level 7 (tools: deploy-box /tmp/w3cs-stage/w3cs_{sweep,classes,stats}): - zstd level is a DEAD lever: 34.3% ratio at level 3 -> 33.2% at level 11 for 8x the CPU. Level 7 stays. - DEFINE_BLOB_QUANT is 78.1% of the encoded wire (286 of 367 MiB; 52% of plain, compressing only to 48.6% - high-entropy int16 residuals). ~74 records/frame, ~61 KiB/frame plain. This is the whole ballgame: halving the wire means halving quant. - Texture XOR deltas: 12.0% of wire. DRAW_INDEXED: 6.7% (its records are 0% prev-frame-identical because payloads embed per-frame blob ids; a stable-slot reference would let the chain erase them). - Every classic state class (transforms, render/sampler/stage state, materials, lights) is ALREADY erased by the chained dictionary: SET_TRANSFORM is 11.9% of plain but 1.6% of wire (73% of its bytes are byte-identical to the previous frame and zstd finds them). Temporal-delta work on state classes would win nothing. Follow-up studies on the same capture, all decided by measurement: - Velocity prediction (delta-of-delta) on quant positions: REFUTED. Skinned vertices move non-linearly at replay speed; residuals grew (nnz 64.8% -> 79.7%, encoded +2.9%). Do not revisit without a fundamentally different motion model. - Byte-plane and component-split reorderings of the int16/f16 value streams: ALL measured worse than natural interleave (zstd's matching exploits vertex-locality across components). Do not revisit. - Semantic decomposition: NORMALS were the real payload - 176.5 MiB of float16 deltas (~101 MiB encoded, ~28% of the whole wire), because CPU-skinned meshes rotate ~80% of their normals every frame. Raw position words (teleports/slot reuse, |delta| > 512 units) are the next block (~56 MiB encoded); headers' 16 B of random hashes/record ~11 MiB (kept: they are the reconstruction tripwire). Octahedral normals SHIPPED behind W3CS_OCTA (mode bit 8 on the quant record; see w3cs_protocol.h): each changed near-unit normal ships as one int8 (u, v) octahedral point + a per-vertex mask instead of three float16 deltas, reconstructed identically on both ends and interned for error feedback; degenerate/non-unit/out-of-tolerance normals fall back to the classic path per vertex. Measured on full same-replay games, per frame: total encoded wire -13.4% (37.9 -> 32.8 KiB/frame), quant class -14.9%, visuals and lighting clean, zero hash mismatches. Deadband follow-up (same evening, from external review triage): a vertex whose true normal stays within ~0.57 degrees of the previous reconstruction (inside octa's own quantization error) is HELD and costs zero bytes, with the 1.15-degree encode gate as the hysteresis outer band; fresh reconstructions equal to the base are suppressed the same way. Encoder-only - a held vertex is also masked out of the float16 walk (a separate skip bitmap; without it the classic path would re-send what the hold saved). Measured: quant -5%/frame, total -2%/frame - at the run-variance noise floor, kept because it is strictly non-increasing and free. Verdict on the review's noise hypothesis: most normal churn is real rotation at replay speed, not float noise. The toolchain proof is now airtight: the exhaustive cross-check runs the production i686-mingw build under wine over ALL 65,536 (u,v) byte patterns (-128 included), bit-identical to the browser chain. Cumulative wire-size program result so far: 37.9 -> 32.2 KiB/frame encoded (-15.0%). HARD LESSON (cost one evening hour): the first octa deploy failed as a blob-hash-mismatch storm. The i686 proxy compiles to x87, which keeps float expressions at 80-bit precision BETWEEN operations, so compound expressions never round intermediates to float32 and C reconstruction diverged from the browser's per-op Math.fround chain. Fix: a volatile float store after EVERY operation in octa_decode (x87 80->32 store is double-rounding-safe for float32 +,-,*,/,sqrt, so each stored step equals the correctly-rounded op). PROVEN, not assumed: all 65,025 (u,v) points decode bit-identically between x87-compiled C and the JS fround chain, and captured records verify through the real client blob-store offline (tools: scratchpad octa_cross.*, octa_pairs.mjs). Any future cross-language float32 contract in this codebase must repeat that exhaustive cross-check before deploying. Also learned: the browser console persists across in-page session restarts - a "still failing" read after a fix MUST come from a fresh tab, and offline verification of the actual capture outranks console archaeology. Resource bundles on R2 + CDN SHIPPED (2026-08-20 night). The "Loading WC3 textures" phase is the resource snapshot: ~30 MiB/game of immutable content-addressed bundles (~4k per game, SHA-256 keys). New pipeline: - Every cache-safe bundle is spooled once to the shared /home/ubuntu/w3cs-lab/bundle-spool (atomic rename, all seats); w3cs-bundle-upload.timer pushes new digests to r2:war3replays/resource-bundle/.bin with immutable Cache-Control and marks them in /uploaded/. - The relay sends a 36-byte reference instead of inline bytes when the browser announced the digest OR it is confirmed uploaded (>=4 KiB); keys are object-scoped so bucket config (CORS, cache rule) is dashboard-only. The /resource-bundle endpoint now falls back to the spool on disk, so references survive relay restarts and LRU eviction (the old 404 hole). - Client resolution: IndexedDB -> https://cdn.war3replays.com (digest- verified before trust; ANY failure falls through) -> relay endpoint. ?bundleCdn=0 disables, ?bundleCdn= overrides. - Verified cold end to end: a wiped-IndexedDB client (announced 0 hashes) received 271 references / 25 MiB kept off the stream and resolved them all at full stream health (38/38 drawn at target 40). - The 1,024-hash announcement cap became irrelevant (references no longer depend on announcements), so the planned chunked-announcement protocol was deliberately NOT built. - PENDING USER DASHBOARD ACTIONS (uploads/serving work today, but edge caching and browser CDN fetch wait on these): (1) R2 bucket war3replays -> Settings -> CORS: allow GET/HEAD from https://*.war3replays.com; (2) a Cache Rule making cdn.war3replays.com eligible for cache (respect origin TTL) - R2 custom domains do NOT edge-cache by default (measured: cf-cache-status DYNAMIC even with immutable Cache-Control). Known-content blob references (W3CS_BLOBREF, opcode 37) are implemented and verified functionally (29k references in one game, zero client errors; the client retains draw-touched blobs for 16 applied frames, the encoder references only within 8 captured frames of the same epoch, applied being a subset of captured makes eligibility provably safe; keyframes/anchors stay self-sufficient). Enabled on seat 3 only; the promotion A/B needs one clean full game with idle seats - the first attempt was contaminated by a bandwidth-contention lesson: measurement sessions share the developer's downlink with their real viewing, so never run one while the user is watching. Per-map CDN packs SHIPPED (2026-08-20 late night). Measured cross-map content overlap first (two 1285 maps + the 1220 capture): frame-plane pose content is 0.1% shared across games - CPU-skinned vertices depend on continuous animation phase, so IN-GAME streaming can never be pre-seeded (definitively closes that idea; the wire-size program is the only in-game lever). Resource-plane content is ~14% generic across maps, so packs are keyed per (profile, map), exactly the granularity that covers a session. Pipeline: relays append per-session digest manifests in the spool; w3cs-pack-build.py (invoked from the upload timer) unions them per key and builds "W3PK" packs (digest+length table, bodies, zstd -19; ~3.7x smaller than raw - LostTemple: 28.2 MiB -> 7.55 MiB) named by content hash (immutable) plus mutable per-replay manifest JSONs (max-age 300); the page prefetches its replay's packs during the load screen, decompresses in ~60 ms with the existing zstd wasm, digest-verifies every entry into IndexedDB, and localStorage markers skip already-applied packs. All tiers remain graceful: pack -> per-bundle CDN -> relay disk. MEASURED cold load, same replay, wiped browser state: first game frame 7,909 -> 4,325 ms (-45%), 761 bundles seeded, resolution burst 27.2 -> 6.5 Mbps peak. Gotcha recorded: the manifest fetch uses cache:"no-store" - a manifest 404 cached during the upload race otherwise sticks for minutes. Not yet done (next steps): clean blobref A/B + promotion; mid-session QUIC death currently restarts the session instead of handing planes back to the idle DataChannels; audio/WebRTC removal (would need jitter buffer + concealment client-side) stays out of scope until the QUIC planes have production hours. Next optimization program (chosen): wire size - per-opcode attribution says geometry defines ~48%, transforms ~26%, draws ~20% of plain frame bytes; plan is zstd level sweep with the cached CDict, then temporal delta-encoding of geometry/transform records, then anchor/GOP economics. ### Cast overlay (2026-08-20, live on all lanes) Ported the live site's cast overlay (race badge, name, rolling APM, hero chips with live levels) into the WebGPU client: web/cast-overlay.js + tools/make-cast-descriptors.mjs + static cast/.json served by the demo web root. Verified on 1140/1220/124cde/1285/1311-d3d9, 1v1 and 2v2. APM is the rolling 60 s figure only (w3g_actions 0.95 counting); the whole-game APM stays on the site's replay page. Clock: anchors on first perspective frame, normalizes replay speed to 1X via the panel buttons (ladder floor 1/2X, unit steps to 8X), tracks viewer speed/pause/restart clicks; known drift when a cold lane's bootstrap delays the first rendered frame (~30 s worst observed) - proper fix is a relay-side pos message like the streaming worker's OCR clock. **RELEASE ITEM: the static cast/ JSONs are demo staging only.** Before release, per-replay overlay data must move to the catalog DB (written at ingest next to detail_json, served by the web tier) or to an in-browser .w3g parse (no store; also what R2 on-demand replays will need). They exist only because the London box has no Node and the demo aliases are not catalog rows. ## Any patch 1.10-1.28f is now a buildable engine profile (2026-08-21) Adding an era lane used to mean finding a preserved install. It does not any more. `replays/classic-client/ReplayKitPatch.mpq` (already ours) carries, for every patch from 1.10 to 1.28f, the four files an overlay needs: War3Patch.mpq = steady_basis.mpq + the patch's recipe members, then UI/FrameDef/Glue/MainMenu.fdf <- disable_bnet.fdf war3.exe / Storm.dll / game.dll = verbatim pool members `infra/runtime/build-classic-engine-profile.py` reads ReplayKit's `.rcp` recipes (12-byte header, then length-prefixed UTF-16LE target/source pairs) and emits a tarball that `lab-install-engine-profiles.sh` accepts as is. It is validated against a lane that already streams: rebuilding 1.24e reproduces the installed classic-124cde overlay exactly - all three binaries byte for byte and all 879 War3Patch.mpq members. Rebuilding 1.14b reproduces the shipped war3-1140 install's binaries, so that install was itself an RK overlay. MPQ member names are case-insensitive, so the verification folds case: a recipe asking for `Melee_V0\UI\FrameDef\...` lands on the basis entry spelled `.../Framedef/...`. Two bugs this exposed, both older than the 1.12 work: - The profile case set `ENGINE_OVERLAY`, and a *second* copy of the profile list decided whether to apply it. Any profile added to one and not the other silently ran the base engine. The apply step now keys off `ENGINE_OVERLAY` itself. - The cold launch never passed `-classic`. The CRIU claim path always did (`W3_CLASSIC_GAME_MODE=roc`), so every Reign of Chaos replay that had a claim image worked and every one that fell back to a cold launch died in game.dll on a null dereference (+0xb0) with an empty War3Log. That is why the 1.12 replay first crashed on 1.14 as well. Remaining acquisition item is unchanged: 1.29 is not in the kit, so the 4,143 replays that need it are still blocked. ## ESWC 2003 on a native 1.10 lane (2026-08-21) The demo now carries ToD's three ESWC 2003 series on patch 1.10 (quarter- final vs SK.Insomnia, semi-final vs SK.MaD)Q(FroG, 3rd place vs [aT]FaTC; eight Reign of Chaos replays, original archive bytes from `replays/archive/.w3g`, eventId `eswc-2003` in manifests/tournaments/verified-tournament-classic-v2.json). ToD's two second-group-stage games from the same event (vs SK.HeMaN) are patch 1.11 and run on `native-1110`, built the same way (artifact sha256 `2f5e104c…`, same scripts as 1.10/1.12, same maps by CRC). Rule for release: every patch the AWS streaming lane serves must have a London lane, or the router falls back to AWS for it; with the builder each lane is a ~10 minute job (build, install, wire, claim images), so the remaining kit patches should be added ahead of demand rather than on request. Engine: `classic-1100` overlay built by infra/runtime/build-classic-engine-profile.py from the ReplayKit pool on the London box (recipes files/patch_1.10.rcp + mpqs/war3patch_mpq_1.10.rcp, 114 War3Patch members, game.dll 1.0.10.5610). Artifact sha256 `9ce3083b…`, pinned at worker-image/classic/runtime/v1/inputs/. The camera zoom patcher's 1.12/1.14 mov-immediate blocks match the 1.10 game.dll exactly once each, so the lane gets the 2250 ceiling too. Maps: the ReplayKit pool stores stock maps per patch as the version valid for replays *up to* that patch (not "changed by" it), and `map_table.ht` in the kit is a `map CRC -> pool path` index. None of the three ESWC CRCs is in that table, so identity was settled by recomputing the statstring map CRC (GHost++ "xoro": common.j ^ blizzard.j, rotl 3, ^0x03F1379E, then war3map.j/.w3e/.wpm/.doo/.w3u/.w3b/.w3d/.w3a/.w3q each folded with rotl 3) with the 1.10 overlay's Scripts\common.j + Blizzard.j (byte-identical to 1.12's). The implementation was validated first on the proven 1.12 lane (PlunderIsle `b69f5862` -> classic-stock/1.12, the map that streams). Hits: | Replay map | recorded CRC | map bytes (sha1) | corpus key | |---|---|---|---| | (6)GnollWood.w3m | 67078ba1 | 64a66282… | maps/classic-stock/1.28f (unchanged since RoC) | | (4)TranquilPaths.w3m | 67f16553 | cf83ae21… | maps/classic-stock/1.11 | | (4)LostTemple.w3m | 08bbd799 | e00ee23b… | maps/classic-stock/1.12 | The checker is infra/runtime/classic-map-crc.py (needs smpq; extract the profile's Scripts\\common.j + Blizzard.j first). Worth wiring into the ingest map resolution so classic candidates are verified, not accepted by name. ## Patch coverage matrix (tested end to end through the live pipeline) | Patch | Engine profile | Result | |---|---|---| | 1.28.5 | native-1285 | PASS (long-session baseline, 40 FPS) | | 1.24 c/d/e | native-124cde | PASS (38 FPS, replay patch24proof) | | 1.24 a/b | native-124ab | NOT TESTED - overlay installed, no test replay staged | | 1.14 | native-1140 | PASS (38 FPS, expired-patch map install path works) | | 1.12 | native-1120 | PASS (RoC, claim-restored, zoom + cast overlay). | | | | Engine built in-house from the ReplayKit patch pool. | | 1.11 | native-1110 | PASS (RoC, claim-restored on all 5 seats, zoom + cast | | | | overlay; ESWC 2003 ToD vs HeMaN, LT + TP verified | | | | in-browser 2026-08-21). Same builder/kit as 1.12. | | 1.10 | native-1100 | PASS (RoC, claim-restored on all 5 seats, zoom + cast | | | | overlay; ESWC 2003 ToD set, all three maps verified | | | | in-browser 2026-08-21). Same builder/kit as 1.12. | | 1.31 | native-1311-d3d9 | PASS (37 FPS, win64 prefix + CASC). Open: cursor | | | | suppression misses 1.31's cursor atlas (cyan reticle | | | | visible); per-patch atlas registration needed. | | 1.17 / 1.24 via ReplayKit | replaykit | ROOT-CAUSED (fix in progress). | | | | RK locates the game via the Blizzard registry keys; | | | | the prefix's InstallPath/GamePath pointed at a DEAD | | | | per-session overlay (`Z:\tmp\w3cs-game-session.*`), | | | | so RK exited silently under -totalsilence. Working | | | | recipe (from streaming/server.py): run RK from its | | | | own writable install (C:\war3rk), replay at | | | | C:\war3rk\Replays\, maps into the INSTALL Maps dir | | | | (<=1.27 engines read maps there), registry stamped | | | | to C:\war3rk, serialized (RK mutates its install), | | | | args ` -totalsilence -window` (no -names/ | | | | -colors). Session scripts must stop leaking their | | | | ephemeral GamePath into the shared prefix registry. | Also open: the 124/1140 profiles render at 1432x1046 and 1311 at 1440x1080 instead of the requested 1024x768 - the resolution stamp only reaches the native-1285 registry path. Harmless but should be unified (bandwidth). ## Archive coverage math (classic-readiness-v1, 27,552 replays) | Runtime class | Ready | Lane | Status | |---|---|---|---| | classic-convert-128 (c28 pre-converted) | 14,441 | native-1285 | VALIDATED | | classic-native-128 | 1,374 | native-1285 | VALIDATED | | classic-native-131 | 66 | native-1311-d3d9 | VALIDATED | | requires-classic-129-engine | 0 (4,143 blocked) | none | needs a 1.29 profile (acquisition item) | | missing-conversion | 222 | offline c28 converter | run the ingest converter, not runtime RK | | map-ambiguous / missing-map | ~6,600 | any | map-corpus curation, not engine work | **Verdict: runtime ReplayKit is NOT release-blocking.** 57% of the archive plays today on validated lanes; the biggest blocked slice (15%) needs a 1.29 engine profile, which no amount of RK work provides. The 222 missing-conversion replays are better served by running the existing offline converter. RK-at-runtime remains a nice-to-have for exact-engine fidelity; findings so far: registry InstallPath must point at C:\war3rk (session scripts leak dead overlay paths into the shared prefix), the prefix needs wine-mono (installed 10.1.0), and after both fixes RK v1.28b still idles at its window under wine 11.13-staging/Xvfb without spawning the game - next step is an A/B against a streaming worker's wine version, or driving its UI once via xdotool to learn its state. ## Fast viewer start: CRIU checkpoint per engine profile Goal: viewer connect starts at REPLAY LOAD (seconds), not engine boot. Today the lab keeps ONE warm engine (SIGSTOP-parked) for ONE profile; switching profiles is a cold boot (20-60 s, worst on 1311). Design (reuses the streaming pipeline's proven machinery): - classic-replay-open-gate114.c (LD_PRELOAD) blocks the game exactly at replay-file open; W3_REPLAY_GATE_DIR/W3_REPLAY_GATE_SUFFIX are already plumbed in streaming/server.py. - Per profile, at image-build time: boot engine -> reach the gate -> CRIU dump (criu-4.2.1-war3-procfs.patch + criu-dri-reopen-plugin.c on the box; benchmark-classic-criu.sh has timings). - Per viewer session: inject the chosen replay at the gated path -> CRIU restore -> release the gate -> game loads the replay directly. - One checkpoint image per (profile, resolution). Images are immutable and bakeable into the VPS provisioning. - The relay keeps the current warm-seat flow as fallback when no checkpoint exists for a profile. ## Production topology (Germany VPS + AWS fallback) - Primary: command-stream relay on a dedicated VPS in Germany (NOT AWS). No GPU required: the proxy runs with rasterization disabled, so seat cost is CPU-only (~0.5 core + ~1 GB per active seat measured; an 8-core VPS suits ~12-15 seats). Requirements: UDP open (ICE range), CAP_SYS_ADMIN for CRIU, wine + profiles + checkpoints provisioned, good EU peering. - Fallback: the existing AWS video-streaming pipeline stays live. - Selection: war3replays.com (web tier) decides per viewer: 1. Page attempts WebRTC to the VPS; the existing in-page latency admission (idle-RTT gate, entry-only) decides viability. 2. On rejection, UDP-blocked networks, or missing patch support (e.g. ReplayKit-only replays until unblocked), serve the streaming player instead. 3. Selection happens before seat claim; production ranks workers by measured RTT (the admission module already carries product copy). - The content-addressed resource bundles (/resource-bundle/) are the CDN artifact when cold-start bandwidth matters: same digests across sessions, CloudFront in front, browser cache path already implemented. ## CRIU claim fast path (2026-08-21) Any same-family replay now starts from a generic CRIU checkpoint instead of a cold engine boot. One image per (engine profile, game family, seat) holds a fully booted engine — d3d9 command-stream proxy loaded, Xvfb and an embedded pulseaudio inside the image — blocked at the replay-open gate (the proven classic-criu-worker mechanism from the GPU program). Restore is 130-150 ms; the requested replay is installed at the gate's fixed path before release, so the claim pays only WC3's own map load. Pieces and their owners: - `infra/runtime/classic-criu-worker.sh` (installed as `war3-classic-criu`): launch/dump/restore/stop of the namespaced session tree. Gained `W3_CLASSIC_GAME_ENV_FILE` (root-owned file of W3CS_*/W3_D3D9_* variables sourced inside the namespace — the proxy configuration is baked into the image) and a display-number override. - `deploy/w3cs-criu-build-images.sh` (`w3cs-criu-build-images`): the image matrix builder. Assembles one game view per profile (shared MPQs as symlinks, engine files as PRIVATE COPIES — see the hard-won rule below), writes per-seat env files, and rebuilds only images whose fingerprint (proxy DLL + env file + engine files, symlinks followed) changed. Runs at the end of deploy-cpu-server.sh; ~80 s per image. - `native/lab-game-session.sh`: dispatches a claim when an image exists for the resolved profile and the replay's probed family (product bytes at 0x30: W3XP=tft, WAR3=roc). Stages the map through the normal metadata stager, restores, publishes the claim environment, releases the gate. `W3CS_CRIU_DISABLE=1` falls back to cold launches. - The relay follows `/tmp/w3cs-claim-seatN.json`: injector reopens on the claim display (libX11 IO error handler + IO-error-exit handler installed — the DEFAULT handler exits the process when a claim's Xvfb dies), audio binds the claim's embedded pulse (creation deferred 500 ms so classic sessions keep instant audio), park/stop signal the claim's process group, and a replay change on a claim always restores fresh (the warm-switch wedge stages a path claim engines never read). - Root boundary: /etc/sudoers.d/w3cs-criu lets ubuntu run only war3-classic-criu (SETENV; the worker fail-closes on its inputs). Verified live: viewer sessions over restored claims stream 39/39 FPS via QUIC with working input injection and 96 kbps Opus from the embedded pulse daemon, across native-1285 tft/roc. RULES PAID FOR IN BLOOD tonight: 1. `cp` onto a `cp -as` symlink writes THROUGH it into the shared install. The session script's overlay branches had done this on every ReplayKit and native-1220/124cde session — the shared war3rk Storm.dll/War3Patch.mpq/"Warcraft III.exe" ping-ponged between engine generations, which is why the ReplayKit lane was flaky for weeks (CD-key and "outdated Warcraft 3" dialogs depending on which session ran last). Canonical war3rk state = the 1.28f trio recorded in classic-wcg-native-runtime-manifest.json. Always `rm` every copy target first. 2. A CRIU image holds files open BY PATH: an engine file that is a symlink in the view bakes the shared target into the image, and any later rewrite of that target breaks the restore ("Unable to open fd=N"). Claim views must hold private copies of every engine file, and the image fingerprint hashes through symlinks so shared-target changes rebuild the image. 3. native-1140 "does not reach either replay-open gate" — SOLVED 2026-08-21: W3_REPLAY_GATE_DEBUG_PATH (now forwarded into the build namespace by the worker) showed the 1.14 engine opens its replay through a perfectly matching libc open — exactly ONCE. The default W3_REPLAY_GATE_OPEN_INDEX=2 (right for 1.28, which opens the replay twice) left the gate waiting forever. The matrix builder now sets open index 1 for the 114 gate; 1140 images build on every seat and the lane streams from claims (verified live, 39/39 fps). Still open: claim-vs-classic first-frame A/B on a clean path; replaykit runtime lane images (needs the replaykit gate variant); native-1311. ### Claim hardening + engine-renderer findings (2026-08-21, night 2) The first live claim day surfaced a failure chain and two architecture facts. All fixes are deployed and committed; every seat verified healthy (patch28 streams 31/31 fps over QUIC through a claim). Failure chain (all three links fixed in classic-criu-worker.sh): 1. Orphaned claim trees. `stop` only killed the pid recorded in the state files. Any restore that overwrote root.pid while an older tree still ran (park/kill races, crashed sessions) left that tree untracked forever — holding the slot display, the embedded pulse socket, and the wine prefix file locks. Fix: `sweep_slot_orphans` after every stop and on every restore failure. It matches namespace inits by the same strict identity stop uses (service UID + NSpid 1 + the slot's unique `export DISPLAY=":2N"` cmdline stamp), so it cannot select another slot's tree. Session-scoped mode skips the sweep. 2. CRIU blocks forever inside file-lock restoration when an orphan holds the prefix locks — with nothing written to restore.log, which made the hang look like a mystery. Fix: the restore runs under `timeout` (W3_CRIU_RESTORE_TIMEOUT, default 60 s) and failure paths sweep the partial tree instead of leaking it. 3. The blocked CRIU daemon inherits the worker's global dump-lock fd, so after the worker died every restore on EVERY slot hung at `flock` before logging anything. Fix: all three lock sites use `flock -w` (W3_CRIU_LOCK_TIMEOUT, default 120 s) and fail loudly. Also fixed: the claim branch now passes `W3_CLASSIC_GAME_EXE` to the worker. Without it the worker's game-identity check defaulted to "Warcraft III.exe" and rejected every war3.exe engine (1220/124cde) right after a successful restore — the origin of several orphans above. lab-live.sh gained a start-time sweep that thaws and kills frozen session scripts from parked claims (SIGSTOP means their TERM trap never ran at unit stop) before claiming the seat. Architecture facts (verified live, screenshots in the session log): - Restored exact-version engines PLAY correctly: the 1.22 claim ran the WCG 2008 GG.ToD-vs-Grubby replay on Twisted Meadows, the 1.24cde claim ran patch24proof on Lost Temple, both after a ~160 ms restore plus map load. - Pre-1.28 native engines (1140/1220/124cde) call Direct3D 8 — but they STREAM anyway, because the seat prefixes carry DXVK's native d3d8.dll in system32 (1691662 bytes, all three prefixes), which translates D3D8 onto D3D9 and therefore onto the w3cs proxy. Verified live: the 1.14 lane streams client-side with both the DXVK d3d8 and the proxy d3d9 mapped in the game process. The first claim image generation missed this: its builder override string forced d3d8 to wine's BUILTIN (d3d8 -> wined3d, proxy never loads), so war3rk-lineage claims restored fine, played fine on their X display, and streamed nothing ("engine stalled"). Fixed the same night: profile_d3d_overrides() in w3cs-criu-build-images.sh gives the old-engine profiles d3d8=n,b, the override string is part of the image fingerprint, and the matrix was rebuilt. Corollary: an initial conclusion that pre-1.28 lanes "can never stream via w3cs" was wrong — only those first-generation claim images couldn't. - patch24 is NOT a corrupted replay. ReplayKit's pipeline works: it asks the 1.24ab-vs-cde question (totalsilence defaults to cde), then fail-closed checksum map search. The blocker is solely the exact map bytes: the recorded checksum matches neither the Trophy of Legends (4)TurtleRock[SL].w3x (sha1 ee65e23d..., identical in both clan.su packs) nor any of 12 other TurtleRock editions from S3 (wcg-exact-v1 x2, netease x2, english_tft, classic-stock 1.10-1.28f), under BOTH the ab and cde script-checksum contexts (verified with ReplayKit's own exhaustive search, 195 files). Next lead: Chinese Shadowleague archives (uuu9/u9) — the SL ladder shipped several season edits of its maps. Known issue, not yet fixed: the relay segfaulted once during `systemctl stop` while tearing down a resumed claim (journal 2026-08-21 01:59:52, w3cs-live@2). Teardown-path crash after the session already ended; systemd restarted cleanly. Worth a pass over stop_session's claim block ordering (kill pgid vs. GStreamer element teardown vs. X display close). ### Reboot contract (2026-08-21) A full reboot test surfaced one gap: the claim path's gate directory (/run/war3-replay-gate/slotN) only existed as a leftover of the last image build, so the first boot after the claim rollout left all three seats "active" with no engine behind them (session scripts died on the missing release-marker path, their traps stopped the freshly restored trees). Fixed twice over: /etc/tmpfiles.d/w3cs-war3.conf declares the tmpfs directories (gate slots + /run/war3-no-dbus) and the restore worker recreates its slot's gate directory itself. Verified with a second clean reboot: every unit active, three warm claims ready, UDP 4443 listening, zero failed units, viewer streaming 39/39 fps — no manual steps. ### AVX-512 / znver4 decision (2026-08-21) The seat env's GLIBC_TUNABLES AVX-512 cap stays, and the znver4 llvmpipe overlay from the server-render study is NOT promoted here. Rationale: with W3_D3D9_NO_RASTER the proxy captures draws and returns without reaching the DXVK→lavapipe backend, so rasterization is ~0% of seat CPU and the study's −7.7% cycles/frame applies to nothing. The cap also keeps glibc from baking AVX-512 function pointers into claim images, preserving restore on AVX2-only hosts. Reopen only if a server-side raster consumer (e.g. a video-fallback lane) returns to this box. ### Extended zoom-out for the old engines (2026-08-21) infra/runtime/w3cs-camera-zoom-patch.py rewrites the six-rung camera ladder in the claim views' PRIVATE game.dll copies (shared installs stay byte-exact; the image fingerprint hashes the patched dll so images rebuild automatically): distances: 1200 1275 1400 1500 1600 1650 -> 1200 1400 1650 2100 2550 3000 AoA: 335 332 325 318 311 304 -> 335 325 304 304 304 304 The default boot view (1650 @ AoA 304) is untouched; wheeling out past it is a pure dolly (no tilt), and the stock pan-down tilt survives only on the two closest rungs. Engine coverage: 1.22/1.24cde via a unique .rdata table (AoA+distances sit together before 'ListenerMaxCinematic'), 1.14 via four exact code-block transcripts (the second AoA block interleaves an fstp — patterns are verbatim byte transcripts, and the patcher refuses anything but an exact single match). 1.28.5 is NOT covered: its ladder loads from a shared SSE constant pool (1650 next to unrelated constants at rdata a8d900) and needs real disassembly first. Verified live on 1.22: zoom to 3000 streams (30/30 fps fresh client, ~900 draws vs ~450 at default; wire grows to ~39 KiB/frame). Two notes: draw count roughly doubles at full zoom, so weak clients dip — and at 8X speed + full zoom a client can fall behind entirely. Separately, a 16-tick wheel burst mid-stream wedged one client in a permanent recovery loop (1 fps until reload) — client-side resilience bug, not engine: the same state streamed at 30/30 after a reload. Revised the same night: ceiling lowered 3000 -> 2250 (ladder 1200/1400/1650/1850/2050/2250). At 3000 the view exposed culled edges and ~2x draws; at 2250 the same 1.22 test streams 38/38 fps at ~700 draws with no recovery churn. All nine old-engine images rebuilt. ### Idle seats hold nothing (2026-08-21) Seats used to keep a pre-launched engine parked (SIGSTOP) whenever no viewer was attached — a design from before CRIU claims, when a cold engine boot cost seconds. With claims restoring in ~150 ms it earned nothing and cost a lot: 520-644 MB per seat, and a measured reuse rate of 0 of 6 viewer arrivals (the warm engine holds the DEFAULT replay, every viewer asks for a specific one, so each arrival paid a teardown before its restore anyway). Three changes: 1. Session over -> `stop_session()` instead of `park_engine()` in `deactivate_viewer_stream()`. That path only runs when the viewer is gone for good (data channel closed, replay finished, peer torn down), so there is no replay position left to preserve. Logs "seat released: engine closed". 2. No engine at boot. `W3CS_WARM_PRELAUNCH=1` restores the old pre-launched-and-parked engine for lanes with no claim image (ReplayKit, 1311), where a cold boot really is seconds. 3. `refresh_claim_environment()` compares nanosecond mtime. `st_mtime` is whole SECONDS, so a claim file written ready=false then ready=true inside one second (which ~150 ms restores do routinely) silently lost the root pid: park then froze only the session script while the claim tree ran on at ~22% of a core with no viewer, indefinitely. Faster restores made this MORE likely. Mid-session park is unchanged and still matters: a hidden tab freezes so the replay position survives, since a CRIU restore always resumes at the replay-open gate (the beginning). Only the end-of-session park became a teardown. Measured after: idle box at 100.0% CPU idle and 824 MB used (was ~95% idle and 2715 MB). Viewer arrival still streams at 40/40 fps; departure leaves zero games, zero claim trees, zero session scripts. Follow-up the same night: the first cut of change 1 called `stop_session()` inline from `deactivate_viewer_stream()` and aborted a live seat (systemd status=134 = SIGABRT). `stop_session()` joins the capture and ring threads, but `deactivate_viewer_stream()` is reachable from a GStreamer streaming thread — `channel_closed` fires there when a data channel drops under a failed send — and joining a thread from itself throws std::system_error, which nothing catches, so std::terminate aborts the process. The teardown now goes through `g_idle_add(release_seat_main)`, matching how the capture and ring loops already marshal their errors onto the main loop; g_idle_add always queues rather than running inline, so it is safe from any thread. Verified with four back-to-back Watch Again (reload) cycles: relay uptime unbroken, zero restarts, four clean seat releases, and the session streaming at 39 fps afterwards.