# Headless YouTube gateway VM

> **Current emergency asset, verified 2026-08-13:** `vr-emergency-standby` in `europe-west1-b` is
> `TERMINATED` and deletion-protected. Its persistent disk is retained, so storage still costs money
> while VM compute does not. No YouTube key or other backup destination is installed. It is therefore
> cold recovery capacity, not a warm or automatic failover encoder. Commissioning an explicit backup
> ingest destination and completing the public failover drill are required before this path may be
> described as redundant. See the
> [emergency runbook](../../../docs/trwc-emergency-broadcast-runbook.md).

A Compute Engine VM renders the live broadcast page in headless Chrome and pushes **RTMPS to
YouTube** with no desktop/OBS — [`gcp_gateway.py`](../live/gcp_gateway.py) (Xvfb + Chrome + FFmpeg),
designed in [youtube-live-demo-hr36.md](../../../docs/plans/youtube-live-demo-hr36.md). The OBS path
([`director_obs.py`](../live/director_obs.py)) is the primary production route. The local technical
scenes cover application and producer failures; this VM covers a different failure domain only after
an output destination has been configured.

```text
                          ┌─ all-in-one sim:  race_day --ksail-live  → BroadcastBridge :8766
gateway VM ── Chrome ─────┤
             (headless)   └─ DB replay:  cloud-sql-proxy → serve_v3_live --live :8765
   Xvfb :99 + Chrome(scene) ── FFmpeg (x11grab → x264) ──── RTMPS ────▶ YouTube
```

> **⚠️ Read this first — this Compute Engine/compute-driver path is SOFTWARE-only
> (verified 2026-07-18).**
> Headless Chrome **cannot** get a hardware **WebGL** context on a cloud NVIDIA L4 — every config
> fails with `Error creating WebGL context` (EGL and ANGLE/Vulkan under Xvfb, *and* even on a real
> NVIDIA `Xorg :0`; the driver + X server both come up fine, `DFP-0 connected`, but Chrome's WebGL
> context creation still fails). NVENC is also unavailable (`libnvidia-encode.so.1` not exposed by
> the DL compute image). **So the cloud path renders with SwiftShader (software) + x264 (CPU)** —
> good picture, correct data, but choppy and **no Cesium terrain** (stylised water only).
> **A GPU VM with this tested driver stack gives zero benefit—use a cheap CPU VM for this runbook.**
> Full-quality TRWC rendering remains on the **on-site RTX A4000 desktop**
> ([`docs/plans/`](../../../docs/plans/), memory `project_trwc_broadcast_architecture`), where
> Chrome's WebGL runs natively. **This dead end is specific to the *compute/Tesla* driver** — full-GPU
> cloud rendering may still be reachable via **Cloud Run GPU**, the **vWS/GRID driver**, or a
> **Windows GPU VM**. Those paths are unexhausted and laid out in
> **[§6](#6--appendix--full-gpu-rendering-in-the-cloud-the-untried-paths)**.
> For the current data-plane picture, don't re-chase the compute-driver headless path.

> **Capability update, 2026-07-23:** Cloud Run now supports L4 and RTX PRO 6000 GPUs and explicitly
> names 3D rendering as a use case. This is a new candidate, not proof that our Chromium/Cesium/audio/
> capture stack works there. The TRWC placement decision, two-hour proof-of-concept, promotion gates,
> and rollback rules are in
> [`docs/plans/cloud-broadcast-deployment.md`](../../../docs/plans/cloud-broadcast-deployment.md).

## 0 · One-time project prerequisites

- **Service-account grants** (only needed for the `--live` DB-replay mode; the all-in-one sim mode
  needs none). The VM's default compute SA needs Cloud SQL + the DSN secret:
  ```bash
  PROJ=apps-viewregatta
  SA="$(gcloud projects describe $PROJ --format='value(projectNumber)')-compute@developer.gserviceaccount.com"
  gcloud projects add-iam-policy-binding $PROJ --member="serviceAccount:$SA" --role=roles/cloudsql.client
  gcloud secrets add-iam-policy-binding VR_DATABASE_URL --project $PROJ \
      --member="serviceAccount:$SA" --role=roles/secretmanager.secretAccessor
  ```
- **YouTube** — channel live-enablement is done (verified 2026-06-28). You need a live event + its
  **raw stream key** (Studio → Go Live → Stream). **One encoder per key**; rotate the key if a stale
  encoder is stuck ("Preparing…").

## 1 · Create the VM

**Recommended — cheap CPU VM** (software render is the ceiling anyway, ~$0.27/hr):

```bash
gcloud compute instances create vr-gateway --project apps-viewregatta --zone europe-west4-c \
  --machine-type e2-standard-8 \
  --image-family ubuntu-2204-lts --image-project ubuntu-os-cloud \
  --boot-disk-size 60GB --scopes cloud-platform
```

> **Zone/quota gotcha:** `g2` (the only L4-bearing machine family) is **not offered in
> `europe-north1`** even though that region has L4 quota — use **`europe-west4-c`**. The global
> `GPUS_ALL_REGIONS` quota was 0 and had to be raised (IAM & Admin → Quotas → `GPUS_ALL_REGIONS` →
> 1). All of that only matters for the GPU path, which we no longer use — the CPU VM above needs
> none of it.

**GPU VM (historical — only if re-testing the GPU path, which does NOT work for WebGL):** the plain
`ubuntu-2204-lts` + `nvidia-driver-550` **fails to build** on kernel 6.8 (objtool `Error 241`). Use
a **pre-baked Deep Learning image** whose driver already matches:

```bash
gcloud compute instances create vr-gateway --project apps-viewregatta --zone europe-west4-c \
  --machine-type g2-standard-8 --accelerator type=nvidia-l4,count=1 \
  --image-family common-cu129-ubuntu-2204-nvidia-580 --image-project deeplearning-platform-release \
  --boot-disk-size 100GB --maintenance-policy TERMINATE --scopes cloud-platform
```

`nvidia-smi` then works (driver 580, L4 detected) and a GPU `Xorg :0` can be stood up — but Chrome
still can't create a WebGL context (see the warning at the top). Bills ~$1/hr.

**Stop or delete the VM when not streaming.**

**Fast rebuild — clone an existing gateway's disk** (skips the ~10 min bootstrap; the disk already
has Chrome + the python3.12 venv + code):

```bash
gcloud compute instances stop vr-gateway --zone europe-west4-c            # snapshot cleanly
gcloud compute images create vr-gw-clone --source-disk vr-gateway --source-disk-zone europe-west4-c --force
gcloud compute instances create vr-gateway-cpu --zone europe-west4-c --machine-type e2-standard-8 \
  --image vr-gw-clone --image-project apps-viewregatta --boot-disk-size 120GB --scopes cloud-platform
```

> **Clone gotcha — clear Chrome's profile lock.** The image carries the source VM's
> `~/.config/google-chrome/Singleton{Lock,Socket,Cookie}`, which pin the *old* PID/hostname. Chrome
> then refuses the profile ("The profile appears to be in use by another Google Chrome process … on
> another computer") and paints a **blank black frame** instead of the scene. Before the first run:
> `rm -f /root/.config/google-chrome/Singleton*`. (Cloning a GPU→CPU disk is fine — the unused
> NVIDIA bits just don't start.)

## 2 · Ship the code (private repo — no GitHub creds on the VM)

A git *bundle* carries the tree; `cesium-config.js` is gitignored so it goes separately (it only
matters for the on-site GPU path — the cloud software render can't show Cesium anyway):

```bash
git bundle create /tmp/trap.bundle HEAD main   # include HEAD, else clone checks out an empty tree
gcloud compute scp /tmp/trap.bundle vr-gateway:~ --zone europe-west4-c
gcloud compute ssh vr-gateway --zone europe-west4-c --command "git clone trap.bundle TRAP"
# optional (GPU/on-site path only):
gcloud compute scp v3/broadcast/core/web/cesium-config.js \
    vr-gateway:~/TRAP/v3/broadcast/core/web/ --zone europe-west4-c
```

## 3 · Bootstrap the VM

```bash
gcloud compute ssh vr-gateway --zone europe-west4-c
~/TRAP/v3/broadcast/deploy/setup_gateway_vm.sh     # apt deps, Chrome, python3.12 venv, cloud-sql-proxy
```

The bootstrap installs **python3.12** (via deadsnakes) for the venv — the stock 3.10 crashes
`serve_v3_live` on `datetime.UTC`. On a GPU VM also `sudo reboot` then `nvidia-smi`.

## 4 · Run

Two feed modes — pick one. **Both** need `TZ=Europe/Stockholm` (venue-local race clock; without it
the overlay shows UTC) and the software-render flags `--encoder x264 --no-gpu`.

### 4a · All-in-one TRWC simulation (no DB, no proxy)

The sim runs its own broadcast bridge on **:8766**. On the VM:

```bash
cd ~/TRAP && source .venv/bin/activate
# window 1 — the 12-boat Stockholm sim with the live race-office roster, broadcasting on :8766
TZ=Europe/Stockholm python -m v3.sim.race_day --sandbox --target prod \
    --ksail --ksail-live --seed 7        # drops MarkSetBot fleet in sandbox; DEF/USA nation mapping

# window 2 — headless gateway → YouTube (points at the sim's :8766)
TZ=Europe/Stockholm YOUTUBE_STREAM_KEY=… \
  python v3/broadcast/live/gcp_gateway.py \
    --url 'http://localhost:8766/core/web/index.html?feed=live' \
    --chromium google-chrome --encoder x264 --no-gpu --fps 24
```

### 4b · DB replay of a real/prod race (`serve_v3_live --live`)

```bash
# window 1 — Cloud SQL proxy (leave open)
~/cloud-sql-proxy apps-viewregatta:europe-north1:vr-v3-db --port 5432

# window 2 — serve the live scene from the DB on :8765
VR_DATABASE_URL="$(gcloud secrets versions access latest --secret=VR_DATABASE_URL)" \
  python v3/broadcast/live/serve_v3_live.py --live --port 8765 --instance <instance>

# window 3 — gateway → YouTube (points at :8765)
TZ=Europe/Stockholm YOUTUBE_STREAM_KEY=… \
  python v3/broadcast/live/gcp_gateway.py \
    --url 'http://localhost:8765/core/web/index.html?feed=live' \
    --chromium google-chrome --encoder x264 --no-gpu --fps 24
```

`gcp_gateway.py` already emits **CFR** with periodic keyframes
(`-vsync cfr -r <fps> -force_key_frames expr:gte(t,n_forced*2)`) — without that YouTube gets stuck
on "Preparing stream". Verify on the YouTube Live dashboard: **"Stream health — excellent"**; expect
~10–30 s CDN latency.

### Restart cleanly (kills orphaned Chrome/FFmpeg/Xvfb that survive `pkill`)

```bash
for p in $(pgrep -f 'gcp_gateway|ffmpeg|Xvfb|google-chrome'); do kill -9 "$p"; done; sleep 5
```

## 5 · Smoke tests (each isolates one layer)

```bash
# egress + key only (proves the VM can reach YouTube, independent of the render stack):
ffmpeg -re -f lavfi -i testsrc2=size=1920x1080:rate=30 -f lavfi -i sine=frequency=440 \
  -c:v libx264 -preset veryfast -b:v 4500k -g 60 -pix_fmt yuv420p -c:a aac -ar 44100 \
  -t 60 -f flv "rtmps://a.rtmps.youtube.com/live2/$YOUTUBE_STREAM_KEY"

# render only (grab one frame off the Xvfb display — >200 KB PNG = a real scene, ~13 KB = WebGL error):
DISPLAY=:99 ffmpeg -y -f x11grab -video_size 1920x1080 -i :99 -frames:v 1 /tmp/shot.png; ls -l /tmp/shot.png
```

## 6 · Appendix — full-GPU rendering in the cloud (the untried paths)

The top-of-file warning says cloud rendering is *software-only*. That is only proven for **one**
stack: headless Chrome on the **compute/Tesla** NVIDIA driver (the DL image `…-nvidia-580`), where
Chrome's WebGL context creation fails under EGL, ANGLE/Vulkan, **and** a real `Xorg :0`. That does
**not** mean full-GPU cloud rendering is impossible — it means we used the **wrong driver**. The
compute driver is not qualified for the OpenGL/display path; GCP ships a separate **NVIDIA RTX
Virtual Workstation (vWS / GRID)** driver *specifically* for Vulkan/OpenGL/Direct3D visualization on
L4, and GCP's own docs call out **Chromium GPU acceleration** with it on **Ubuntu 22.04**
([install-grid-drivers](https://docs.cloud.google.com/compute/docs/gpus/install-grid-drivers),
[grid-drivers-table](https://docs.cloud.google.com/compute/docs/gpus/grid-drivers-table)). So there
are real, unexhausted options. Ranked by likelihood of delivering smooth + Cesium + NVENC:

| # | Path | Why it should work | Proven? | Effort | Extra cost |
| - | ---- | ------------------ | ------- | ------ | --------- |
| 1 | **Cloud Run GPU** (L4 or RTX PRO 6000) | Managed GPU with drivers preinstalled; 3D rendering is an advertised workload. Requires a new service/capture integration and supported-region placement. | untried | med | full-lifecycle GPU + CPU/RAM + redundancy |
| 2 | **Linux + vWS/GRID driver** (Ubuntu 22.04, Xorg) | The qualified GL driver; GCP docs name Chromium GPU acceleration on 22.04. | untried | low–med | VM + vWS license |
| 3 | **Windows Server + GRID driver** | Chrome→ANGLE→D3D11 renders on-GPU with a virtual display; the standard GPU-Chrome path. NVENC included. | untried | med | VM + vWS license |
| 4 | **VirtualGL / new-headless Chrome on vWS** | Redirect or expose Chrome GL using the qualified visualization driver. | untried | med | VM + vWS license |
| 5 | **On-site RTX A4000 desktop** (not cloud) | Native desktop GPU + compositor; Chrome WebGL + Cesium + NVENC are known-good. | ✅ primary | owned hardware | none incremental |

The Compute Engine vWS/GRID and Windows drivers include the NVENC user-space library that was absent
from the tested compute image. Cloud Run GPU encoder exposure must be tested rather than assumed.
In every path, require positive encoder identification before selecting `--encoder nvenc`.

### Recommended order to try

For the existing Compute Engine gateway, test the **Linux vWS/GRID path** first. For the product-level
cloud-render experiment, evaluate **Cloud Run GPU** in parallel under the canonical two-hour gate.
Do not interpret a successful CUDA probe as success: the browser must positively report hardware
WebGL, the intended hardware encoder must be active, and the return feed must remain synchronized.

Compute Engine vWS/GRID recipe:

```bash
# 1. Create a G2/L4 VM WITH a vWS license (plain Ubuntu 22.04, NOT the compute DL image):
gcloud compute instances create vr-gateway-gpu --project apps-viewregatta --zone europe-west4-c \
  --machine-type g2-standard-8 --accelerator type=nvidia-l4,count=1 \
  --image-family ubuntu-2204-lts --image-project ubuntu-os-cloud \
  --boot-disk-size 100GB --maintenance-policy TERMINATE --scopes cloud-platform
# (a vWS license is attached automatically when you install the vWS/GRID driver below; it bills per GPU-hour)

# 2. Install the vWS/GRID driver (NOT the Tesla/compute driver):
sudo apt-get update && sudo apt-get install -y build-essential "linux-headers-$(uname -r)"
LATEST=$(gcloud storage ls gs://nvidia-drivers-us-public/GRID/ | sort -V | tail -1)   # newest vGPU dir
gcloud storage cp "${LATEST}NVIDIA-Linux-x86_64-"*-grid.run /tmp/grid.run   # pick the -grid.run file
sudo bash /tmp/grid.run --dkms -s              # answer: 32-bit=Yes, modify xorg=No
sudo reboot
nvidia-smi                                     # then set up Xorg :0 (see the historical GPU note in §1)

# 3. Point gcp_gateway.py at the real display and drop --no-gpu:
#    run with --fps 30 (GPU can hold it) and, once NVENC verifies, --encoder nvenc.
```

Then re-run the **render-only smoke test** (§5): a ≥200 KB frame with Cesium terrain = success. If the
frame is still ~13 KB "Error creating WebGL context" even on the *vWS* driver, escalate to Option 2
(Windows), which is architecturally the most Chrome-friendly GPU path.

**Cost note:** Compute Engine visualization options add the **vWS license** per GPU-hour. Cloud Run
GPU requires instance-based billing and charges the GPU through the instance lifecycle; warm
redundancy, cross-region GCS/Cloud SQL movement, relay/transcode, and egress must be included. These
paths
only make sense when you actually need smooth motion or Cesium terrain — for the data-plane-only
picture, the CPU VM in §1 is the right tool. If cloud full-GPU proves fiddly under time pressure,
**The on-site RTX desktop is the guaranteed path** and is already the TRWC production
design — cloud stays the CPU data plane.

Sources: [Install vWS/GRID drivers](https://docs.cloud.google.com/compute/docs/gpus/install-grid-drivers) ·
[vWS driver table](https://docs.cloud.google.com/compute/docs/gpus/grid-drivers-table) ·
[Linux GPU workstation](https://cloud.google.com/compute/docs/virtual-workstation/linux-gpu) ·
[Windows GPU workstation](https://docs.cloud.google.com/compute/docs/virtual-workstation/windows-gpu) ·
[Cloud Run GPU](https://docs.cloud.google.com/run/docs/configuring/services/gpu)