# IAM-private live broadcast-feed producer on Cloud Run > **Production state verified 2026-08-15:** `trap-v3-producer` is Ready in `europe-north1` and remains > a service-level singleton. Cloud Run's Invoker IAM check is enabled and there is no `allUsers` > invoker grant: anonymous requests to both `broadcast.viewregatta.com` and the `run.app` origins > receive `403` before the container. The current race-day consumer is the loopback-only authenticated > relay on the trusted OBS host; direct Cloud Run URLs must never be stored in an OBS Browser source. Runs the **existing `serve_v3_live` producer** as a Cloud Run service in `europe-north1`, co-located with the production database. It serves the 3D scene and live **SSE** feed to an explicitly authorised render host. The render host holds no database connection or Cloud SQL proxy; its signed-in Google identity has only the Cloud Run invocation capability needed by the relay. ``` PHONES → ingest → Postgres (europe-north1) ▲ in-region reads (sub-ms) [Cloud Run] trap-v3-producer = serve_v3_live --live │ IAM + TLS, no anonymous invoker render PC ── private_feed_relay.py (127.0.0.1:8766) │ loopback only └── OBS Browser sources ``` ## 1. Build the image **Nothing builds this image automatically. You build it, by hand, from the repo root:** ```bash gcloud builds submit --config v3/broadcast/deploy/cloudbuild.yaml \ --substitutions=_SHA=$(git rev-parse HEAD) --region=europe-north1 --project apps-viewregatta . ``` `_SHA` is required, and it is what makes the deployed service able to say which commit it is — `PRODUCER_CLOUDRUN`'s own §4 verify, and the operations monitor's "is this the frozen build?" check, both read that tag. Against `:latest` neither can answer. **This changed on 2026-08-07 and the change is easy to miss.** CI used to build this image on every push to `main` (`deploy-v3.yml`), which is what kept it current; that push trigger was removed when GitHub Actions became unavailable — see [`../../../docs/trwc/deploy-without-github.md`](../../../docs/trwc/deploy-without-github.md). So the sentence that used to be here — *"CI builds it on every push"* — is no longer true, and this service has lost the one thing that stopped it ageing. That matters more here than for the other services because a **newly built image is not promoted automatically**: this is a live broadcast, and rolling it on every push would cut the feed mid-race. The service itself is deployed; the gap is between its running revision and a newer image. It is the image most likely to lag the repo and has reached 183 commits behind before. **Promote a build when the racing allows — §2 — and check the running tag against the commit you meant to ship.** The repo-root **`.gcloudignore`** trims the upload to the four subtrees the Dockerfile COPYs (~30 MB). Without it `gcloud builds submit` uploads the whole tree (77+ GiB: `.git`, Android build outputs, media) — if you ever see "Creating temporary archive of … GiB", that file is missing or mis-scoped. ## 2. Deploy — the settings that matter Two shapes. **A race day wants FOLLOW**: one producer for the whole programme, staying with the race on air until it has a winner and then moving to the next, with the renderer swapping the race in place and never reloading. Leave `VR_INSTANCE` **empty** — the image omits `--instance` entirely then, which is what selects the follow path. **Follow needs no course id.** A race carries its own `courseId`, so the producer takes each race's course from the race; asking the deploy to name one in advance would be asking it to know something the race plane already says, and to be wrong the day the course changes. Scope by `--owner` — whose regatta this broadcast shows. Pass `VR_COURSE` only to pin one course when several race at once. An **empty `VR_COURSE` omits `--course` entirely**, exactly as an empty `VR_INSTANCE` omits `--instance` — so `VR_COURSE=` (this page's own race-day shape) and removing the variable are equivalent, and both mean "no course scope". This sentence is load-bearing history: the image used to pass `--course "$VR_COURSE"` unconditionally, and the empty string reached `StartedRaces` as a course *filter* no real course id could match — the producer sat on "Waiting for a race" for ever while armed races sailed past it, with nothing anywhere saying why. That now holds for the **live wind feed** too, which it did not before. `--wind` writes readings against a course, and with no `VR_COURSE` it had none, so the shape documented right here started with `Wind feed: OFF - needs a course_id to attach WindReadings to` and broadcast a whole regatta with no wind. One line, in a boot log, once. The feed resolves its course **per poll** instead, from the race on air — so `--wind` and follow mode finally mean together what each means alone. Between heats nothing is on air and no reading is written: weather filed against a race that is over is worse than a gap. **Promoting a build onto the EXISTING service — `--image` and nothing else:** ```bash gcloud run deploy trap-v3-producer --image europe-north1-docker.pkg.dev/apps-viewregatta/trap/trap-v3-producer: --project apps-viewregatta --region europe-north1 --no-allow-unauthenticated --invoker-iam-check ``` The full command below is for **first creation only**. Re-running it on a live service re-states the whole secret set: its `--set-secrets … :latest` REPLACES the pinned `VR_CESIUM_ION_TOKEN:1` that is the only thing keeping the service booting (`latest` resolves to a DISABLED v2 of that secret) — the deploy looks green and the next revision dies on the token. The service already carries its secrets, the Cloud SQL attachment, `--no-cpu-throttling`, service min/max 1 and the timeout; a promote must change the image and nothing else. To change env afterwards, use `gcloud run services update … --update-env-vars` (never `--set-env-vars`, which replaces the set). ```bash # FIRST CREATION of a race-day follow producer (existing service: use the promote command above) gcloud run deploy trap-v3-producer --image europe-north1-docker.pkg.dev/apps-viewregatta/trap/trap-v3-producer: --project apps-viewregatta --region europe-north1 --port 8080 --service-account trap-v3-runtime@apps-viewregatta.iam.gserviceaccount.com --set-secrets VR_DATABASE_URL=VR_DATABASE_URL:latest,VR_PRODUCER_READ_TOKEN=VR_PRODUCER_READ_TOKEN:latest --set-cloudsql-instances apps-viewregatta:europe-north1:vr-v3-db --set-env-vars "VR_INSTANCE=,VR_COURSE=,VR_EXTRA_ARGS=--follow --owner --wind" --no-cpu-throttling --min 1 --max 1 --min-instances default --max-instances default --scaling auto --concurrency 80 --timeout 3600 --no-allow-unauthenticated --invoker-iam-check ``` Before the first gun and between heats it publishes the **waiting picture** — the venue and the fleet of the next allocated race, `idle` set so the renderer runs its area-panning look-down camera. It answers with nothing only while nobody is allocated yet. (That used to be missing outside the simulator, so the OBS browser source sat blank until the first gun.) **Pinning ONE race** is the other shape — a single instance, no schedule: ```bash gcloud run deploy trap-v3-producer \ --image europe-north1-docker.pkg.dev/apps-viewregatta/trap/trap-v3-producer \ --project apps-viewregatta --region europe-north1 --port 8080 \ --service-account trap-v3-runtime@apps-viewregatta.iam.gserviceaccount.com \ --set-secrets VR_DATABASE_URL=VR_DATABASE_URL:latest,VR_PRODUCER_READ_TOKEN=VR_PRODUCER_READ_TOKEN:latest \ --set-cloudsql-instances apps-viewregatta:europe-north1:vr-v3-db \ --set-env-vars "VR_INSTANCE=ulf:walk,VR_COURSE=ulf:walk,VR_EXTRA_ARGS=--wind" \ --no-cpu-throttling \ --min 1 --max 1 --min-instances default --max-instances default --scaling auto \ --concurrency 80 \ --timeout 3600 \ --no-allow-unauthenticated \ --invoker-iam-check ``` Why each non-obvious flag: - **`--service-account trap-v3-runtime@…`** — pin the **same runtime SA the ingest service uses**. It already holds `roles/secretmanager.secretAccessor` on `VR_DATABASE_URL` and `roles/cloudsql.client`; the default Compute SA usually does **not**, so without this the producer boots but can't read the DB secret or open the Cloud SQL socket (403 / connection refused). Reusing the ingest SA needs no new IAM grants. - **`--no-cpu-throttling`** — the producer's `broadcast_live` loop runs in a **background thread**; Cloud Run throttles CPU to ~0 between requests by default, which would freeze it. CPU must stay allocated. **Without this the feed stalls — this is the single most important flag.** - **`--min 1 --max 1` with revision min/max set to `default`** — exactly **one producer across the service**. The service minimum keeps the traffic-serving revision warm and the service maximum prevents duplicate feed/scoring. Do not use `--min-instances 1` here: that setting is immutable on every revision, so old zero-traffic revisions remain alive after promotion and continue polling, scoring and synthesising speech. On 2026-08-13 fourteen retired revisions exhausted the shared TTS request quota in parallel. One active instance fans out to all SSE consumers via `--concurrency`. - **`--timeout 3600`** — Cloud Run's max request length; the SSE stream is one long request. The scene reconnects when it closes (verify the reconnect re-sends `init` — see plan §6 open item). - **`--set-cloudsql-instances`** + `VR_DATABASE_URL` secret — the **in-region** DB socket (`host=/cloudsql/…`), same as the ingest service. No proxy involved. - **`VR_PRODUCER_READ_TOKEN`** — the dedicated capability the producer adds as `X-VR-Producer-Token` when it polls the public read base. It is separate from the owner's bearer token and authorizes only raw `positions`/`track`/`tracks` reads. - **`--no-allow-unauthenticated --invoker-iam-check`** — both removes the legacy public invoker grant and enables Cloud Run's current request-time IAM check. These flags are restated on every promotion by `deploy-all.ps1`; omitting either can recreate the raw-data exposure during service repair. - `VR_EXTRA_ARGS` — optional; e.g. `--wind` or `--wind --simple`. **Do not** add `--ais` (the image has no `websockets`). - **The voice needs no flag here.** The container runs `--live`, and `--neural-tts` is **on by default for a live broadcast** — it used to be off, and every path that forgot it went out with a perfect picture and no sound (2026-08-11). It needs the Cloud Text-to-Speech API enabled on the project; ADC comes from the runtime SA. A failure degrades to the renderer's own browser voice with a line in the log, never to no show, so a missing grant costs quality rather than the broadcast. Pass `--no-neural-tts` in `VR_EXTRA_ARGS` to turn it off. Note the renderer half is separate and equally required: an OBS browser source only speaks when its URL carries `trustedAutoplayAudio=1` (the shipped `obs/*.json` layouts do). ### Producer-only race movement: one-time safe rollout When `VR_PRODUCER_READ_TOKEN` is absent from the server, existing public movement reads continue to work. As soon as the secret is attached to the server, anonymous requests to `positions`, `track`, `tracks`, and `tracks.list` receive `401` **before admission and before SQL**, while the producer's matching header is allowed to read both public and private races. Course, roster, camera and other viewer routes keep their existing visibility. The order is a broadcast invariant. Perform both deploys between races, and never attach the secret to the server first: ```bash # 1. Promote the new producer and add the capability in the same restart. gcloud run deploy trap-v3-producer \ --image europe-north1-docker.pkg.dev/apps-viewregatta/trap/trap-v3-producer: \ --project apps-viewregatta --region europe-north1 \ --update-secrets VR_PRODUCER_READ_TOKEN=VR_PRODUCER_READ_TOKEN:latest # 2. Only after the producer is healthy, deploy the matching server build with the capability. gcloud run deploy trap-v3-server \ --image europe-north1-docker.pkg.dev/apps-viewregatta/trap/trap-v3-server: \ --project apps-viewregatta --region europe-north1 \ --update-secrets VR_PRODUCER_READ_TOKEN=VR_PRODUCER_READ_TOKEN:latest ``` Verify an anonymous raw-track request now returns `401`, while the producer continues publishing fresh frames without `producer_auth_required` errors. Do not print or pass the secret value on the command line; Cloud Run resolves it from Secret Manager. The server workflow retains the binding on later deploys. If the producer stops reading after activation, restore service immediately by removing enforcement from the server; the code treats an absent server secret as the staged, backward-compatible mode: ```bash gcloud run services update trap-v3-server --project apps-viewregatta \ --region europe-north1 --remove-secrets VR_PRODUCER_READ_TOKEN ``` ## 3. Connect the trusted OBS host through the private relay Grant `roles/run.invoker` on **this service only** to the Google identity used on the render host. Do not grant project-wide invocation and do not create a service-account key. On that host: ```powershell gcloud auth login gcloud config set project apps-viewregatta & v3/broadcast/deploy/install_private_feed_relay.ps1 ``` The installer records the loopback race-day variables for the current Windows user and installs least-privilege login tasks with restart-on-failure for the relay and the data-only local standings card (`127.0.0.1:8767`). The relay launcher resolves the current `run.app` origin, obtains a short-lived Google identity token, preflights `/livez`, and then listens only on IPv4 loopback. Tokens are refreshed before expiry and are never written to the log, URL, OBS configuration, or browser storage. Configure race-day tools and OBS with the loopback origin: ```powershell $env:VR_RACE_DAY = "1" $env:VR_PRIVATE_FEED_RELAY = "1" $env:VR_PRODUCER_BASE = "http://127.0.0.1:8766" python -m v3.broadcast.live.secure_obs_sources --apply ``` ```text http://127.0.0.1:8766/core/web/index.html?feed=live&lang=en http://127.0.0.1:8766/core/web/overlay.html?feed=live ``` The relay permits GET/HEAD and one bounded JSON POST to `/client-stats` for renderer health. It rejects all other writes, cross-site browser requests and non-loopback Host headers, strips CORS, and marks responses `no-store`. Raw scene data therefore travels over IAM-authenticated TLS outside the render host and never becomes a public or LAN endpoint. Loopback HTTP does not leave the host's network stack. Do not point OBS at `broadcast.viewregatta.com` or a `run.app` URL. Those anonymous requests must remain `403`. The PC still holds no database connection or Cloud SQL proxy. ## 4. Verify - Anonymous `GET https://broadcast.viewregatta.com/events` and both `run.app` origins return `403`. - `GET http://127.0.0.1:8766/livez` on the authorised render host returns `{"ok": true, "clients": N}` with `Cache-Control: no-store` and no `Access-Control-Allow-Origin` header. - Open the loopback programme URL in OBS Preview: boats move at approximately 1 Hz. - The producer logs (`gcloud logging read … trap-v3-producer`) should show one publish tick ~every second — not the 30 s stalls of the PC-side run. - `GET http://127.0.0.1:8766/stats.json` shows who is actually rendering and how well. Never put a feed token in a URL; production authentication is the relay's Google identity header. ### The renderers' back-channel Every renderer POSTs a small health sample to `/client-stats` every five seconds ([`core/web/client_beacon.js`](../core/web/client_beacon.js)); `renderer_registry.py` keeps the newest per renderer, expires any that stops reporting, and serves them on `/stats.json` behind the same feed token as `/events`. **Two counts, two different questions.** `clients` is open SSE subscribers; `renderers` is who is drawing. A socket can be open behind a page that has stopped — which is precisely the gap this closes, and the reason a subscriber count on its own was never the health signal it looked like. It exists because every other measurement watches from outside. OBS reports its own compositor and this service reports itself; neither can see whether the *renderer* is keeping up, and a browser source drawing at 12 fps inside a 30 fps composite reports 30 fps to all of them. It matters most where we have least reach — a partner rendering for themselves has no OBS of ours in the room. Consumed by [`v3/ops`](../../ops/README.md), which turns it into advisories. Note that the monitor reports renderers as `unknown` until a build carrying this is **promoted** — a producer without `/stats.json` is not a producer with no renderers. ## 5. Open items to close (from the plan) - **SSE reconnect across the 60-min timeout — VERIFIED OK (2026-07-19), no change needed.** `LiveFeedServer` keeps a `_history` backlog that always starts with the latest `init` (`_fan_out(reset=True)`), and `_serve_events` replays that backlog to every new subscriber before live deltas. The browser's `EventSource` auto-reconnects and `handle_error` swallows the disconnect — so a client dropped at the 60-min cap reconnects and gets `init` + recent deltas; it never goes blank. *Reconnect-snap polish — DONE:* the live `run()` passes `LiveFeedServer(live_backlog=LIVE_BACKLOG)`, so a reconnecting live viewer gets `init` + only the last few seconds of deltas and snaps straight to now instead of replaying up to ~2 h (regenerate/VOD keeps the full `MAX_BACKLOG` for late joiners). `LIVE_BACKLOG` **must exceed the renderer's presentation lag** (`race_queue.PRESENTATION_LAG_S`, 22 s) — the client presents `frontier − lag`, so at the old value of 5 a reconnect had nothing behind its cursor and the picture held one frame for ~17 s. Now 30. - **One race after another — `--follow` — is DONE and is the race-day shape (§2).** It stays with the race on air until that race has a winner (plus a short hold on the finish) or the operator closes it, then puts up the next one — joined in progress, with its elapsed track back-filled so the standings are whole (`live/race_queue.py` `OnAirQueue`/`StartedRaces`, `serve_v3_live.broadcast_following`). The renderer swaps the race in place, it never reloads (`live/web/live_feed.js` → `onRaceChange`). The joined race's last `PRESENTATION_LAG_S` seconds go on the wire BEFORE its first live tick (`coverage_window`, `_OnAirRace._publish_history`) — without them the cursor sits below the published history and the whole picture, camera included, froze for the delay at every hand-over. The drone flies the seam as one shot (`core/web/handover_camera.js`) and the production layer names the new race and its clock over it (`scoring/programme.py` `frame_race`). Pinning a single `VR_INSTANCE` still works and is unchanged. - **Per-instance vs multi-instance** — `--follow` covers races run in SEQUENCE. For several races broadcast CONCURRENTLY ([parallel-race-broadcasts](../../../docs/plans/parallel-race-broadcasts.md)) either deploy one service per race or teach the producer to key the SSE by an `?instance=` query param. - **Dependency check — DONE (2026-07-19).** The producer chain (`streaming_score`, `score_race`, `simple_commentary`, `live_feed_server`, `v3_adapter`, `broadcast_feed`) was verified to import with `anthropic` **blocked** — the AI commentary path (`ai_director`/`ai_author`) imports the SDK lazily with a no-SDK fallback and is unused by the live producer. So the image needs **no deps beyond `v3/ingest/requirements.txt`**. (Don't pass `--ai`; and `--ais` stays off — no `websockets` intent for this service, though it happens to be in the ingest reqs.) - **Private feed authentication — CLOSED 2026-08-15.** Cloud Run Invoker IAM rejects anonymous traffic and the trusted render host uses a loopback-only identity-token relay. URL tokens are not part of the production path.