SOAR — A Voice-Directed Surgical Co-Pilot

One line

SOAR is a voice-directed surgical co-pilot: a Python FastAPI WebSocket gateway bridging a browser console to Vertex AI's Gemini Live API through Google's Agent Development Kit, exposing one Orchestrator Coordinator and nine specialist sub-agents that call twenty-six in-process tools to put patient data, CT slices, a 3D anatomy model, phase checklists, drug-safety checks, blood-loss tracking and an operative log on screen — in one browser session, served from a Cloud Run container.

That sentence took me longer to write than some of the code.

Why voice

In an operating theatre the surgeon's hands are the one input device you cannot borrow. Every second spent looking away from the field to scrub-out, click, and scrub back in is a second of anaesthesia. So the interaction model is: say what you need, and the right panel updates.

The interesting engineering is not speech recognition — Gemini Live handles that. It is deciding which of nine specialists should own the request, and making sure a tool call cannot render something nobody asked for.

Architecture

One browser session opens one WebSocket. Microphone PCM goes up at 16 kHz mono, buffered into 100 ms chunks; model audio comes down at 24 kHz through an AudioWorklet. JPEG frames from screen capture ride the same socket, one per second, but only while the Screen Share Advisory Agent is the active author.

The agent hierarchy lives in app/soar_orchestrator/agent.py. Twenty-six tool functions live in tools.py as plain Python that mutates eight in-memory fixtures and returns render commands — structured descriptions of what the browser should display. The browser is dumb on purpose: it decodes a function-call event and dispatches it to the matching display module (CT viewer, 3D anatomy, clinical panel, checklist, log, summary, screen-share controller). Each module is a self-contained IIFE that owns its own DOM node and injects its own styles lazily.

Argument whitelisting and response-schema validation are enforced by ADK before- and after-tool callbacks. That is the safety story: a model cannot invent an argument the tool never declared.

The five paths that cover almost everything

1. Console boot — browser hits the landing page, follows the link to /console, opens a socket carrying freshly generated user and session identifiers.
2. Session establishment — the server resumes or creates a session, builds a live run configuration for bidirectional audio, and starts two cooperating async pumps, upstream and downstream.
3. Voice round-trip — audio up, transcription and function calls down, tools resolved in-process, render commands dispatched to panels.
4. Specialist routing — the coordinator emits a transfer-to-agent call, the specialist becomes the author of subsequent events, and the console highlights the matching agent chip so the surgeon can see who is answering.
5. Continuous deployment — a push to main triggers Cloud Build, which tags the image by commit SHA and latest, and redeploys the Cloud Run service with two vCPU, 2 GiB, and a sixty-minute request timeout.

What surprised me

Making the routing visible mattered more than making it correct. Early on, a specialist would take over silently and the whole thing felt like it was guessing. Adding agent chips and a routing log — literally showing which of the nine is talking — changed how much I trusted my own system, before I changed a single line of the routing logic.

The other surprise was dedup. Models re-emit tool calls. A cache on the browser side, keyed by call signature, was the difference between a console that flickers and one that looks calm.

Everything runs on fixtures. No real patient data touches it, and that is deliberate for a student project — I would rather demonstrate the interaction model honestly than pretend to a clinical claim I cannot support.