AI triage agent cut average response time by 73%

The challenge

Helix's patient support team was drowning. A small group of nurses was answering thousands of questions a week — appointment scheduling, medication refill inquiries, post-visit follow-ups, insurance questions — and response times were creeping past 24 hours. CSAT was sliding, and the clinical staff was burning out. They needed AI assistance. But in healthcare, hallucinations aren't an acceptable trade-off, and every answer touching diagnosis, dosage, or prognosis had to be reviewed by a licensed clinician.

Our approach

• Built an eval harness on day one with 300+ real (de-identified) patient questions, each scored for correctness, safety, and tone by a panel of clinical reviewers. Every model iteration was measured against this baseline before any deployment.
• Designed a four-layer guardrail system: (1) input filter regex for PII/PHI leakage, (2) intent classifier that routed clinical-touch questions to human-only, (3) output validator that checked every generated response against the knowledge base, (4) human-in-the-loop escalation for any response scoring below 0.95 confidence.
• Used pgvector for retrieval-augmented generation against a curated knowledge base of 2,000+ verified clinical documents, policy PDFs, and FAQ entries. The model was explicitly prohibited from generating answers from its own pre-trained knowledge — every response had to cite a retrieved source.
• Built a multilingual router that detected the patient's language on inbound, retrieved answers from the English knowledge base, and generated the response in the patient's language using a translation-augmented chain. The round-trip stayed under 3 seconds for 12 supported languages.
• Instrumented every inference with structured logging: prompt, retrieved chunks, generated response, guardrail scores, latency, and escalation decision. A Grafana dashboard let the operations team monitor accuracy, coverage, and escalation rates in near-real-time.
• Ran a three-week shadow deployment alongside the human team before any patient-facing traffic. The AI's answers were surfaced to nurses as suggestions, not sent to patients. We collected 4,000+ comparison pairs and tuned thresholds before flipping the switch.
• Set up scheduled weekly eval re-runs on the full 300-question test set using Modal's serverless GPUs. Each run generated a regression report that flagged any accuracy, safety, or latency degradation before it reached production.

The outcome

73% reduction in average response time (from 24 hours to under 6.5 hours). The AI now handles 60% of inbound questions end-to-end without human touch, freeing nurses for the conversations that actually need clinical judgment. CSAT climbed 38 points in the first quarter. Zero clinically incorrect responses were detected in the first six months of production operation. The guardrail system maintained 99.7% accuracy in routing clinical-touch questions to human reviewers.