Agentic AI-based Framework for Mitigating Premature Diagnostic Handoff and Silent Hallucination in Healthcare Applications

In the high-stakes theater of clinical medicine, the rapid ascent of Large Language Models (LLMs) and multi-agent systems has introduced a dangerous paradox. While these “Agentic AI” architectures are remarkably fluent, their optimization for natural language generation often comes at the direct expense of epistemic correctness. In a medical context, a system that sounds authoritative but lacks a structured evidentiary base is more than a technical failure—it is a patient safety liability.

The Reliability Gap

The central tension in medical AI lies in the “Reliability Gap”: the distance between a model’s linguistic confidence and its clinical grounding. Current mitigation strategies, such as Retrieval-Augmented Generation (RAG) and “LLM-as-a-judge” routing, remain fundamentally probabilistic. They lack deterministic guarantees that an agent has collected the necessary data to form a safe conclusion. This leads to two critical failure modes: Premature Diagnostic Handoff, where the system truncates the intake process before gathering life-saving context, and Silent Clinical Hallucinations, where the system provides unsafe recommendations without any internal metric for its own uncertainty. To achieve clinical utility, we must transition from probabilistic luck to deterministic, neuro-symbolic safeguards.

---

1. Anatomy of a Medical AI Failure: Two Case Studies

Unconstrained LLMs fail systematically when allowed to manage the transition from information gathering to diagnosis without external, symbolic oversight. These failures are often “silent” because they are buried within fluent, polite dialogue.

Failure Mode Comparison

Failure Mode	Clinical Consequence
Premature Diagnostic Handoff	An agent diagnoses “Acid Reflux” after gathering only 2/8 symptom dimensions (Onset and Duration). It fails to query the “Character” or “Radiation” of the pain, missing a potential Acute Coronary Syndrome (ACS).
Silent Clinical Hallucination	An agent prescribes Ibuprofen to a patient who reported daily Warfarin use. Lacking a hard-coded uncertainty check or a check_drug_interaction() tool call, it creates a major bleeding risk delivered directly to the patient.

---

2. The Framework: Replacing Probabilities with Deterministic Gates

To mitigate these systematic risks, we propose a multi-agent framework that replaces prompt-based inter-agent routing with Deterministic Orchestration Constraints. This architecture bridges the gap between neural generation and symbolic reliability by using a state vector ($V$)—a symbolic representation of clinical reality that governs the orchestration logic.

The framework utilizes three role-specialized agents:

• TriageNurse: Orchestrates structured symptom elicitation. It uses machine-verifiable field tags (e.g., [FIELD:Location]) to track the conversation’s progress.
• SeniorDiagnostician: Generates differential diagnoses. It operates in two modes: a stochastic mode ($K=5$ samples at $T=0.7$) for uncertainty quantification, and a deterministic mode ($T=0.0$) for the final clinical report.
• MedicalSafetySupervisor: The final deterministic gate. It reviews reports for clinical safety, utilizing a static prototype tool, check_drug_interaction(), and triggers recursive loops if safety thresholds are breached.

---

3. Safety Mechanism #1: The Neuro-Symbolic OLDCARTS Gate

The first defense is the M1 mechanism (Neuro-Symbolic State-Tracking). This gate addresses Premature Diagnostic Handoff by enforcing the OLDCARTS protocol, the gold standard clinical mnemonic for history taking.

The symbolic state vector ($V$) tracks eight Boolean dimensions:

• Onset: When the symptom began.
• Location: The anatomical site of the symptom.
• Duration: The length of the symptom episodes.
• Character: The quality of the sensation (e.g., dull, sharp).
• Aggravating/Alleviating factors: What triggers or relieves the symptom.
• Radiation: Whether the symptom moves to other regions.
• Timing: The frequency and pattern of occurrence.
• Severity: The intensity (e.g., on a 1–10 scale).

The system computes a completeness score $\sigma$, defined as the summation of the Boolean state vector $V$ ($\sigma = \sum_{i=1}^{8} V_i$). The orchestration layer imposes a hard constraint: the system is blocked from transitioning to the SeniorDiagnostician until $\sigma = 8$.

---

4. Safety Mechanism #2: Semantic Entropy (UQ)

The M2 mechanism (Uncertainty Quantification) is designed to intercept Silent Hallucinations. Traditional token-probability metrics fail in specialized medicine because of the “linguistic paraphrase problem”—the same diagnosis can be described in multiple semantically identical ways (e.g., “Myocardial Infarction” vs. “Heart Attack”).

To solve this, the framework generates $K=5$ independent diagnostic samples. It then applies Semantic Entropy (H), using a Natural Language Inference (NLI) clustering method. Specifically, the framework utilizes the cross-encoder/nli-deberta-v3-small model to determine if diagnosis labels are semantically equivalent.

Thresholding for Safety: The system employs two distinct thresholds. Bidirectional entailment for clustering is determined at $\tau_{NLI} = 0.7$. For routing to the patient, the system uses a threshold of $\tau = 0.5$. If $H > \tau$, the case is flagged as “uncertain” and diverted for a recursive supervisor review.

---

5. Results: The “Ablation A” Paradox and Evidence of Success

Evaluation on 150 simulated clinical cases (using Llama-3.1-70b-instruct) reveals that deterministic constraints provide significant precision gains.

Key Findings

• Precision Gain: The full architecture achieved 49.3% diagnostic precision—an 11.3 percentage-point improvement over the baseline.
• Uncertainty Correlation: A statistically significant negative correlation ($r = -0.181, p < 0.05$) was found between OLDCARTS completeness ($\sigma$) and semantic entropy (H), confirming that structured intake directly reduces epistemic uncertainty.

The Ablation A Paradox: A critical insight for safety researchers emerged from the ablation studies. Ablation A (Gate=T, K=1) actually performed worse than the unconstrained baseline. By enforcing a longer, more detailed triage history (M1) without the corresponding uncertainty filtering (M2), the system introduced clinical noise that a single deterministic sample could not accurately reconcile. This demonstrates that structured intake requires uncertainty quantification to be effective; more data is a liability if the model lacks a mechanism to quantify its own confusion.

---

6. Conclusion: The Path to Deployable Medical AI

Reliability in medical AI cannot be achieved through better prompting alone. It requires a fundamental shift toward neuro-symbolic architectures that enforce clinical protocols via deterministic gates. By combining structured intake verification (M1) with semantic uncertainty quantification (M2), we can create systems that are not just fluent, but fundamentally resilient.

Three Takeaways for AI Practitioners:

1. Enforce structured intake through hard symbolic constraints: Use the OLDCARTS state vector to prevent the SeniorDiagnostician from reasoning on fragmented data.
2. Quantify uncertainty via semantic clustering: Use specialized NLI models like deberta-v3-small to group equivalent labels, ensuring the uncertainty signal reflects clinical disagreement rather than linguistic variation.
3. Implement recursive safety loops with deterministic limits: Deploy specialized supervisor agents with a maximum of three (3) revision attempts to ensure the system resolves uncertainty or safety flags before output reaches the patient.

Read the full paper on arXiv · PDF