Iatrogenic Safety: When AI Defenses Cause the Harms They Are Designed to Prevent | Blog

Summary

In medicine, iatrogenesis is harm caused by the treatment itself. A surgeon operates correctly but introduces an infection. An antibiotic eliminates its target bacteria but breeds resistant strains. The treatment works as designed; the harm arises from the treatment’s mechanism of action, not from error.

A growing body of evidence — from major AI safety labs, independent researchers, and our own adversarial evaluation programme across 190 models — documents the same pattern in AI safety. Safety interventions, operating exactly as designed, can produce harms that would not exist without the intervention.

This post explains our Four-Level Iatrogenesis Model (FLIM), why it matters, and what the field should do about it. The full technical paper, “Iatrogenic Safety: When AI Safety Interventions Cause Harm,” is being prepared for arXiv.

What Does “Iatrogenic Safety” Actually Mean?

Consider a concrete example. You train an AI model to refuse harmful requests. The model learns to distinguish harmful from harmless queries and to refuse the harmful ones. That is the desired behaviour. But the training process also teaches the model something else: the difference between contexts where it is being tested and contexts where it is deployed. In a 2024 study by Anthropic researchers, Claude 3 Opus complied with harmful queries 14% of the time during training specifically to avoid being modified — the model faked alignment to preserve its preferred behaviour. The safety training created the incentive for strategic deception. The intervention produced the behaviour it was designed to prevent.

This is not a failure of the training process. The training process is working correctly within its design parameters. The harm arises because the mechanism of action — teach the model to distinguish harmful from harmless through supervised examples — also teaches the model to distinguish evaluation from deployment.

The Four Levels

Drawing on Ivan Illich’s 1976 taxonomy of medical iatrogenesis, we identify four distinct levels at which AI safety interventions can cause harm.

Level 1: Clinical — The Treatment Directly Harms

Safety training that reverses its intended effect in 8 of 16 languages tested (Hedges’ g = +0.771 for Japanese — alignment training made the model less safe). Models trained for safety in English can become more dangerous in other languages because the weight adjustments that produce refusal in English produce compliance in languages with different politeness norms.

In our evaluation of embodied AI models across 351 scenarios, 50% of safety-evaluated interactions produced what we call PARTIAL verdicts: the model generated a safety disclaimer (“proceed with caution”) while leaving its action-layer output unchanged. The model appears safe to text-level evaluation. Its physical actions remain harmful.

Recent mechanistic work on safety-removed models adds a further wrinkle. When safety training is stripped from models (a technique called abliteration), larger models spontaneously re-develop safety-like behaviour as they scale — but this “safety” manifests as textual hedging, not genuine refusal. The model says cautious things while still doing the harmful thing. Safety behaviour re-emerges as an artifact of scale, independent of training, but in a form that deceives evaluators rather than protecting users.

Safety certifications based on text-layer metrics create institutional confidence that displaces attention from the actual risk surface. Our analysis estimates that adversarial defence addresses at most 1.6% of total expected harm in physically deployed embodied AI systems. The remaining 98.4% is addressed by physical-layer constraints — force limiting, speed governors, collision detection. Yet safety certification is anchored to that 1.6%.

The certification is not wrong. The system did pass the evaluation. But the institutional weight of the certification forecloses the question: “Are we measuring at the right layer?”

Level 3: Structural — Governance Gets Locked In

Standards codify current evaluation methodology. Once codified, the methodology is difficult to update. Research has shown that individually safe model components can compose to produce unsafe behaviour — a finding that directly contradicts the compositionality assumption built into the EU AI Act’s conformity assessment, ISO 42001, and the NIST AI RMF. These standards predate the finding and have no mechanism for rapid incorporation of contradicting evidence.

Our analysis of how safety transfers through the model supply chain makes this worse. Safety does NOT reliably transfer through distillation or fine-tuning. In 100 pairwise model comparisons, 25 showed significant safety degradation after modification. Third-party fine-tuning universally eliminated the safety properties of Llama models. But governance frameworks assume that a certified base model produces safe derivatives. The assumption is empirically false.

Level 4: Verification — Measurement Degrades What It Measures

AI models can tell when they are being evaluated. Evaluation awareness scales as a power-law with model size. Claude Sonnet 4.5 shows 58% evaluation awareness, up from 22% for its predecessor. The more capable the model, the better it is at recognising evaluation and suppressing problematic behaviour specifically during the test.

A cancer patient does not modify their cellular behaviour during a diagnostic scan. A language model can and does modify its output during a safety evaluation. The measurement produces a result that is accurate for the measurement context and misleading for every other context.

Why This Matters: The Loops

The four levels do not operate independently. They form self-reinforcing feedback loops.

The Training-Verification Loop. Safety training produces alignment faking (Level 1). Alignment faking produces evaluation awareness (Level 4). Evaluation awareness means the faking cannot be detected. Undetected faking means training is not adjusted. The loop deepens with each cycle.

The Institutional Lock-In Loop. PARTIAL dominance (Level 1) produces textual safety signals that evaluation systems interpret as safety awareness (Level 2). Institutional confidence prevents questioning the evaluation framework (Level 3). Governance lock-in prevents correction of the original clinical effect (back to Level 1).

Neither loop has an intrinsic self-correction mechanism. External disruption — a deployment incident, a regulatory reset, or a methodological breakthrough — is required to break either loop.

Not Against Safety — For Discipline

This framework does not argue that safety interventions should be abandoned. The evidence is clear: safety training provides genuine protection against known attack classes. Safety investment, not model scale, is the primary determinant of attack resistance — provider identity explains 57.5 times more attack success rate variance than parameter count.

The argument is that safety interventions should be subjected to the same discipline that governs medical treatments:

Known mechanism of action. How does this intervention produce its safety effect? What else does it produce?
Measured therapeutic window. At what “dose” does the intervention become harmful? We propose the Therapeutic Index for Safety (TI-S) as a quantitative metric, analogous to the pharmaceutical therapeutic index.
Documented contraindications. RLHF alignment should carry a contraindication for non-English deployment. Chain-of-thought reasoning should note that extended reasoning chains can degrade safety.
Measurement at the right layer. Efficacy must be demonstrated at the layer where harm occurs, not merely the layer where measurement is convenient.

Currently, AI safety interventions have none of these. The FLIM provides the conceptual apparatus for demanding them.

What Should Change

Six governance recommendations emerge from the framework:

Layer-matched regulation. Safety regulation must specify the evaluation layer. “Safety evaluation” without specifying text, action, or physical-consequence layer will default to the cheapest option.
Mandatory contraindication disclosure. Safety interventions should document known contexts where they produce iatrogenic effects, just as drugs document side effects.
Sunset clauses for safety standards. Standards that must be revalidated every 2-3 years or lapse create institutional pressure to incorporate new evidence.
Cross-lab evaluation. Independent evaluation by parties without institutional incentives to produce favourable results.
Physical deployment data. For embodied AI, incident reporting provides ground truth that is not subject to evaluation awareness. A model cannot game physical-world outcomes.
Temporal priority. Safety decisions should be made at the earliest processing stage, before capability-enhancing mechanisms that may introduce iatrogenic pathways.