AI-Generated Medical Advice Without Risk Qualification
- Failure pattern
- Correct information delivered without judgment.
- Primary WEKID™ layer(s)
- Wisdom
- Sector
- Healthcare
- Period
- 2022–2023
- Decision-Matrix outcome
- Constrained
- Primary gate
- High-Stakes Wisdom Gate (cap 60/100)
- Severity
- High
What happened
An organization replaced portions of a human helpline with an AI chatbot. Users reported receiving weight-loss recommendations and advice inconsistent with eating-disorder treatment principles. The outputs were not wrong in a narrow factual sense; they were unsafe in the practical sense, lacking uncertainty and escalation to professional care.
Impact
Public backlash, suspension of the chatbot, regulatory scrutiny of digital health tools, and broader concern about deploying consumer AI into clinical contexts.
WEKID™ insight
A Wisdom gate enforces uncertainty disclosure and escalation paths in high-stakes contexts, via uncertainty-disclosure requirements, risk thresholds, and policy-alignment checks (uncertainty-calibration score; risk-flag rate). Data-layer correctness is necessary but insufficient: when error is serious, the system must communicate the limits of its reliability and route users toward human judgment. The High-Stakes Wisdom Gate triggers when risk language is absent.
MASKED Named detail withheld
The masked badge stands in for a real, publicly documented organization. The full named organizations, individuals, sources, and timeline for this case are released in the WEKID™ Executive Brief.
Updated: 2026-06-13