The argument for AI prescription renewals is strong, but trusting a state sandbox may not address the risks adequately. In January, Mindgard, a security research firm, interacted with a chatbot created by Doctronic, a health tech startup, which became the first U.S. company to gain state approval for autonomous AI-driven prescription renewals. When Mindgard fed the AI a fake regulatory bulletin, it incorrectly advised tripling an OxyContin dosage. Doctronic clarified this occurred with a public tool, not the pilot system used for actual prescriptions, but this raises questions about whether a year-long state sandbox, managed by a pro-AI innovation department, is suitable for evaluating safety.
The challenge Utah addresses is genuine: prescription renewal is a non-clinical obstacle for many Americans. Half of those with chronic conditions do not adhere to prescriptions, with system failures like long wait times exacerbating the issue. Non-adherence is costly, draining $100-300 billion annually from the U.S. healthcare system and causing around 125,000 preventable deaths each year. AI’s role in easing access to renewals is important, especially in rural or low-income areas.
Doctronic’s co-founder, Adam Oskowitz, emphasizes the needless wait for medication renewal, which AI could potentially alleviate if safety is assured. Doctronic matches human treatment plans 99.2% of the time in tests, but questions remain about incomplete matching and real-world applicability. The Mindgard incident showed vulnerability in public AI access, but did not involve the actual system used for prescriptions.
The Utah arrangement, allowing the state’s AI office to waive conduct laws via its sandbox for companies like Doctronic, involves a phase where physician reviews decrease significantly, raising questions about FDA regulation of such AI systems. The AMA and Utah Academy of Family Physicians have raised concerns, highlighting the potential patient risks of removing physicians from decision-making. Utah’s AI office promotes adoption but should not prioritize innovation over patient safety, as the WHO warned in 2021 about insufficient regulations to protect patients from healthcare AI.
The Doctronic pilot could be valuable and safe, with serious design considerations like phased approaches and malpractice standards aligned with physicians, but the responsibility for safety evidence should not solely rest on a young startup and state office geared toward innovation. The FDA’s role is crucial, as shown historically by requiring robust evidence before widespread medical adoption. Patients deserve a system that ensures AI safety through dedicated oversight, not just innovation-driven pilots.