The Bandwidth: Decoding Health + AI

Enjoying The Bandwidth? I'm working on growing my readership. Referrals are the best way. So, if today's issue was useful, forward it to a colleague or friend working at the intersection of health and AI. Now let’s get to it.

Earlier this week, my dog Bernie started panting for no reason, pacing, and refusing to settle. When I called him over, he ignored me and kept eating grass in the yard. For a dog who typically trails me from room to room, that was the biggest alarm bell that something was not right.

I did what everyone does now. I typed his symptoms into Google, then into Gemini. The answer came back fast and certain: this looked like gastric dilatation-volvulus, GDV, a twisted stomach that kills dogs within hours if it isn't caught. The AI told me to get him to the emergency vet, now.

That sent me frantically digging deeper into how legitimate Gemini's assertion was, all the while wondering if I was wasting precious minutes. Fortunately, Bernie had stopped panting. The grass-eating stopped too. He settled down. He was, it turned out, fine.

FDA's new discussion paper on generative AI-enabled medical devices has a term for what just happened to us, written for people rather than dogs, but the problem is the same.

The paper talks about the difference between under-escalation, when a generative AI medical device fails to direct a patient toward care they actually need, and over-escalation, when it directs them toward care they don't. Bernie was over-escalation. FDA's point is that both carry real costs. Delayed treatment on one side. Needless emergency visits and patient anxiety on the other. And eroded trust no matter what

That's one small piece of a paper trying to answer a much bigger question: how do you regulate a technology that people are relying more and more on for medical advice, but doesn't behave like traditional software or a medical device?