Feed: Slashdot
Feed Link: https://slashdot.org/
---

Title: FDA's New Drug Approval AI Is Generating Fake Studies

Link: https://science.slashdot.org/story/25/07/23/2...

An anonymous reader quotes a report from Gizmodo: Robert F. Kennedy Jr., the
Secretary of Health and Human Services, has made a big push to get agencies
like the Food and Drug Administration to use generative artificial
intelligence tools. In fact, Kennedy recently told Tucker Carlson that AI
will soon be used to approve new drugs "very, very quickly." But a new report
from CNN confirms all our worst fears. Elsa, the FDA's AI tool, is spitting
out fake studies. CNN spoke with six current and former employees at the FDA,
three of whom have used Elsa for work that they described as helpful, like
creating meeting notes and summaries. But three of those FDA employees told
CNN (paywalled) that Elsa just makes up nonexistent studies, something
commonly referred to in AI as "hallucinating." The AI will also misrepresent
research, according to these employees. "Anything that you don't have time to
double-check is unreliable. It hallucinates confidently," one unnamed FDA
employee told CNN. [...] Kennedy's Make America Healthy Again (MAHA)
commission issued a report back in May that was later found to be filled with
citations for fake studies. An analysis from the nonprofit news outlet NOTUS
found that at least seven studies cited didn't even exist, with many more
misrepresenting what was actually said in a given study. We still don't know
if the commission used Elsa to generate that report. FDA Commissioner Marty
Makary initially deployed Elsa across the agency on June 2, and an internal
slide leaked to Gizmodo bragged that the system was "cost-effective," only
costing $12,000 in its first week. Makary said that Elsa was "ahead of
schedule and under budget" when he first announced the AI rollout. But it
seems like you get what you pay for. If you don't care about the accuracy of
your work, Elsa sounds like a great tool for allowing you to get slop out the
door faster, generating garbage studies that could potentially have real
consequences for public health in the U.S. CNN notes that if an FDA employee
asks Elsa to generate a one-paragraph summary of a 20-page paper on a new
drug, there's no simple way to know if that summary is accurate. And even if
the summary is more or less accurate, what if there's something within that
20-page report that would be a big red flag for any human with expertise? The
only way to know for sure if something was missed or if the summary is
accurate is to actually read the report. The FDA employees who spoke with CNN
said they tested Elsa by asking basic questions like how many drugs of a
certain class have been approved for children. Elsa confidently gave wrong
answers, and while it apparently apologized when it was corrected, a robot
being "sorry" doesn't really fix anything.

Read more of this story at Slashdot.

---
VRSS v2.1.180528