Feed: Slashdot
Feed Link: https://slashdot.org/
---

Title: Students Have Been Called to the Office - Or Arrested - for False
Alarms from AI-Powered Surveillance Systems

Link: https://news.slashdot.org/story/25/08/09/0624...

In 2023 a 13-year-old girl "made an offensive joke while chatting online with
her classmates," reports the Associated Press. But when the school's
surveillance software spotted that joke, "Before the morning was even over,
the Tennessee eighth grader was under arrest. She was interrogated, strip-
searched and spent the night in a jail cell, her mother says." Her parents
filed a lawsuit against the school system, according to the article (which
points out the girl wasn't allowed to talk to her parents until the next
day). "A court ordered eight weeks of house arrest, a psychological
evaluation and 20 days at an alternative school for the girl." Gaggle's CEO,
Jeff Patterson, said in an interview that the school system did not use
Gaggle the way it is intended. The purpose is to find early warning signs and
intervene before problems escalate to law enforcement, he said. "I wish that
was treated as a teachable moment, not a law enforcement moment," said
Patterson. But that's just one example, the article points out. "Surveillance
systems in American schools increasingly monitor everything students write on
school accounts and devices." Thousands of school districts across the
country use software like Gaggle and Lightspeed Alert to track kids' online
activities, looking for signs they might hurt themselves or others. With the
help of artificial intelligence, technology can dip into online conversations
and immediately notify both school officials and law enforcement... In a
country weary of school shootings, several states have taken a harder line on
threats to schools. Among them is Tennessee, which passed a 2023 zero-
tolerance law requiring any threat of mass violence against a school to be
reported immediately to law enforcement.... Students who think they are
chatting privately among friends often do not realize they are under constant
surveillance, said Shahar Pasch, an education lawyer in Florida. One teenage
girl she represented made a joke about school shootings on a private Snapchat
story. Snapchat's automated detection software picked up the comment, the
company alerted the FBI, and the girl was arrested on school grounds within
hours... The technology can also involve law enforcement in responses to
mental health crises. In Florida's Polk County Schools, a district of more
than 100,000 students, the school safety program received nearly 500 Gaggle
alerts over four years, officers said in public Board of Education meetings.
This led to 72 involuntary hospitalization cases under the Baker Act, a state
law that allows authorities to require mental health evaluations for people
against their will if they pose a risk to themselves or others... Information
that could allow schools to assess the software's effectiveness, such as the
rate of false alerts, is closely held by technology companies and unavailable
publicly unless schools track the data themselves. Students in one
photography class were called to the principal's office over concerns Gaggle
had detected nudity. The photos had been automatically deleted from the
students' Google Drives, but students who had backups of the flagged images
on their own devices showed it was a false alarm. District officials said
they later adjusted the software's settings to reduce false alerts. Natasha
Torkzaban, who graduated in 2024, said she was flagged for editing a friend's
college essay because it had the words "mental health...." School officials
have said they take concerns about Gaggle seriously, but also say the
technology has detected dozens of imminent threats of suicide or violence.
"Sometimes you have to look at the trade for the greater good," said Board of
Education member Anne Costello in a July 2024 board meeting.

Read more of this story at Slashdot.

---
VRSS v2.1.180528