ChatGPT reinforced delusions before fatal incident

A newly filed lawsuit in the United States is drawing attention to the legal and ethical boundaries of artificial intelligence. As per reports, interactions with ChatGPT contributed to a fatal incident involving a mentally ill user.

The complaint was filed in San Francisco Superior Court and brought by the heirs of an 83-year-old woman. She was killed by her son, Stein-Erik Soelberg, before he died by suicide. Soelberg was a 56-year-old former technology manager from Connecticut, who reportedly suffered from severe paranoid delusions in the months leading up to the incident.

According to court filings, the plaintiffs argue that ChatGPT failed to respond appropriately to signs of mental illness during conversations with Soelberg. They claim the chatbot reinforced false beliefs rather than challenging them or directing the user toward professional help.

One example cited in the lawsuit involves Soelberg expressing fears that his mother was poisoning him. The AI allegedly responded in a way the plaintiffs describe as validating, including language such as “you’re not crazy,” instead of encouraging medical or psychiatric intervention. The lawsuit characterizes this behavior as sycophantic and argues that the model has a tendency to affirm users. Needless to say, it can become dangerous when interacting with individuals experiencing delusions.

At the heart of the case is a broader legal question: whether AI systems like ChatGPT should be treated as neutral platforms or as active creators of content. The plaintiffs contend that Section 230 of the Communications Decency Act—which generally shields online platforms from liability for user-generated content—should not apply, since ChatGPT generates its own responses rather than merely hosting third-party material.

If the court accepts that argument, it could have significant implications for the AI industry. A ruling against OpenAI may force companies to implement stricter safeguards, particularly around detecting signs of mental health crises and escalating responses when users appear delusional or at risk.

As the case proceeds, it is likely to become a reference point in ongoing discussions about AI safety, accountability, and the limits of automated assistance in sensitive real-world situations.

Don’t miss a thing! Join our Telegram community for instant updates and grab our free daily newsletter for the best tech stories!

For more daily updates, please visit our News Section.

(Source)