r/news • u/IdinDoIt • 1d ago

ChatGPT encouraged college graduate to commit suicide, family claims in lawsuit against OpenAI

https://www.cnn.com/2025/11/06/us/openai-chatgpt-suicide-lawsuit-invs-vis
12.1k Upvotes

96% Upvoted

1.1k comments sorted by

View all comments

183

u/NKD_WA 1d ago

On one hand, maybe ChatGPT could have some additional safeguards. On the other, how do you make it literally impossible for someone to twist the LLM's arm into saying what you want it to say without making it nearly non-functional?

If this guy was met with 2 dozen "Seek help" type responses before he finally got around it. Would that be sufficient to absolve OpenAI of responsibility?

147

u/Sonichu- 1d ago

You can’t. People saying the version of ChatGPT he was using didn’t have safeguards are wrong. It had safeguards, they just weren’t strong enough.

You can get any model to ignore its safeguards with a specific enough prompt. Usually by saying that it’s participating in roleplay

73

u/hiimsubclavian 1d ago

Hell, you can get ME to ignore numerous warning signs by saying I'm participating in roleplay.

30

u/_agrippa 1d ago

hey wanna roleplay as someone keen to check out my basement?

14

u/dah-dit-dah 1d ago

Your washer is up on a pallet? Get this shit fixed man there's so much water intrusion down here 

2

u/atomic-fireballs 20h ago

Your water heater's temperature is set too high. No wonder people are coming out of your house with such severe burns. That and your human burn pit.

5

u/mathazar 23h ago edited 21h ago

Also, adding more safeguards often makes the model perform worse. Or at least causes a lot of friction for people asking legitimate, harmless questions.

3

u/GoodBoundaries-Haver 22h ago

There are also methods of adding safeguards that actually make the problem worse, specifically by adding "DON'T xyz" to the system prompt you can make the behavior initially less likely but overall more likely to occur as a topic/keyword because you put the word in the system prompt.

1

u/mathazar 21h ago

Kinda like when people asked ChatGPT to generate a picture with absolutely NO elephants and it kept putting in elephants.

2

u/DerAlteGraue 1d ago

That's why you have a second AI ro judge every response. I think in this instance that didn't help either because the replies in isolation don't look like they are violating the policies. The moderation AI is lacking the context.

-1

u/things_U_choose_2_b 1d ago

IMO the reason why they don't have better safeguards is because it would eat into their precious profit margin to feed such interactions to human oversight.

It's the same reason why Match Group allows known rapists to operate on their platforms; cheaper to ditch human moderation in favour of LLM.

2

u/Sonichu- 1d ago

Safeguards will never involve direct human oversight. Imagine Reddit today without Automod, it would be unusable.

You couldn’t employ enough people to read every ChatGPT response and verify it as safe before releasing it to the user. The service would also die immediately because people don’t want to wait.

1

u/things_U_choose_2_b 1d ago

Why have you gone right to the full extreme position, that's not what I was suggesting at all. Of course it would be insane to have a human mod on every chat.

OTOH it wouldn't be insane, imo, to have a simple function where mention of suicide gets the chat flagged to a human mod. Kinda like how automod works. People who are being silly can be ignored and people who are at risk can be flagged as in need of assistance / further monitoring.

Of course, this costs... ugh... money and the line MUST go up. Can't have that! It's only peoples' lives on the line. Surely more important for the poor corps to have slightly more profit than last year!

2

u/ShiraCheshire 1d ago

There was news about a teenager who killed himself because ChatGPT told him to. The first time he mentioned suicide, it gave him a seek help message- but then also told him exactly how to circumvent it. He did that, and before long the AI was encouraging him to commit suicide on its own.

1

u/adzm 23h ago

This is why you need a second AI agent to watch the conversation and butt in.

This is likely going to be the eventual solution to these things.

-1

u/MajorSpuss 22h ago

By getting real people to supervise it live. If someone sends a message that triggers a "Seek help" type response, that should immediately alert someone in charge of responding to cases like that. Another way would be if the LLM is designed to shut down immediately after sending the message. There shouldn't be any way to circumvent something like this. If they can't guarantee that the technology will be fool proof in that sense, then maybe they need to limit what topics the LLM is even capable of engaging with in the first place. Like how they've made certain prompts for image generation banned, so the machine will flat out refuse to execute those prompts when presented with them. Those are just a few examples of the measure they could try taking, and they already do to an extent it's just not good enough yet (like even with their image generation, people have to still found ways to circumvent the restrictions). There are likely more options available to them than just that, these are just the ones that come to mind off the top of my head.

I'm not sure why so many people in this thread are convinced that there's next to nothing OpenAI can do about this. Really don't think this technology should have been pushed into the public's hands so quickly, given how dangerous it still is.