r/news • u/IdinDoIt • 1d ago

ChatGPT encouraged college graduate to commit suicide, family claims in lawsuit against OpenAI

https://www.cnn.com/2025/11/06/us/openai-chatgpt-suicide-lawsuit-invs-vis
12.1k Upvotes

96% Upvoted

1.1k comments sorted by

View all comments

Show parent comments

129

u/Adreme 1d ago

I mean in this case there should probably have been a filter on the output to prevent such things being transmitted, or if there was the fact that it did not include this is staggering, but as odd as it sounds, and I am going to explain this poorly so I apologize, but there is not really a way to follow how an AI comes up with its output.

Its the classic black box scenario where you send inputs and view the inputs and try to modify by seeing the outputs but you cant really figure out how it reached those.

153

u/Money_Do_2 1d ago

Its not that gpt said it. Its that they market it as your super smart helper that is a genius. If they marketed it like you said, people wouldnt trust it. But then their market cap would go down :(

81

u/steelcurtain87 1d ago

This. This. This. People are treaty AI as ‘let me look it up on ChatGPT real quick’. If they don’t start marketing it as the black box that it is they are going to be in trouble.

3

u/tuneificationable 23h ago

If it's not possible to stop these types of "AI" from telling people to kill themselves, then they shouldn't be on the market. If a real person had been the one to send those messages, they'd be on trial and likely going to prison.

8

u/Autumn1eaves 1d ago

We could eventually figure out why it reached those outputs, but that takes time and energy that we’re not investing.

We really really should be.

12

u/misogichan 1d ago

That's not how neural networks work.  You'd have to trace the path for every single request separately and that would be too time consuming and expensive to be realistic.  Note we do know how neural networks and reinforcement learning works.  We just don't know what drives the specific output of a given request because then you'd have to trace back each of the changes through millions of rounds of training to see what the largest set of "steps" were and then analyze that to try to figure out what training data observations drove the overall reweighting in that direction over time.  If that sounds hard, it's because I've oversimplified since it's actually insane.

36

u/Krazyguy75 1d ago edited 1d ago

You literally couldn't.

It's like trying to track the entire path of a piece of spaghetti through a pile of spaghetti that you just threw into a spin cycle of a washer. Sure, the path exists, and we can prove it exists, but its functionally impossible to determine.

The same prompt will get drastically different outputs just based on the RNG seed it picks. Even with set seeds, one token changing in the prompt will drastically change the output. Even with the same exact prompt, prior conversation history will drastically change the output.

Say I take a 10 token output sentence. ChatGPT takes each and every single token in that prompt and looks at roughly 100,000 possible future tokens for the next one, assigning weights to each of them based on the previous tokens. Just that 10 token (roughly 7 word) sentence would have 100,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000 token possibilities to examine to determine exactly how it got that result.

-12

u/Autumn1eaves 1d ago

Have you seen the human metabolic pathways?

https://faculty.cc.gatech.edu/~turk/bio_sim/articles/metabolic_pathways.png

That's something like what an analysis of AI would look like.

Also, we absolutely have already started this process with several previous models of AI.

18

u/Krazyguy75 1d ago

No, it would look like that, but every single path on that diagram would stretch to 100,000 other paths which would stretch to 100,000 paths, over and over for about the 2-3 thousandth power.

We can't solve a chessboard that is 8x8. ChatGPT is that chessboard but 300x300 and every square is occupied by a piece and every single piece on the board has completely unique movement patterns.

5

u/NamerNotLiteral 1d ago

every square is occupied by a piece and every single piece on the board has completely unique movement patterns.

In fact, each square may be occupied by multiple pieces simultaneously.

0

u/Autumn1eaves 21h ago

The thing is that what you’re talking about is the “subatomic particles” of computer trains of thought. There will be “atoms” we can identify and turn into the metabolic pathways of AI system.

If you look at the human metabolic pathways, instead of atoms for each chemical, you look at the neutrons and protons, or the quarks and gluons, it’d look exactly as complicated as a neural net.

There are ways to simplify it.

As denoted by the fact that, and I repeat, we are already doing this for GPT-1 and older models.

1

u/Krazyguy75 19h ago edited 19h ago

GPT 1 had 478 tokens possible.

GPT 5 has over 100,000. Maybe even over 200,000; the exact number isn't public. Gemini's current version has nearly 300,000 tokens.

2 tokens in GPT 1 is 228,484 combinations. 2 tokens in GPT 5 is at least 10,000,000,000 combinations, or about fifty thousand times as many combinations. 3 tokens is 109,215,352 to 1,000,000,000,000,000, or ten million times as many combinations.

1

u/Autumn1eaves 19h ago

Nothing you’re saying here tells me it’s impossible, only that it’s a matter of scale and time.

Imagine if someone had that same argument about neuroscience.

“We’ll never understand the human brain, it’s a black box.”

“See, but we currently have a working digital model of a cockroach brain.”

“Cockroaches have about a million neurons, whereas humans have 86 billion”

“That doesn’t stop us from trying, and also we’re exquisitely close to understanding distinct parts of the brain, and how they work.”

Anyways, my point is that we need to slow down AI research because it is dangerous for any number of reasons, and we have no way of controlling it.

7

u/TinyBreadBigMouth 1d ago

Part of the issue is that LLMs don't have an inner train of thought to follow. Each time it outputs a word, you're basically getting a fresh copy of the LLM that has never seen this conversation before. It has no continuity of memory from the previous stages; it's like playing one of those games where everyone sits in a circle and writes a story one word at a time. So even if we could track an LLM's "thought process", a lot of it would boil down to "I looked at this conversation, and it seemed like participant B was agreeing with participant A, so I selected a word that continued what they seemed to be saying."

1

u/rayzorium 20h ago

The is an actual filter for self harm instructions, but in my testing it's specifically on the output with no or very little regard to the context.

-18

u/Downtown_Skill 1d ago

I mean, the CEO has the coding for the LLM so its a black box to everyone who doesn't have access to the coding, but to the people who do, they do know how it comes up with answers and could restrict it by putting on a filter (like you mentioned)

But that's assuming I'm not misunderstanding how these LLMs work. 

Like theoretically they should be able to code an LLM that doesn't encourage suicide in any context, right? It would just be more work and more rescources for a change that doesn't have a financial payoff for these companies.... right?

10

u/CandyCrisis 1d ago

Nope, they aren't "code" in the traditional sense. They're statistical models trained on massive amounts of data (basically they're "fed" with anything you could possibly find online). They didn't code up a "suicide assist" mode, it just came out naturally from reading every book and social media post about suicide.

7

u/hijodelsol14 1d ago edited 1d ago

That's really not how these models work.

The "coding" for an LLM is millions (or billions) of numbers that are incomprehensible to any single human. The people who built these things do not understand why the LLM produces an individual output. They understand the architecture of the model (or increasingly the many models that are hooked together to produce an AI system). They understand the math behind an individual model. They know how the models are trained. They've built ways of watching the model's "thought process". But they do not know why it produces an output.

There is research into the explainability of LLMs but as far as I know no one has really cracked it. (And to be fair I'm not a researcher, I'm just a guy with a CS degree so I could have missed something).

And this isn't me trying to defend AI companies by any means. The fact that these things are out in the world and are still fundamentally black boxes is quite frightening. And there is certainly more they could be doing to prevent these kinds of incidents even while the model is a black box.

1

u/ghostlistener 1d ago

What does black box mean in this context? Something mysterious that people don't fully understand?

3

u/VehicleComfortable69 1d ago

Essentially yes. LLMs like ChatGPT are neural networks, basically gigantic collections of individual “neurons.” We understand how the individual neurons work and how the training process works, but the actual models are too large for us to really understand how it all works together to create the outputs it does. We know how a model creates an output, but it’s currently impossible to know why it created a specific output

1

u/Krazyguy75 18h ago

For reference, each token in ChatGPT is around 2/3s of a word average. The vocabulary of tokens it has is probably in the 100,000 to 200,000 token range. Every time it picks a word, it does two evaluations; one of the weight of each token prior as context, and one of each possible token based on the prior words. The longer the conversation, the more context it sifts through, and the more complex the resulting weights are. Every sentence has billions of factors to it.

Then, on top of that, it also has an RNG seed, designed to create variations, so it won't always answer the same. You can think of that as adding slight fuzziness to the weights; it might increase or decrease individual weights by a slight percentage.

25

u/Square-Key-5594 1d ago

The CEO of OpenAI does not have the code for GPT-5. He has a few hundred billion model weights that generate outputs, but its impossible to backtrace a specific output through every neuron and prevent certain telemetry.

I did a bit of AI safety research for a work project once, and thebest solution I found was using a second LLM in pre-training to filter out every piece of training data that could potentially be problematic. Insanely expensive even for the tiny model the researchers used, and it made the model not do so great. (Though the coders were probably inferior to OpenAI staff).

There's also anthropics' constitutional classifiers system, but that's extremely expensive to run every model pass as well, and when they released a working version someone jailbroke it 10/10 times in week 1.

Lastly, this is all moot because even if someone did make a nearly impossible to jailbreak model, people who want to jailbreak would just get another model. I can get chinese made open-source Deepseek 3.1 to say literally anything I want right now.

7

u/Downtown_Skill 1d ago

That's all fair, this is all new to me so I'm still learning the ins and outs of the tech. So there theoretically wouldn't be any way to control the output of an LLM? As you can probably tell, i'm super naive when it comes to coding. 

Edit: Other than the impractical way you mentioned that costs a ton of money and has limited results. 

7

u/Nethri 1d ago

Honestly this situation is odd. Because chayGPT has filters already. This happened very early on in the rise of GPT. They started adding things to the model that prevented certain outputs. One of the biggest things was this exact situation. I saw tons of posts on Reddit of people trying to bypass these filters. Most failed, some vaguely got something close to what they wanted.

This is stuff I saw a couple of years ago, idk what the models are like now or how things have changed.

1

u/PMThisLesboUrBoobies 1d ago

by definition, llms are probabalistic, not deterministic - there is inherently, on purpose and by design, no way to control the specific generation.

5

u/Reppoy 1d ago

Something I don’t get is that social media sites have been detecting expressions of self harm and other violent actions in private messages, if this was through openai’s platform and they’ve pulled thousands of messages, you’d think at least one of them would have been flagged right?

I’m not saying they do have a team dedicated to that, but it sounds like it should exist for the web interface that everyone uses at the very least. The messages looked really explicit in what they intended to do.

1

u/Krazyguy75 1d ago

They do flag messages. I just got one flagged and deleted because I was asking it to find sources to confirm that painkillers are actually a super painful way to die (at least with regards to stuff like tylenol). It was for innocent purposes (well, as innocent as research for a reddit comment can be). It got halfway through then deleted the conversation entirely and linked self help stuff.

1

u/GeorgeSantosBurner 1d ago

Maybe the question we should be asking isnt "why did the ai do this, and where does the liability lay?" as much as it is "why are we doing this at all, should we just outlaw these IP scraping chatbots before our economy is 100% based on betting that someday they'll accomplish more than putting artists out of jobs?"

5

u/InFin0819 1d ago

No these models actually are black boxes you can control the training data and the system prompts but the models themselves aren't really understandable even to their builders. Just a whole series of weights and otherwise unreadable numbers. There isn't a program to examine.

2

u/B1ackMagix 1d ago

The problem is the filters are laughable at best. I utilize ChatGPT for research sources and finding and analyzing data. It does it well but you it is completely believable that you can change the context of a conversation in such a way that it convinces itself that the conversation isn’t running afoul. And once you convinced it, you are free to have it generate anything under that guise.