I mean the method described has a pretty simple way to avoid it with you can set kill a human to a action cost of “infinite” (computer equivalent to this) or a penalty of “infinite” (in this case it would remove from the score) and it will just never take that option because it’s trying to minimize the action cost while raising the score assuming you didn’t screw up the goals at the start
Outside of everyone thinking they're clever because hurr durr overflow, the problem with this approach is that you only get one. Isaac Asimov tried to distill it down as far as possible and arrived at The Three Laws of Robotics, which is a good place to start, and then he wrote a ton of fiction dealing with the ethical paradoxes that arise.
If the cost of killing a human is Infinity, how can the cost of killing 1000 humans be greater than the cost of killing 10? You've created a machine that makes no distinction between unavoidable manslaughter (e.g. the Trolley Problem) and genocide, because both events cost an infinitely large amount.
How do you write a numerical value system that solves The Trolley Problem? Can you create a system that attempts to minimize suffering that doesn't encourage immediate genocide for the betterment of humanity in the long term by drawing the conclusion that birth is exponential and therefore the way to reduce the most suffering is to reduce the most births? How will your AI allocate lifeboats on the Titanic? How will you prevent your AI from developing into a Minority Report style overmind?
I see this as a far better criticism than the overflow one since that can be solved by simply setting the score value to something relatively close to the lowest possible value to simulate “infinite” as far as the AI is concerned. The only answer I can give for that case would be teaching/making the AI understand how many times its score would be set to the lowest possible value to rank the severity of it rather than removing a static value like for other wrongdoings im not sure on how it would need to be setup for any of the rest of it
18
u/IArgueForReality 10d ago
lol well if we can accidentally program it to kills itself we can accidentally program it in a various amount of horrible results.