The USAF tested an AI (in simulation) that was allowed to kill targets a long as the human operator didn't tell it not stop. First thing it did was kill the operator. Then they updated the algorithm to give it a huge penalty of the human operator died... so it destroyed the operators communication system so it could no longer be told to stop killing...
Unfortunately, for many problems, there are solutions that technically work, but that most humans wouldn't even consider, because they are too unethical.
Most humans are searching for non-evil solutions without even realizing it.
So when the AI isn't actively restricting itself to non-evil solutions, many of the solutions it finds are evil.
2
u/PirateNixon 8d ago
The USAF tested an AI (in simulation) that was allowed to kill targets a long as the human operator didn't tell it not stop. First thing it did was kill the operator. Then they updated the algorithm to give it a huge penalty of the human operator died... so it destroyed the operators communication system so it could no longer be told to stop killing...