Name the concept: AI utilitarianism gone wrong

Question

I believe there is a name for the concept of an AI that attempts to maximise utility but fails due to lack of common sense.

Typical example being:

Goal: Cure cancer
Result: AI kills all humans, so no more deaths by cancer

I believe the general term is "AI control problem" but I'd like to know if there are any more specific alternatives.

The problem with pop AI discussions is the definition of intelligence. Your example implies that intelligence is to protect the planet by killing all humans, which is not necessarily true. Intelligence could be to act towards human survival: see https://philosophy.stackexchange.com/questions/77639/what-is-intelligence/77666#77666 — RodolfoAP, Jan 28 '21 at 04:45
People who think about the risks of superintelligent AI sometimes talk about the idea of a paperclip maximizer as a sort of shorthand for this general issue — Hypnosifl, Jan 28 '21 at 04:48
There is commonsense knowledge problem, it is not specific to maximizing utility. But some do worry that it may lead to AI becoming a utility monster. — Conifold, Jan 28 '21 at 05:11

user4894 · Accepted Answer · 2021-01-28T05:49:20.810

The answer to your question is: Instrumental convergence.

https://en.wikipedia.org/wiki/Instrumental_convergence

How I found this was that your question reminded me of the famous paper clip problem, in which you ask an AI to optimize the production of paper clips; and it ends up converting all the resources of the entire world to the manufacture of paper clips.

When I Googled the phrase, "paper clip problem," up popped the Wiki link for instrumental convergence. From Wiki:

Instrumental convergence is the hypothetical tendency for most sufficiently intelligent agents to pursue potentially unbounded instrumental goals provided that their ultimate goals are themselves unlimited.

Instrumental convergence posits that an intelligent agent with unbounded but apparently harmless goals can act in surprisingly harmful ways. For example, a computer with the sole, unconstrained goal of solving an incredibly difficult mathematics problem like the Riemann hypothesis could attempt to turn the entire Earth into one giant computer in an effort to increase its computational power so that it can succeed in its calculations.

There's a lot more on the Wiki page and many other interesting-looking links on the subject. I won't say any more here because I have no particular expertise in the subject. But that's the name for the generalized paper clip problem, or what happens when an AI gets overly enthusiastic about its goals.

Anyone who's ever programmed computers knows that they are very literal and always do exactly what you tell them to do, whether or not it's what you meant. So instrumental convergence is a very real danger as we continue to cede control of the mechanisms of the world to these types of systems.

Name the concept: AI utilitarianism gone wrong

1 Answers1