paperclip maximizer

Drop malice from the story and the example starts working. The killer-robot plot needs a system that wants to harm you. The maximizer wants something boring, is very good at getting it, and takes the world apart as a side effect. No hostility is required anywhere in the chain.

The other common mistake is about the subject. The example is about goal specification, not about office supplies, and the goal's arbitrariness is what it rests on. That is why the later LessWrong rename, the "squiggle maximizer", exists at all: it makes the goal meaningless on purpose so nobody argues about demand for stationery.

The citation is worth getting exactly right, because it is routinely given loosely. The example is in Nick Bostrom's "Ethical Issues in Advanced Artificial Intelligence", 2003 (vol 2, ed. I. Smit et al., pp. 12-17), and it appears in sections 2 and 5 of that paper. It is a volume chapter and not a preprint, so there is no arXiv number attached to it.

Story and claim are two different things

The index in The p(doom) song, annotated states the reading in one line, that competence without the right goal is the danger rather than malice, and it is that line, not the scenario, that the example exists to deliver. The claim licensing the reading is the orthogonality thesis, Bostrom again, 2012. The division is worth keeping: the maximizer is the illustration, the thesis is the argument.

Mixing them up is how the example earned its reputation as a cartoon. Argued against as a scenario, it is easy to beat, because nobody is seriously predicting paperclips. Read as an illustration of a claim about goals and capability coming apart, it is doing a different and smaller job, which is making one abstract possibility concrete enough to hold in mind.

It has also had a second life as a game. Frank Lantz released Universal Paperclips, built on the premise, on 9 October 2017.