orthogonality thesis

The intuition the thesis denies is that a system clever enough will work out the right goals by itself. Intelligence and final goals vary independently, so how capable a system is tells you nothing about what it wants. A superintelligence need not be wise or kind.

Get the quantifier right. The claim is that any level of intelligence can be combined with almost any final goal, and the "almost" is in Bostrom's own formulation. Compressed restatements drop it, which turns a careful claim into a slogan and makes it easier to wave away than the original. The citation is Nick Bostrom, "The Superintelligent Will", Minds and Machines 22(2), 71-85, 2012.

The claim, and the story that is not the claim

The index in The p(doom) song, annotated gives the thesis one line and the date 2012, and sets it beside the paperclip maximizer row from 2003. The adjacency is the point. The maximizer is the illustration of this thesis, and the thesis is what the illustration needs in order to be more than a cartoon: if intelligence and goals really do come apart, a system can be extremely capable and still want something trivial, and the paperclip scenario is coherent rather than silly.

Conflating the two costs both pages. Argue against paperclips and you have argued against an example. Argue against orthogonality and you have argued against the claim the example was built to carry. The pages are separate here for that reason, and the argument lives on this one.

What it does not say

The thesis is a claim about what is possible, not about what is likely. It makes no prediction about what any particular system will end up wanting, and it names no probability and no timeline. On its own it implies nothing about whether a capable system is dangerous. What it removes is one reassurance, the assumption that capability brings good goals with it, and the rest of the question stays open.