RLHF

Pre-train a model on text. Put its answers in front of people and have them rank which ones they prefer. Train a second model, the reward model, on those rankings. Then use the reward model to steer the first one toward the answers that ranked well. That four-step sequence is RLHF.

The sequence is not a guarantee about behaviour. It moves a model toward ranked-preferred answers and says nothing about what the model does on an input nobody ranked. It is also not a content filter: no step inspects an output at the moment of use and blocks it. The work happens during training, and what comes out is weights. The mechanism that refuses a request at runtime is a different one.

osmarks, who wrote the annotation behind The p(doom) song, annotated, goes further and says the technique does not usefully constrain the model. That is his view, not a finding, not a consensus, and not this site's position.

Two dates doing two jobs

The method and the standard are different papers, which is why RLHF gets dated to two different years by people who are both right.

Christiano, Leike, Brown, Martic, Legg and Amodei posted the method paper as arXiv 1706.03741 on 12 June 2017. OpenAI's InstructGPT came later, arXiv 2203.02155, 4 March 2022, Ouyang and co-authors, and it names RLHF in its abstract. InstructGPT is what made the technique the standard that produced ChatGPT. So 2017 is where the procedure comes from and 2022 is where it became the default way a chat assistant gets built.

What it produces

The annotated essay's index calls the shoggoth the RLHF meme. In the picture, the friendly assistant is a smiley mask and RLHF is what produces the mask; the mass behind it is the base model, whose nature nobody can see. @TetraspaceWest posted it on 30 December 2022, the year InstructGPT came out.

RLHF is a procedure that shapes what a model says. What it does to the thing underneath is what the meme asks about, and the procedure does not settle it.