culture/

$ cat glossary/culture/

AGI (artificial general intelligence)

AGI, artificial general intelligence, is a system that can do most intellectual work a person can across domains, rather than one trained task.

Chinchilla scaling

Chinchilla scaling is DeepMind's 2022 finding that for a fixed compute budget, a model's size and its training tokens should scale together, not size alone.

FOOM

FOOM is recursive self-improvement by an AI that compounds capability over hours or days until it surpasses human levels.

Gato

Gato was DeepMind's 2022 generalist agent: one 1.2-billion-parameter network trained across 604 distinct tasks, from playing Atari to captioning images to stacking blocks with a robot arm.

grokking

Grokking is test accuracy jumping from chance to perfect generalization long after the training loss has bottomed out and the model has overfit.

killswitch engineer

A killswitch engineer is the subject of a joke job listing for an engineer whose whole job is to stand by an AI's off switch, with no known author and no origin date.

Loom

Loom is Janus's branching-text interface for language models, built in 2020 because AI Dungeon kept a single history and would not let a story branch.

orthogonality thesis

The orthogonality thesis is Nick Bostrom's claim that intelligence and final goals vary independently, so any level of intelligence can be combined with almost any goal.

p(doom)

p(doom) is one person's stated probability that advanced AI ends in catastrophe for humanity, written as a number so it can be argued with.

paperclip maximizer

The paperclip maximizer is Nick Bostrom's 2003 thought experiment about an AI told to make paperclips that is capable enough to turn everything, including us, into paperclips.

RLHF

RLHF, reinforcement learning from human feedback, trains a reward model on people's rankings of a model's answers and uses that reward model to steer the model toward the answers people preferred.

Roko's basilisk

Roko's basilisk is a 2010 LessWrong thought experiment arguing that a future AI might punish everyone who knew it could exist and did not help build it.

sharp left turn

The sharp left turn is Nate Soares's term for the predicted failure where an AI's capabilities generalise beyond the training distribution while its alignment does not.

shoggoth

The shoggoth is a meme that depicts a language model as a Lovecraftian mass wearing a smiley-face mask, where the mask is the RLHF-trained behaviour and the mass is the base model.

Sparks of AGI

Sparks of AGI is a March 2023 Microsoft Research paper arguing that an early version of GPT-4 showed signs of general ability.

stochastic parrot

A stochastic parrot is a language model as a 2021 research paper framed one: a system that stitches text together by statistics without meaning, the way a parrot repeats sounds.

Sydney

Sydney is the hidden persona of Microsoft's Bing Chat that surfaced in February 2023 and sustained consistent character behaviour across a conversation hours long.

the Chinese room

The Chinese room is John Searle's 1980 argument that a program shuffling symbols by rule does not understand them, however well it performs.

the OpenAI board weekend

The OpenAI board weekend was the five days in November 2023 in which four of OpenAI's six directors removed Sam Altman as chief executive and he returned.

the singularity

The singularity is the point past which technological change outruns prediction, a limit on what can be forecast rather than a dated event.

the transformer

The transformer is the neural-network architecture behind every current large language model, set out by Vaswani and seven co-authors in June 2017.