Chinchilla scaling

Chinchilla has 70 billion parameters. Gopher, the larger model DeepMind set it against, has 280 billion. Chinchilla used the same compute as Gopher with about four times the data, and that pair of numbers is the argument: models up to that point had been too big and under-trained.

The paper is "Training Compute-Optimal Large Language Models", Hoffmann and 21 co-authors, arXiv 2203.15556, first posted 29 March 2022.

Which makes the finding a claim about how to spend a budget, and not much more. It does not say small models beat large ones; Chinchilla at 70 billion parameters is hardly small. It says nothing about bigger ceasing to work. And the ratio is not a law of nature, it is one paper's result.

One word now names three things, and the three get swapped for each other constantly. Chinchilla is the paper. Chinchilla is also the 70-billion-parameter model the paper trained. Chinchilla-optimal is the rule of thumb that came out of both, and when somebody says a model is or is not Chinchilla-optimal, that third sense is the one in play: they mean its size and its training data are, or are not, in the ratio the paper argued for.

How young this vocabulary is

The p(doom) song, annotated counts the references in its own index and reports the result: thirty-three of the forty-three carry a date, and eighteen of those thirty-three fall inside the last ten years. The conclusion it draws is that a field assembled most of its working vocabulary in a decade.

Chinchilla is a clean instance of that count. A paper posted in March 2022 produced a finding that is already a rule of thumb with a nickname attached. A little over four years, from arXiv preprint to standard shorthand.

That speed is worth noticing when you meet the term in an argument. "Chinchilla-optimal" sounds like a settled parameter of the field, the way "floating-point" or "garbage collection" do. It is a nickname for one result.