Why Small Language Models Still Matter

There is a recurring argument in AI research that goes something like this: small models are interesting stepping stones, but the real action is at scale. Once you have a sufficiently large model trained on sufficiently large data, the small-model questions become moot — capability emerges, and that is that.

I think this argument is wrong, or at least importantly incomplete. Here is why.

The scientific question is distinct from the engineering question

When people argue that scale is all that matters, they are usually making an engineering argument: for downstream task performance, larger models win. That is largely true, and I have no quarrel with it. But the scientific question — what computational properties of a learning system produce grammatical competence from linguistic input? — is not the same question. And for that question, scale actively obscures the answer.

If you want to understand what inductive biases are necessary for language acquisition, you cannot run the experiment on a 70-billion-parameter model trained on a trillion tokens. You cannot isolate the contribution of architecture from the contribution of sheer data volume. The model is too large, the training signal too rich, for controlled hypothesis testing.

Small language models — BabyLMs, in the workshop’s terminology — are valuable precisely because they are tractable. The PicoLM framework we released in March 2025 is built on this conviction: tight parameter budgets make each architectural decision consequential, and that consequentiality is a feature, not a bug. It means experiments are informative.

The multilingual dimension matters more than people think

Most BabyLM work has been monolingual. Most large-scale work is also, functionally, monolingual — English-dominant even when nominally multilingual. But the human language learner is not monolingual. Code-switching, bilingual acquisition, heritage languages, immigrant language contact: these are the norm, not the exception, for most of the world’s speakers.

Our work on bilingual and multilingual small language models (BLiSS, BabyLM 2025) suggests that multilingual exposure does not simply add capacity — it shifts inductive biases. A model that has seen two languages acquires structural representations that are genuinely different from a model that has seen one language with twice as much data. The implications for how we think about language universals, typological variation, and the cognitive basis of multilingual competence are significant.

The BabyLM 2026 Workshop, which I am co-organising, introduces a Multilingual Shared Task for the first time. I think this is the most important methodological expansion the challenge has made since its founding.

Interpretability starts small

The mechanistic interpretability programme has made genuine progress in understanding what large language models compute. But its methods — activation patching, causal tracing, circuit analysis — become computationally and conceptually unwieldy at scale. Small models are the right substrate for the most precise interpretability work.

One of the NeurIPS CogInterp papers I presented this year argued that theoretical linguistics should constrain the causal abstractions we use in mechanistic interpretability. You cannot do that work rigorously in a 70B model. You can, potentially, do it in a model with 10 million parameters trained on child-directed speech.

The stakes are not only scientific

Small, data-efficient language models are also important for practical reasons that have nothing to do with academic elegance. They run on consumer hardware. They can be trained by researchers in low-resource settings. They are more auditable, more controllable, more amenable to the kinds of safety and fairness interventions that large models resist.

Democratised language modelling is not a consolation prize for people who cannot afford to train large models. It is a genuine goal, with genuine value, for the communities that benefit least from the current centralisation of AI capability.

Small language models still matter. The scientific questions they are best placed to answer are among the most important ones in the field. And the engineering constraints they impose are, increasingly, a feature rather than a limitation.