Teaching a Tiny Language Model to Hold a Conversation

Watch a parent talk to a two-year-old and you’ll notice something that no scraped web corpus contains: contingency. The replies are prompt, they’re direct, and they’re about the thing the child just said. Nobody in that exchange is optimising a benchmark. They’re staying on topic, and that turns out to be one of the most powerful teaching signals a young learner ever gets.

Our paper, Teacher Demonstrations in a BabyLM’s Zone of Proximal Development for Contingent Multi-Turn Interaction, asks a deceptively simple question: what happens if we make contingency a training objective rather than an accident of the data? It picked up an Outstanding Paper Award at the BabyLM Workshop at EMNLP 2025 in Suzhou — one of two we were lucky enough to bring home that year.

The setup: a Teacher and a Student

Most language models are trained on oceans of text — trillions of tokens. A BabyLM gets 100 million words, roughly what a child hears by age ten. That constraint is the whole point: if you want to understand learning, you have to study it under a realistic budget, where every design decision actually matters.

We built a framework we call ContingentChat. It’s a Teacher–Student setup. The Student is a BabyLM trained on 100M words. The Teacher’s job is to demonstrate what a good, contingent turn looks like — the conversational equivalent of a caregiver modelling how to answer. We built a novel alignment dataset for the post-training stage, so that the signal the Student learns from is specifically about staying grammatical, cohesive, and on-topic across multiple turns.

What we found

Two things stand out.

First, targeted post-training works. After learning from the Teacher’s demonstrations, the BabyLM produced responses that were measurably more grammatical and more cohesive across a multi-turn exchange. You don’t need a bigger model to get a better conversationalist — you need a better teaching signal.

Second — and this is the honest part — clever decoding wasn’t the answer. We experimented with adaptive Teacher decoding strategies, expecting them to squeeze out extra gains. They didn’t do much. The heavy lifting came from the alignment data and the post-training, not from decoding tricks at inference time.

Why it matters

Contingency turns out to be a genuinely hard target for small models — which is exactly why it’s interesting. It sits at the intersection of the two things I care about most: building language models under human-scale constraints, and taking seriously what developmental science already knows about how language is actually learned.

If large models improve by brute scale, BabyLMs have to improve by insight. ContingentChat is one small piece of that argument: that the structure of the interaction — not just the size of the corpus — is something a model can be taught.

This work was joint with Hongyi Gu, Donya Rooein, Diana Galvan-Sosa, Gabrielle Gaudeau, Andrew Caines, Zheng Yuan and Paula Buttery. The paper is open access on the ACL Anthology.