Pico — Turning Small-Model Research from Art into Science

Here’s an uncomfortable truth about building language models: a lot of it is still guesswork. Large models often improve simply because they got bigger, which conveniently hides why any particular design choice worked. Shrink the model down and the guesswork stops being convenient. With a tight parameter budget, every decision — the architecture, the optimiser, the data order — actually counts. And yet researchers still lack a clean, systematic way to test a new idea and watch its effect.

Pico is our answer to that. It’s a lightweight, modular framework for doing hypothesis-driven research on small and medium-scale language models — and it was presented as a Systems Demonstration at EMNLP 2025 in Suzhou.

A sandbox, not a monolith

The core idea is simple: make it trivial to change one thing and observe the consequence. Pico is built from two libraries that together give you a practical sandbox. You make a targeted change to a model’s architecture or its training procedure, and you directly observe how that change shows up in the model’s behaviour. No re-plumbing a giant training stack. No confounds from ten other things you changed at the same time.

That “change one thing, watch what happens” loop is the whole scientific method, quietly smuggled into an ML codebase.

Baselines you can trust

Reproducibility is the part everyone nods along to and nobody enjoys. So Pico ships with a suite of baseline models — pico-decoder — trained under standardised conditions and open-sourced for the community. If you want to test whether your idea beats a fair baseline, the baseline is already there, trained the same way every time. That’s the difference between “it seemed to help in my run” and an actual controlled comparison.

Why I care about this one

Pico is the instrument underneath a lot of the work our group does on learning dynamics — the study of how a model’s knowledge forms over the course of training, not just where it ends up. You can’t study a process you can’t observe cleanly, and you can’t observe cleanly if every experiment is a bespoke one-off.

The case studies in the paper show Pico supporting exactly this: iterative small-LM design and analysis, where each step is a real hypothesis rather than a hopeful tweak. That’s the goal — to make small-model research feel less like alchemy and more like science.

Pico is joint work with Richard Diehl-Martinez, David Demitri Africa, Yuval Weiss, Ryan Daniels and Paula Buttery. Read the demo paper on the ACL Anthology or explore the project at picolm.io.