RESEARCH

What can small models learn?

We study how smaller AI models learn with limited memory and computing power, and whether what they learn can help with new tasks.

Back to research
01

The question

Can a model learn a reusable approach from a few examples without losing abilities it already has?

02

What we observed

In a limited simulation task, the model showed signs of adapting to new situations, but it also lost some existing abilities. A later on-device adjustment trial still did not outperform the unmodified model overall.

03

What we cannot claim yet

The current results do not show that the model can reliably apply what it learned to new, independent tasks or that these methods are ready for a product.

EXP-003 · 2026-10-07

Can memory help a model perform better?

This technical report is based on the original experiment records; it is not a published paper. It explains the research question, method, results, and scope.

STUDY ABSTRACT

We asked whether showing a small model past experience would help it handle a similar problem. We compared no memory with one or two examples, then checked whether turning experience into rule cards worked. The clearest finding was that every card failed validation before behavioral testing, so this run never actually tested whether rule cards improve learning.

01 / STUDY DESIGN

How did we test it?

The preregistered study covered eight synthetic task families, completed 264 model calls, and recorded 612 events. Conditions were A0 (no memory), B1 (one raw experience), B2 (two raw experiences), and C2 (experience converted into rule cards). An independent checker validated each card; only valid cards could enter the C2 behavioral comparison.

02 / RESULTS

What do the numbers show?

Accuracy was 50.0% for A0, 9.4% for B1, and 15.6% for B2. None of the eight rule-card families passed validation, so C2 followed the preregistered fallback to A0 and also scored 50.0%. Both A0 and the C2 fallback answered every no-change item correctly and every state-change item incorrectly. Overall averages alone would have hidden where the model failed.

03 / INTERPRETATION

What can we conclude?

These results do not support a claim that rule cards helped the model learn new tasks: the cards never entered a valid behavioral comparison. C2 scored 34.375 percentage points above B2 because it fell back to the no-memory baseline, not because of the cards. The next practical question is how to represent experience so it can be validated and reused, then compare again in a fresh preregistered study.

04 / NEXT STEPS

What should be tested next?

A next run should first test whether the card format passes blinded validity checks, then fix the questions, prompt, model version, and scoring rules. It should use new task families and also check whether earlier abilities regress. If cards are still rejected or work only on familiar tasks, the claim should be narrowed rather than treating repaired outputs as new evidence.

Key results

Accuracy is taken from the original result record. C2 used the no-memory fallback and must not be read as a rule-card effect.

RESULTS AT A GLANCE

How did past experience affect task accuracy?

Observed accuracy across eight synthetic task families. None of the three valid comparisons scored above the no-memory baseline.

Valid memory-condition comparisons

No memory (A0)
50.0%
One raw experience (B1)
9.4%
Two raw experiences (B2)
15.6%

All eight rule-card families (C2) failed validation and fell back to A0 by protocol, so this is not a rule-card test score.

Test conditionObserved valueInterpretation
No memory (A0)50.0%Baseline; all no-change items correct, all state-change items incorrect.
One raw experience (B1)9.4%Lowest observed accuracy in this run.
Two raw experiences (B2)15.6%Still below the no-memory baseline.
Rule cards (C2)50.0%All eight card families failed validation and fell back to A0; this is not an improvement from cards.

Source record: EXP003-RUN02-v1.md; SHA-256 5a6764416ed86ca4dacc0f265d14c8fed1e5a141a83037f528c8c242707a25c9. Research note based on the experiment record; not a peer-reviewed paper.

FURTHER READING / EXTERNAL PAPER
Sub-goal Distillation: A Method to Improve Small Language Agents ↗

This external study separates planning from execution: a larger model labels subgoals along successful paths, then 770M planning and execution modules are trained. The authors report a 16.7-point absolute gain over imitation of basic actions on ScienceWorld. It offers context on breaking tasks down for smaller models; it is not this memory experiment or a result we reproduced.

What the evidence supports

The findings apply only to the tests completed so far.

The current results do not show that the model can reliably apply what it learned to new, independent tasks or that these methods are ready for a product.

Explore other research topics

GENESIS / GITHUB

Coming soon

The GitHub entry will open when our products launch.