The question
Can a model learn a reusable approach from a few examples without losing abilities it already has?
RESEARCH
We study how smaller AI models learn with limited memory and computing power, and whether what they learn can help with new tasks.
Back to researchCan a model learn a reusable approach from a few examples without losing abilities it already has?
In a limited simulation task, the model showed signs of adapting to new situations, but it also lost some existing abilities. A later on-device adjustment trial still did not outperform the unmodified model overall.
The current results do not show that the model can reliably apply what it learned to new, independent tasks or that these methods are ready for a product.
EXP-003 · 2026-10-07
This technical report is based on the original experiment records; it is not a published paper. It explains the research question, method, results, and scope.
We asked whether showing a small model past experience would help it handle a similar problem. We compared no memory with one or two examples, then checked whether turning experience into rule cards worked. The clearest finding was that every card failed validation before behavioral testing, so this run never actually tested whether rule cards improve learning.
The preregistered study covered eight synthetic task families, completed 264 model calls, and recorded 612 events. Conditions were A0 (no memory), B1 (one raw experience), B2 (two raw experiences), and C2 (experience converted into rule cards). An independent checker validated each card; only valid cards could enter the C2 behavioral comparison.
Accuracy was 50.0% for A0, 9.4% for B1, and 15.6% for B2. None of the eight rule-card families passed validation, so C2 followed the preregistered fallback to A0 and also scored 50.0%. Both A0 and the C2 fallback answered every no-change item correctly and every state-change item incorrectly. Overall averages alone would have hidden where the model failed.
These results do not support a claim that rule cards helped the model learn new tasks: the cards never entered a valid behavioral comparison. C2 scored 34.375 percentage points above B2 because it fell back to the no-memory baseline, not because of the cards. The next practical question is how to represent experience so it can be validated and reused, then compare again in a fresh preregistered study.
A next run should first test whether the card format passes blinded validity checks, then fix the questions, prompt, model version, and scoring rules. It should use new task families and also check whether earlier abilities regress. If cards are still rejected or work only on familiar tasks, the claim should be narrowed rather than treating repaired outputs as new evidence.
Accuracy is taken from the original result record. C2 used the no-memory fallback and must not be read as a rule-card effect.
Observed accuracy across eight synthetic task families. None of the three valid comparisons scored above the no-memory baseline.
All eight rule-card families (C2) failed validation and fell back to A0 by protocol, so this is not a rule-card test score.
| Test condition | Observed value | Interpretation |
|---|---|---|
| No memory (A0) | 50.0% | Baseline; all no-change items correct, all state-change items incorrect. |
| One raw experience (B1) | 9.4% | Lowest observed accuracy in this run. |
| Two raw experiences (B2) | 15.6% | Still below the no-memory baseline. |
| Rule cards (C2) | 50.0% | All eight card families failed validation and fell back to A0; this is not an improvement from cards. |
Source record: EXP003-RUN02-v1.md; SHA-256 5a6764416ed86ca4dacc0f265d14c8fed1e5a141a83037f528c8c242707a25c9. Research note based on the experiment record; not a peer-reviewed paper.
This external study separates planning from execution: a larger model labels subgoals along successful paths, then 770M planning and execution modules are trained. The authors report a 16.7-point absolute gain over imitation of basic actions on ScienceWorld. It offers context on breaking tasks down for smaller models; it is not this memory experiment or a result we reproduced.
The current results do not show that the model can reliably apply what it learned to new, independent tasks or that these methods are ready for a product.
Explore other research topics