RESEARCH

Does a change really make it better?

We compare a model before and after adjustments to see whether gains carry over to unfamiliar tasks, rather than only appearing in the original test.

Back to research
01

The question

Can a changed model do better on new tasks without losing what it could already do?

02

What we observed

Early simulations outperformed a simple baseline, but still fell short of the improvement threshold when compared with a stricter control.

03

What we cannot claim yet

There is no evidence yet that the model can keep improving on its own, and these changes have not been confirmed in any public product.

RSI-RECURSION-PILOT-002 · 2026-10-08

Is the changed strategy actually better?

This technical report is based on the original experiment records; it is not a published paper. It explains the research question, method, results, and scope.

STUDY ABSTRACT

This study compares two ways to choose the next subtask: one adapts its order using prior feedback, while the other follows a fixed curriculum. The question is whether feedback-based selection improves scores under the same budget and keeps that advantage in new synthetic test worlds. The measured gain came from an explicitly written software selector, not an AI model modifying itself.

01 / STUDY DESIGN

How did we test it?

We first compared feedback-based selection, a fixed curriculum, random budgeting, and simple priority selection across 24 development worlds. We then froze the same strategy and compared it again in 48 separately generated descendant test worlds. Both stages used synthetic tasks and the same resource budget, with an independent verifier checking output integrity. This checks whether the strategy reproduces its result in these generated tasks, not whether it passes an independent real-world evaluation.

02 / RESULTS

What do the numbers show?

Mean development scores were 0.7619 for feedback-based selection, 0.4870 for the fixed curriculum, 0.4612 for random budgeting, and 0.4991 for simple priority. Across 48 descendant test worlds, the scores were 0.7560, 0.4744, 0.5173, and 0.4317, respectively. The feedback-minus-fixed difference in the descendant stage was 0.2816, with a 95% bootstrap interval from 0.2097 to 0.3585. All 51 integrity checks passed.

03 / INTERPRETATION

What can we conclude?

In researcher-designed synthetic worlds, the feedback selector scored higher than the fixed curriculum, and the difference appeared again in newly generated descendant worlds. The score measures how the software chooses tasks from feedback; it does not measure model reasoning, learning, or self-improvement. The task worlds and candidate strategy came from the same research effort, with no real product integration or external blinded scoring.

04 / NEXT STEPS

What should be tested next?

A stronger evaluation would use independent researchers to create different task families, preregister the budget and acceptance threshold, score blindly, and include distorted feedback and unfamiliar conditions. If the strategy loses its advantage under those tests, the result should remain limited to the original synthetic setting.

Key results

Scores and confidence intervals are from the synthetic experiment record. The 48 worlds are independent descendant tests within one authored family, not 48 real-world applications.

RESULTS AT A GLANCE

Mean scores across 48 descendant test worlds

Feedback-based task selection scored above the fixed curriculum; random budgeting and simple priority are included for comparison.

Descendant test / 48 synthetic worlds

Feedback-based selection
0.7560
Fixed curriculum
0.4744
Random budget
0.5173
Simple priority
0.4317

The mean difference between feedback-based selection and the fixed curriculum was 0.2816, with a 95% bootstrap interval of 0.2097–0.3585. This is a result from a researcher-designed software selector, not a model improving itself.

Test conditionObserved valueInterpretation
Development worlds (24)0.7619 vs 0.4870Feedback-based selection vs fixed curriculum; random and simple-priority controls were also included.
Descendant test worlds (48)0.7560 vs 0.4744Feedback-based selection vs fixed curriculum; 95% bootstrap interval for the difference: 0.2097–0.3585.
Integrity checks51 / 51 passedChecks output and evidence-chain integrity; this is not external validation.

Source record: RSI-RECURSION-PILOT-002 outcome.json; SHA-256 ae54e163d6d3d48ae73260e92eb46e05678cd33621968290ff8c7525da751567. Research note based on the experiment record; not a peer-reviewed paper.

FURTHER READING / EXTERNAL PAPER
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena ↗

This external study examines bias in model-based judging, including answer position, verbosity, and self-enhancement, and compares judges with human preferences using MT-Bench and Chatbot Arena. It is a reminder to validate evaluation methods too; the experiment on this page used synthetic program scores, not an LLM judge.

What the evidence supports

The findings apply only to the tests completed so far.

There is no evidence yet that the model can keep improving on its own, and these changes have not been confirmed in any public product.

Explore other research topics

GENESIS / GITHUB

Coming soon

The GitHub entry will open when our products launch.