Measuring a discovery
Unseen worlds, a hidden answer, simple methods at the same cost: how Slean avoids fooling itself.
The question
The question is not “does the agent look like a scientist?”. It is “did it find something true that it did not already know, more cheaply than a simple method would have?”. Each rule below closes one way of fooling ourselves.
Worlds nobody has seen
The elementary automata are well known, and a model may have memorised them. They are used only for calibration. Scored worlds are generated from a seed and disguised: states are secretly renamed, the picture is sometimes mirrored, and rules without any law serve as decoys.
A complete hidden answer
For every scored world, the complete answer is computed exactly in advance. The score compares what the explorer found with that answer, so silence and false leads both show up. Trivial laws, conserved by every rule, do not count.
Boring baselines, at the same cost
Every explorer pays one unit per proposal. The comparison always includes:
- random proposals, which give the floor;
- exact data fitting, fifty lines of linear algebra on simulated transitions;
- the oracle, which knows the theory and gives the ceiling.
An AI that does not beat exact data fitting at the same cost has not shown that it discovers.
Hidden laws found out of 20, held-out half
Show the data
| Explorer | Laws found | Proposals |
|---|---|---|
| Random35 proposals refuted. This is the floor. | 3 / 20 | 40 |
| Random, larger budgetMore budget is not enough. | 7 / 20 | 300 |
| Exact data fittingExact linear algebra on simulated transitions. | 20 / 20 | 24 |
| Library, coldNo prior memory. | 20 / 20 | 20 |
| OracleKnows the theory and solves the continuity equations. This is the ceiling. | 20 / 20 | 20 |
| Language modelClaude Sonnet 5.5, no tools. 31 proposals for the 14 laws, 48 in total. 9 calls, about 388,000 input and 26,000 output tokens. | 14 / 20 | 31 / 48 |
On conservation laws, exact data fitting finds all 20 hidden laws in 24 proposals. This kind of law is therefore solved by a generic method. It calibrates the engine; it cannot demonstrate discovery ability.
Metering access to the world
Simulation is the experiment. Labs count cell updates next to proposals, and the model’s tokens with the cost. See Lab.
Cold, then with the library
Memory is only useful if it saves something. The transfer protocol scores the same held-out worlds twice at the same budget: cold, then with the library built on other worlds of the same family. On conservation laws, exact data fitting goes from 24 to 20 proposals: the cold method is already close to optimal, so there is almost nothing to save.
What a positive result will mean
On worlds generated after the system was frozen: recall above exact data fitting and random at equal metered cost, a gain with the library, and the same result over several seeds, with the spread published. We are not there yet.