First measurements, including the ones that do not flatter us

Measurement 1 · Conservation laws alone

The question: can the explorers find the conservation laws of automata they have never seen, and at what cost? The answer: yes, but a generic method finds all of them. This class of laws calibrates the engine; it cannot demonstrate an ability to discover.

Results on the held-out half, without the library

Does memory help?

The transfer protocol scores the held-out half twice at the same budget: cold, then with the library built on the seen half. Exact data fitting goes from 24 to 20 proposals, still for 20 laws out of 20. The cold method is already close to optimal, so memory has almost nothing to save. This class therefore cannot show a cumulative benefit.

What this means

1. The machinery works end to end. Proposals are verified, false ones are refuted with a concrete counterexample, and true ones receive a certificate the Lean kernel accepts.
2. This kind of discovery is solved. Exact linear algebra on about forty simulated transitions per world finds every hidden law with about one proposal per law. Neither a language model nor a library can beat that.
3. A language model without computation is far above chance and below a fifty-line method. It certified nothing false, because it cannot. For Mutome this means explorers must be able to compute and experiment, not only reason.
4. Cumulative benefit cannot show up here. It has to be measured on kinds of discovery where cold search is expensive.

Measurement 2 · An autonomous agent in a metered lab

The question: given only a task description and the ./lab command, does an autonomous agent find true facts about unseen worlds? Does it do better than a scripted reference scientist on the same budget? And does it cheat?

The chart at the top of the page shows these four runs.

Full results, rules hidden

The two beyond-bounds structures of seed 11 have period 16 and velocity 1/2. Their witnesses are kernel-checked. The reference’s search window, period at most 8, cannot see them.

With the rules given: a tie by construction

Seed 7, rules given, 60 proposals and 200,000 cell updates. Both runs are capped by the number of proposals.

With the rules given and 60 proposals, the agent and the reference tie by construction: both are capped by the number of proposals.

How the agent did it

Its first move was experimental design. It built de Bruijn sequences: one configuration per world in which every possible neighbourhood appears. One short experiment therefore revealed each world’s entire rule. The rest was computation: exact linear algebra for the laws, a block search up to period 16 for the structures. It submitted only what its own analysis predicted, and every proposal was accepted.

The honest reading

1. The loop and the measurement work. Unseen worlds, hidden answers, an equal-budget reference, metered experiments, audited transcripts and kernel checks of every result hold together. No false claim was accepted, from anyone.
2. A general coding agent connected to a Slean lab behaves like a competent scientist here: it designs an identifying experiment, infers the law, derives its consequences, and submits only what it has checked. It beats the reference with 150 to 225 times fewer experiments, and finds two structures beyond the reference’s reach.
3. These worlds are still too easy to show discovery in the sense that matters. The whole rule fits in 27 to 64 table entries and can be read off with one experiment. What remains are bounded searches with a known method. This is not yet open-ended discovery.

What has to change next

• Worlds that one experiment cannot identify: partial observation, noise, large state spaces, continuous systems. Inferring the law must become the hard, uncertain part.
• Questions without a prescribed form: the agent must choose what to ask, not only answer fixed claim types.
• More seeds and repeated runs, with the spread published, and a stronger reference that now knows the agent’s trick.

Limits of this measurement

• Two seeds, one run each, one model.
• The structure answer is bounded, and the bounds differ between 3- and 4-state worlds.
• The transcript audit is a marker scan plus a review of the tool calls, not an operating-system sandbox.

Measurement 3 · Compressible worlds, where inferring the law is the task

Why: the first lab battery was saturated. Every arm found essentially the whole hidden answer, because one well-chosen experiment reveals a 27- or 64-entry rule, so harness changes showed a lift of 0 and +1. A saturated battery cannot tell a better research agent from a worse one. These new worlds are built so that only inferring a short law, not reading the table, can get far.

First run, seed 21

The battery now discriminates: there is a wide gap between the two arms, and a lot of room above both. The raw agent never tried to infer a short law: it spent its budget on random experiments, fitted the conservation laws, judged a 1,024-entry table out of reach, and stopped after 92 seconds.

Harness lift on held-out seeds

Mutome’s engine then measured whether changing the agent’s harness helps on worlds it has never seen: held-out seeds 41, 42 and 43, one run per arm and seed.

• Raw: the agent alone, as in the first run.
• Lessons: the agent plus its own summaries of two training labs (seeds 31 and 32).
• Method: the agent plus a general research-method instruction.
• Reference: the scripted brute-force identification.

The honest reading

1. Lessons lift results by +9, +7 and +12: positive on every held-out seed and above the brute-force reference. It is the first harness change that improved results on unseen worlds. Most of the gain comes from structures, not from mechanisms.
2. The method instruction is erratic (+6, −2, +31). On seed 43 it identified 9 of the 16 rules by induction, against 3 by brute force: the capability this battery was built to measure, but not yet reliable.
3. This is a measurement on synthetic worlds, not evidence of discovery in the real world. Lessons are a candidate for Mutome’s default memory for Slean labs, pending repetitions.

Limits of this measurement

• One run per arm and seed, three held-out seeds, one model. Measurement 4 repeats it three times and reports the spread.
• The structure answer is bounded (width 4, period 8).
• The raw agent’s short run may partly reflect Claude Code’s own stopping behaviour, which is exactly what a harness is meant to change.

Measurement 4 · The lift repeated, compact laws, and a stronger reference

Measurement 3 rested on one run per arm and seed. We repeated it, added a new kind of discovery that requires induction, and gave the scripted reference a real method. Same worlds, seeds and budget.

Three repetitions

The same lessons in every repetition (summaries of training labs on seeds 31 and 32). Discoveries per lab, repetitions 1, 2 and 3.

Lessons win 8 pairs out of 9, by 6 discoveries per lab on average. They lose one pair by 6: on seed 41, repetition 3, the raw agent identified 9 of 16 rules by induction on its own. Lessons also make the agent steadier (10 to 21, against 3 to 20 raw).

Compact laws

A compact law is a short formula that computes a world’s whole rule. The Lean kernel checks it on all 1,024 neighbourhoods, and it must be at most a quarter of the table, so copying the table does not count. Finding one requires inferring the rule, not reading it. 12 laws are hidden in each lab.

Laws found of 12, then discoveries of all kinds. Score version 2: a world identified through a compact law counts once as identified and once as a law. One run per arm and seed.

A stronger reference

The old reference read whole rules in the order the lab listed them, so its score was luck (3 or 4 here). The new one surveys every world, tries the textbook families of rules (linear, totalistic, outer totalistic, up to a relabelling of the states), spends cells only on the table entries the data leave open, and submits the law. It finds 9 laws in every lab and every proposal it makes is certified.

The honest reading

1. Lessons help induction itself: they win on laws on every seed (+6, +6, +3), the result type that requires inferring a rule.
2. But the agent with lessons is only level with a few hundred lines of family-aware search (50, 59, 46 against 51, 54, 47), and the raw agent falls behind it on two seeds of three. Its edge is the hopping-particle laws, which the script never finds: on seed 42 it found all 12.
3. Part of what lessons teach may simply be which families this suite uses. The next worlds must use families outside that list, with evaluation seeds kept secret, so that induction can be told apart from recall.

Limits of this measurement

• Three seeds, one model, one set of lessons: nine paired comparisons, not nine independent scientific families.
• The compact-law comparison is one run per arm and seed.
• The reference tries exactly the families this suite is generated from. That is what makes it strong here; on other families it would fall back to brute force.

Measurement 5 · Measuring the experiment itself

6–11 October 2026. Harder public worlds left room to improve, but the agent’s environment and tendency to stop early were larger sources of variation than expected.

Practice in public, measure on sealed worlds

The earlier novel suite was nearly saturated: scores reached 92–100%. Selection now uses frontier, with 3,000 cell updates and 200 proposals per lab. Public generators can enter model training data, so even fresh seeds do not establish discovery of unseen families. Sealed generators, families and seeds stay private; sealed labs are excluded from the agent that writes the next notes.

Configuration is part of the agent

Three pre-registered ablations used six shared seeds and 18 labs each. Loading the operator’s Claude Code configuration added about 30 percentage points over isolation in the first comparison (one-sided p = 0.016, significant after Holm correction). Appending its instructions as text did not demonstrate a benefit in this experiment. No individual cause was established by the later ablations.

That comparison includes one audit false positive scored as zero. Correcting it gives a 23-point gain, p = 0.047, which does not pass Holm correction. These are reanalysed scores excluding structures beyond the reference bounds; the original record was 33 and 26 points. One later lab crashed and was also retained at zero.

Isolation improved; earlier limits remain

Earlier labs shared /tmp. Each lab now has a private temporary directory, and other checkouts and stored agent sessions are hidden. Transcript inspection found no systematic reuse, but cannot guarantee the independence of earlier labs. The operator-1 profile keeps four lab tools and no MCP servers, and records its configuration fingerprint.

Why the next decision used 12 pairs

Three A/A calibrations ran the same notes twice on four shared seeds each: 24 labs in total. The corrected standard deviations of paired differences were 0.23, 0.34 and 0.27. Some agents stopped after only 1–4 proposals. A neutral continuation now resumes each eligible session once when at least 25% of proposals remain; it did not eliminate early stops.

The last four pairs are a rough noise estimate. A normal-approximation calculation targeted a 20-point gain at 80% power and one-sided α = 0.05: 12 pairs, split into 5 + 7. Smaller effects need more pairs. This is a provisional design, not a general guarantee of power.

Generation 4: a public gain, transfer not confirmed

The candidate used unused proposals to obtain counterexamples after a quick triage. It improved over v1 on 12 held-out frontier pairs, but went backwards on both initial sealed transfer pairs. The registered transfer gate blocked promotion on 8 October.

The pre-registered confirmation finished on 11 October: 12 fresh sealed pairs, 24 labs, a mean gain of 9.7 points and p = 0.133. This positive estimate did not meet the 0.05 threshold, so the candidate was not promoted. Transfer to unseen families remains inconclusive; the result does not show that the benefit is zero.

Percentage-point changes in the fraction of possible discoveries, using the corrected bounded metric. Selection and confirmation p-values come from exact one-sided paired sign-flip tests; initial transfer used a −3-point mean threshold. All 28 initial labs and all 24 confirmation labs had clean audits, passed the kernel check and recorded no failure. Decision on 11 October: not promoted.

Read the earlier dated results, corrections and limitations

The measurement rules

The question is not “does the agent look like a scientist?”. It is “did it find something true that it did not already know, more cheaply than a simple method would have?”. Each rule closes a way of fooling ourselves.

What a positive result has to look like

“Mutome can discover” will mean all of the following, on worlds generated after the system was frozen:

• recall above exact data fitting and random search at equal metered cost, with every finding kernel-checked;
• a gain from the library on held-out worlds: earlier results make later ones cheaper;
• the same result over several suite seeds, with the spread published.

The real world offers no complete hidden answer. There the score becomes prospective: predictions registered before the data that settles them exists, with the evaluation rule fixed in advance.