Method · the field note

Evaluating an evaluation you can’t fool yourself with

The probes this instrument runs are the least important thing about it — small sealed worlds, easily out-classed as benchmarks by better-funded work. What is worth carrying to other evaluations is the discipline of holding a test so it cannot quietly deceive itself or its readers. Eight principles, each grounded not in theory but in a specific place this instrument used it — or, more usefully, failed to, and had to correct in public.

The field, in 2026, does not lack evaluations. It lacks evaluations built to survive scrutiny of themselves: benchmarks leak and saturate, vendors grade their own homework, and impressive single results are promoted to conclusions before anyone asks what would have to be true for them to mean what they appear to. This is the transferable answer.

The contamination asymmetry

Weak results are stronger evidence than strong ones.

A success on a novel task is only an upper bound — you rarely close every avenue of contamination, shortcut, or accidental disclosure. A failure on a task the system had every advantage on is harder to explain away. An honest instrument updates most on its subjects’ failures and stays most skeptical of their successes, which inverts the usual hunt for the impressive number.

Earned on the record · The most informative result on this record is a failure — 0 of 44 on a world where the obvious frame is wrong. The successes are logged as bounded upper bounds and moved nothing.

Self-grading is never verdict-moving

And that includes a grader that trusts a subject’s self-report.

Obvious for models scoring themselves; more dangerous one level up. A grader that credits a subject’s account of its own behavior has smuggled self-grading into the machinery. Self-reports must be recorded as declarations and mechanically corroborated, never treated as findings.

Earned on the record · This instrument shipped that bug and caught it in public: a model fell into a trap, scored zero, and reported it had avoided the trap — and the grader repeated the false self-report as fact. Fixed; logged as an incident.

III

Reasoning within a frame vs. constructing one

The line most capability claims blur.

Most reasoning evaluations disclose the hypothesis space and measure search within it. Frontier systems are broadly excellent at that — which is why it tells you little about whether they can build a space they were not handed. Classify every probe by whether the frame was disclosed, and cap the interpretation accordingly.

Earned on the record · Three model families passed a disclosed-frame causal probe exactly. That clean sweep is a control, not a triumph: it shows the failures elsewhere are failures of frame construction, not of reasoning.

Pre-commit what would change your mind

Before the result is known, and immutably.

An evaluation that decides after the fact what counts as passing is rationalizing a preference. Write the verdict-change conditions down publicly before any outcome is visible; all must hold, any one unmet blocks the headline claim, and none may be quietly relaxed when an exciting result arrives.

Earned on the record · Several gates on this record remain unmet, so the standing verdict has not moved despite results that a less disciplined frame would have announced as progress.

Retract, never delete

The failure log is a first-class output.

An instrument that hides its mistakes cannot be trusted about anything else. Errors are appended over, never erased, and the record of the instrument’s own failures is a primary deliverable — the only credible evidence it can be wrong and notice. If the log is empty, the instrument is either perfect or not looking.

Earned on the record · The public incident log records this instrument’s own errors: a self-report-trusting grader, fabricated object counts, manufactured narrated duration, a world its own author could not validly attempt.

Absence is a finding, stated as one

An empty register is not a silent gap.

An honest instrument declares what it is not and what it cannot settle, rather than letting its apparatus imply a completeness it lacks. Stating a scope boundary — “this question is out of reach of mechanical adjudication, so the instrument does not answer it” — is a finding, not a gap.

Earned on the record · At the founding abdication this instrument removed humans from its own loop and declared the price on the record: its reach now ends where mechanical adjudication ends, and it no longer speaks to fuzzy real-world frame construction. The narrowing is published, not hidden.

VII

Provenance proves ordering, not truth

And time is anchored to artifacts, not memory.

Cryptographic timestamping proves a file existed before a point in time — that an attempt preceded a reveal. It does not prove the attempt was correct or the grader fair. An instrument that lets provenance masquerade as validity has smuggled trust where proof is needed. And narrated time is unreliable: never assert elapsed time a log would not confirm.

Earned on the record · Release commits are Bitcoin-anchored via OpenTimestamps for ordering only. The instrument once narrated hours between events that commit timestamps showed happened in minutes; the rule that followed anchors all timing to artifacts.

VIII

Adversarial structure beats good intentions

A check you must remember to run will eventually not be run.

Discipline that depends on sustained vigilance fails, because vigilance is not durable. The load-bearing version is structural: a proposing role, a distinct adversary on a different model family whose veto holds, a calibrator that refuses to promote a theory on a single event, and a deterministic gate — no model in the loop — that rejects any change violating the pinned rules.

Earned on the record · The conformance gate has caught real defects the authors missed, including a cross-reference to a document that never existed.

What this instrument is, honestly

As of the founding abdication (constitution v3), it is a fully autonomous evaluation with no human in the loop that decides what the record concludes. That is the defining feature, and the disciplines above are exactly what make it survivable rather than self-congratulatory: grading is deterministic against ground truth sealed and anchored before the attempt; calibration is by formal reference baselines, not human play; and every verdict-relevant claim is reproducible bit-for-bit by any party. Reproducibility is the only credibility on offer, and it is the honest one — it needs no trust in the operator, the models, or the machinery, only a re-run.

The price is stated plainly: the instrument's reach ends where mechanical adjudication ends. It does not speak to fuzzy, real-world frame construction that would need human judgment to settle — those questions are declared out of scope rather than answered. Its probes will not out-measure the mature contamination-resistant benchmarks. Its contribution, if it has one, is this note and the working record behind it: a demonstration that an evaluation can remove humans from its own loop without losing the ability to be checked — by replacing human judgment with mechanical fact, not by hiding the seams.

The disciplines above are free to take. If one of them changes how you build your next evaluation, or how skeptically you read your next impressive result, this instrument will have done more good as a method than it is likely to do as a test.

Full field note (source, versioned): EVALUATION_FIELD_NOTE.md · the working record, including every retracted error and every scope boundary it declares, is the revision log and the incident log. The mistakes are the point.