The easiest person to fool in an ML experiment is yourself

Why I started deciding what counted as success before seeing the data — and why that turned out to be only half the problem.

The hardest experiments to interpret weren't the ones that failed outright. They were the ones that almost worked.

A scheduler that finishes a fraction of a point behind the baseline. A pruning setting that stays inside your accuracy tolerance almost everywhere. A training curve that runs above the baseline for a stretch before drifting back below. A clear win asks nothing of your judgment — the number is higher or it isn't — and a clear loss asks nothing either. It's the results in between, close enough that you can tell a story either way, where bias quietly walks in. And when you have spent weeks building the intervention, there is always a flattering story available. At that point you are the least objective reviewer your own work will ever have. Feynman put the general form of this best — the first principle is not to fool yourself, and you are the easiest person to fool. That's easy to quote and hard to actually do.

That is the part most talk of experimental rigor understates. The dangerous result isn't the one that fails loudly — you can't talk yourself out of a two-point drop. It's the near-miss, because a near-miss is exactly ambiguous enough to absorb whatever interpretation you bring to it, and you bring the interpretation of someone who wants it to have worked. This matters far more for negative results than positive ones. A positive result is not the one you are tempted to reopen — the number cleared the bar, and a win invites little second-guessing. A negative result is the one you would most like to reopen — which is precisely why it needs the stricter standard.

This post is about the two-part discipline I ended up using to manage that, and about the thing I had wrong when I thought the first part was the whole of it. It is less a results post than a post about how I decide what a result is allowed to mean.

Part one: decide before you look

The first move is the obvious one, and it is not original: decide what counts as success before you see the results. Before running the scheduling experiments from the earlier posts, I wrote down the verdict criteria — three of them, an intervention had to clear all three to count as beating uniform sampling:

• higher average validation accuracy across the run,
• faster to a target accuracy on every seed,
• final accuracy no more than a point below baseline.

The criteria themselves are ordinary. What matters is that they were fixed while I still had no idea how any run would turn out. A criterion you write beforehand is a commitment. A criterion you write afterward is really just a description of the data that you've decided to call a commitment — and from the inside, the two feel identical. It never feels like moving the bar to fit the result. It feels like realizing the bar should have been there all along. Locking it first is the only way I have found to keep my later self from doing that without noticing.

I thought this solved the problem. It solved half of it.

Part two: the rule doesn't remove judgment — it moves it

Here is the part I did not expect. Predefining the criteria does not take judgment out of the evaluation. It relocates it — to two places where it can at least be seen, instead of letting you quietly move the bar.

The first place is before the experiment, when you decide what the criteria are. The second is after, when a result refuses to sit cleanly on the rule and you have to record what the rule could not capture.

The locked criterion produces the verdict. Everything the verdict flattens still exists — and the discipline is putting it somewhere visible rather than quietly rewriting the verdict. That is the whole move, and almost everything else in this post is a consequence of it.

When the rule decides on its own

DBpedia, seed 11, is the easy kind of verdict — the kind that needs no one's judgment at all. The scheduler's integrated AUC came in at 0.7618 against the baseline's 0.7684 — two-thirds of a point behind on the run-integrated measure. It trailed for most of training and briefly nosed ahead about two-thirds of the way through, but never held the lead long enough to move the integrated number. The first criterion is higher average accuracy; lower is not higher; it fails.

What is worth noticing is where it did not fail. Its final accuracy landed within a point of the baseline — a wash of a few tenths — so the most lenient of the three criteria passed cleanly. A verdict that glanced only at the endpoint would have called this fine. The locked rule did not, because beating uniform meant clearing all three criteria and the run-integrated measure was clearly behind. No weighing, no interpretation, no "but look how it closes the gap at the end." The rule made the decision, and the decision required nothing from me. Those are the easy days.

When the rule and the intuition disagree

AG News was the hard kind. The two seeds finished four and six hundredths of a point behind on integrated AUC — a gap indistinguishable from noise — and for stretches of training the scheduler actually ran ahead of the baseline. Final accuracy sat inside the tolerance on both seeds.

If I had not fixed the criteria beforehand, I know exactly what I would have said: basically neutral. Looks promising. Worth pursuing. None of those is an unreasonable reading. They are also precisely the readings that grow more attractive after you have sunk weeks into an idea. The locked rule did not care: higher means higher, a tiny negative is still negative, and the verdict stayed a fail.

But here is where the second half of the discipline does its work — because recording this as a flat "fail" would be its own small dishonesty. A bare fail implies the scheduler degraded training on AG News. It did not. It tied. The verdict and the underlying truth point in slightly different directions: the rule says fail, the data says wash. So the result is logged as a fail, and traveling alongside it is a note recording exactly that gap — that this is a statistical tie within seed-to-seed noise, not a decisive loss. The verdict stays fixed. The nuance stays visible. Neither is allowed to quietly become the other. That note is the relocated judgment: the verdict could not express the distinction, so a human had to, in the open, where a later reader can check it.

Not every line is a verdict

The first kind of judgment — deciding what your criteria are — has a quieter form I didn't see until I sat down to write this up. Deciding what your criteria are includes deciding which of your lines are criteria at all. Not every line you draw is meant to decide anything, and mistaking one kind for the other is the same move in a subtler disguise — subtle precisely because it can slip past you at definition time and only surface, as it did for me, at the writing desk.

The same one-point line appears twice in this series doing opposite jobs. In the scheduling study it was a gate: final accuracy within a point was a pass/fail criterion, and missing it meant failing. In the earlier pruning study, the one-point tolerance was not a gate at all — I called it a reference point, not a hard limit, a place to mark on the curve so a reader could weigh compute saved against accuracy lost. The difference came from the question each study asked. "Does this beat uniform?" is a yes-or-no question and needs a gate. "How much can you prune before it costs too much?" is a how-far question, and the honest answer is a curve plus a tolerance the reader picks — not a line the method either clears or doesn't.

The temptation, at writing time, is to quietly promote a reference mark into a verdict — a threshold crossed makes a cleaner story than a continuous trade-off. "Stayed within a point at ten percent pruning" reads like a pass the method earned, when the line was only ever a place to point. Remembering which of your lines were ever verdicts belongs to that same first decision — what the criteria are — and like the rest of it, it is a judgment the rule cannot make for you. You make it in the open and mark it as yours.

What I actually learned

When I started writing this post, I thought it was about locking evaluation criteria. It isn't, quite. Preregistering a verdict is necessary, but it is the easy, well-known half. The part that took me longer to see is that locking the criteria does not remove judgment from the evaluation — it separates judgment into two pieces and forces both into the open. One piece goes before the experiment, when success is defined. The other goes after, when the ambiguities the verdict could not hold get written down instead of folded silently into the result.

That separation is the actual tool. The locked rule keeps me from rewriting the question after seeing the answer. The notes keep me honest about every place the rule was simpler than the evidence underneath it. And the reason this matters most for negative results is the same reason I opened with: a win is not the result you are tempted to reopen, but a near-miss will quietly become whatever its author needs it to be — unless the author has tied their own hands in advance and then written down, in the open, every place the knot pinched.

Most of the experiments in this series failed. A few almost succeeded. The only way I found to trust either kind was to decide the verdict before seeing the plots, and then mark every spot where the verdict was cleaner than the truth. Not because any single one of those spots would have overturned a result — but because that is not how you fool yourself. You don't do it with one big lie. You do it with a hundred small roundings, each individually defensible, all leaning the same comfortable direction.