solver.press

← All replicated papers

Replicated under our protocolOct 26, 2026

Distilling Directional Verification

Lee et al. · arXiv 2610.00997

The outcome is the verdict on the one claim below, under the protocol we fixed in advance. It is not a verdict on the paper as a whole.

Claim tested

Students trained on known-direction four-teacher-mean labels beat summed-DC reverse-label students on open accuracy.

What we ran and what came back

The authors' repository on five seeds they did not use. Students trained on known-direction pseudo-labels were 12.7 points more accurate on open questions than students trained on summed-DC labels (interval 11.0 to 14.5), against a bound of 7.5 set at about half the paper's reported gap of 14.8. The paper's gap sits just above our interval, so the effect reproduces slightly smaller. We trained in 32-bit floating point on a T4 GPU because that card has no native bf16; the paper used bf16 on larger cards. Label accuracy matched the paper's for all five seeds. Students reproduce their labels 97 to 98% of the time, so the result mostly shows that the gap in label accuracy passes through training; it does not test whether the teachers' knowledge is as claimed. Two controls named in the protocol as not done: a gold-label student and a shuffled-label student.

What this result supports

The known-direction labelling gain survives retraining at this size and recipe.

What it does not support

That the teachers' knowledge is as claimed; larger students; other relations.

Record

Protocol, deposited before the run: 10.5281/zenodo.23430538

The authors were sent this result. They have not reviewed this listing.

How the protocol works is on the method page. If you are an author of this paper and something here is wrong, write to customers@aegismind.app and we will correct it on this page.