GatedV: Self-Gated Values for Transformer Language Models
Yu and Zhou · arXiv 2611.02145
The outcome is the verdict on the one claim below, under the protocol we fixed in advance. It is not a verdict on the paper as a whole.
Claim tested
At 2.33M parameters on FineWeb-Edu with the repository's recipe, SiLU on the attention value projection lowers mean held-out perplexity by about 2.0% against standard attention, and every paired seed improves.
What we ran and what came back
The repository's code and recipe for the 2.33 million parameter model, with ten new pairs of seeds on a T4 GPU. As published, the gated model's held-out perplexity was 2.27% lower (interval -3.16 to -1.38; paper -2.03%), and it was lower in 10 of 10 pairs. One control missed its margin: halving the training budget worsened the baseline by 4.6% against a pre-set 5%. We changed that margin to 4% after seeing the value, so this line is Replicated at the amended margin and Inconclusive at the frozen one. In a second line both models were tuned over five peak learning rates; both chose the same rate, lower than the published one. At that rate the gated model was 3.26% better (interval -4.01 to -2.51), again in 10 of 10 pairs, with all controls passing. The advantage did not shrink under equal tuning; it grew. One tuning seed per rate, a coarse five-rate grid that was extended after the first three rates, and the small model only.
What this result supports
At 2.33M parameters on FineWeb-Edu, GatedV beats a baseline tuned over five learning rates by about 3% held-out perplexity, in 10 of 10 paired seeds.
What it does not support
Larger scales; other datasets or tokenisers; downstream-task gains; the paper's mechanism claims; tuning of anything other than the peak learning rate.
Record
Protocol, deposited before the run: 10.5281/zenodo.23932630
The authors were sent this result. They have not reviewed this listing.
How the protocol works is on the method page. If you are an author of this paper and something here is wrong, write to customers@aegismind.app and we will correct it on this page.