Variational Autoencoders for Collaborative Filtering
Liang, Krishnan, Hoffman and Jebara · arXiv 1802.05814
The outcome is the verdict on the one claim below, under the protocol we fixed in advance. It is not a verdict on the paper as a whole.
Claim tested
Mult-VAE^PR beats the best simple baseline on ML-20M.
What we ran and what came back
A calibration case, run to check that our protocol can return a positive. The released code reproduced the paper's three figures on MovieLens-20M (NDCG@100 0.4247 against 0.426). Against SLIM, the strongest simple baseline by validation score, Mult-VAE was ahead by 0.024 NDCG@100 (interval 0.022 to 0.026), 0.025 Recall@20 and 0.041 Recall@50, in 5 of 5 seeds on each, against a margin of 0.003. Our SLIM matched the paper's SLIM (0.401). We had also pre-registered an expectation, taken from a later reproducibility study, that the NDCG gap would shrink to nothing against a tuned SLIM. That did not happen, and on checking the source the expectation was mis-sourced: the study prints numbers for Netflix only. A wider SLIM grid and an independent SLIM implementation, both run afterwards as exploratory checks, left the gap unchanged. One dataset; Netflix and the Million Song Dataset were not run.
What this result supports
Positive calibration of the pipeline: a well-known claim replicates and the instrument says so.
What it does not support
Other datasets; the choice of baselines was ours (SLIM by validation).
Record
Protocol, frozen by git commit before the run and deposited afterwards (the archive was unreachable at the time): 10.5281/zenodo.23332633
The authors were sent this result. They have not reviewed this listing.
How the protocol works is on the method page. If you are an author of this paper and something here is wrong, write to customers@aegismind.app and we will correct it on this page.