One connector, four scholarly indexes, real DOIs and legal free PDFs. Works in Claude and ChatGPT.
Paste it into Claude or ChatGPT as a custom connector. 3-day free trial.
Sayash Kapoor, Arvind Narayanan
Machine-learning (ML) methods have gained prominence in the quantitative sciences. However, there are many known methodological pitfalls, including data leakage, in ML-based science. We systematically investigate reproducibility issues in ML-based science. Through a survey of literature in fields that have adopted ML methods, we find 17 fields where leakage has been found, collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions. Based on our survey, we introduce a detailed taxonomy of eight types of leakage, ranging from textbook errors to open research problems. We propose that researchers test for each type of leakage by filling out model info sheets, which we introduce. Finally, we conduct a reproducibility study of civil war prediction, where complex ML models are believed to vastly outperform traditional statistical models such as logistic regression (LR). When the errors are corrected, complex ML models do not perform substantively better than decades-old LR models.
Formatted from the source metadata, not typed by hand. For a whole reference list, use the citation generator or ask your assistant through Find Research.
Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. https://doi.org/10.1016/j.patter.2023.100804
Kapoor, Sayash, and Arvind Narayanan. "Leakage and the reproducibility crisis in machine-learning-based science." Patterns, vol. 4, no. 9, 2023, pp. 100804, https://doi.org/10.1016/j.patter.2023.100804.
Kapoor, Sayash, and Arvind Narayanan. "Leakage and the reproducibility crisis in machine-learning-based science." Patterns 4, no. 9 (2023): 100804. https://doi.org/10.1016/j.patter.2023.100804.
Kapoor S., Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science // Patterns. — 2023. — Vol. 4, № 9. — P. 100804. — URL: https://doi.org/10.1016/j.patter.2023.100804
@article{kapoor2023leakage,
title = {{Leakage and the reproducibility crisis in machine-learning-based science}},
author = {Sayash Kapoor and Arvind Narayanan},
journal = {Patterns},
year = {2023},
volume = {4},
number = {9},
pages = {100804},
doi = {10.1016/j.patter.2023.100804},
url = {https://doi.org/10.1016/j.patter.2023.100804}
}TY - JOUR AU - Kapoor, Sayash AU - Narayanan, Arvind TI - Leakage and the reproducibility crisis in machine-learning-based science JO - Patterns VL - 4 IS - 9 SP - 100804 PY - 2023 DA - 2023/08/04 DO - 10.1016/j.patter.2023.100804 UR - https://doi.org/10.1016/j.patter.2023.100804 ER -
Add Find Research to Claude or ChatGPT and ask in plain language. The assistant reads this record, finds the papers around it and cites only what it actually found:
Setup takes about a minute: add the connector, then start the free trial.
Record from OpenAlex; open-access location from Unpaywall and the indexes. Find Research links only to legal copies.