One connector, four scholarly indexes, real DOIs and legal free PDFs. Works in Claude and ChatGPT.
Paste it into Claude or ChatGPT as a custom connector. 3-day free trial.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather J. Cole-Lewis, Stephen Pfohl and 22 more
Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model 1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM 2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA 3 , MedMCQA 4 , PubMedQA 5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics 6 ), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today’s models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.
Formatted from the source metadata, not typed by hand. For a whole reference list, use the citation generator or ask your assistant through Find Research.
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A. K., Cole-Lewis, H. J., Pfohl, S., Payne, P. W., Seneviratne, M. G., Gamble, P. R., Kelly, C. J., Babiker, A., A, N. S., Chowdhery, A., Mansfield, P., Demner‐Fushman, D., … Natarajan, V. (2023). Large language models encode clinical knowledge. Nature, 620(7972), 172–180. https://doi.org/10.1038/s41586-023-06291-2
Singhal, Karan, et al. "Large language models encode clinical knowledge." Nature, vol. 620, no. 7972, 2023, pp. 172–180, https://doi.org/10.1038/s41586-023-06291-2.
Singhal, Karan, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, et al. "Large language models encode clinical knowledge." Nature 620, no. 7972 (2023): 172–180. https://doi.org/10.1038/s41586-023-06291-2.
Singhal K., Azizi S., Tu T. [и др.] Large language models encode clinical knowledge // Nature. — 2023. — Vol. 620, № 7972. — P. 172–180. — URL: https://doi.org/10.1038/s41586-023-06291-2
@article{singhal2023large,
title = {{Large language models encode clinical knowledge}},
author = {Karan Singhal and Shekoofeh Azizi and Tao Tu and S. Sara Mahdavi and Jason Wei and Hyung Won Chung and Nathan Scales and Ajay Kumar Tanwani and Heather J. Cole-Lewis and Stephen Pfohl and Perry W. Payne and Martin G. Seneviratne and Paul R Gamble and Christopher J. Kelly and Abubakr Babiker and Nathanael Schärli A and Aakanksha Chowdhery and Philip Mansfield and Dina Demner‐Fushman and Blaise Agüera y Arcas and Dale R. Webster and Greg S. Corrado and Yossi Matias and Katherine Chou and Juraj Gottweis and Nenad Tomašev and Yun Liu and Alvin Rajkomar and Joelle K. Barral and Christopher Semturs and Alan Karthikesalingam and Vivek Natarajan},
journal = {Nature},
year = {2023},
volume = {620},
number = {7972},
pages = {172--180},
doi = {10.1038/s41586-023-06291-2},
url = {https://doi.org/10.1038/s41586-023-06291-2}
}TY - JOUR AU - Singhal, Karan AU - Azizi, Shekoofeh AU - Tu, Tao AU - Mahdavi, S. Sara AU - Wei, Jason AU - Chung, Hyung Won AU - Scales, Nathan AU - Tanwani, Ajay Kumar AU - Cole-Lewis, Heather J. AU - Pfohl, Stephen AU - Payne, Perry W. AU - Seneviratne, Martin G. AU - Gamble, Paul R AU - Kelly, Christopher J. AU - Babiker, Abubakr AU - A, Nathanael Schärli AU - Chowdhery, Aakanksha AU - Mansfield, Philip AU - Demner‐Fushman, Dina AU - Arcas, Blaise Agüera y AU - Webster, Dale R. AU - Corrado, Greg S. AU - Matias, Yossi AU - Chou, Katherine AU - Gottweis, Juraj AU - Tomašev, Nenad AU - Liu, Yun AU - Rajkomar, Alvin AU - Barral, Joelle K. AU - Semturs, Christopher AU - Karthikesalingam, Alan AU - Natarajan, Vivek TI - Large language models encode clinical knowledge JO - Nature VL - 620 IS - 7972 SP - 172 EP - 180 PY - 2023 DA - 2023/07/12 DO - 10.1038/s41586-023-06291-2 UR - https://doi.org/10.1038/s41586-023-06291-2 ER -
Add Find Research to Claude or ChatGPT and ask in plain language. The assistant reads this record, finds the papers around it and cites only what it actually found:
Setup takes about a minute: add the connector, then start the free trial.
Record from OpenAlex; open-access location from Unpaywall and the indexes. Find Research links only to legal copies.