One connector, four scholarly indexes, real DOIs and legal free PDFs. Works in Claude and ChatGPT.
Paste it into Claude or ChatGPT as a custom connector. 3-day free trial.
Catherine A. Gao, Frederick Matthew Howard, Nikolay S. Markov, Emma Dyer, Siddhi Ramesh, Yuan Luo, Alexander T. Pearson
Background Large language models such as ChatGPT can produce increasingly realistic text, with unknown information on the accuracy and integrity of using these models in scientific writing. Methods We gathered ten research abstracts from five high impact factor medical journals (n=50) and asked ChatGPT to generate research abstracts based on their titles and journals. We evaluated the abstracts using an artificial intelligence (AI) output detector, plagiarism detector, and had blinded human reviewers try to distinguish whether abstracts were original or generated. Results All ChatGPT-generated abstracts were written clearly but only 8% correctly followed the specific journal’s formatting requirements. Most generated abstracts were detected using the AI output detector, with scores (higher meaning more likely to be generated) of median [interquartile range] of 99.98% [12.73, 99.98] compared with very low probability of AI-generated output in the original abstracts of 0.02% [0.02, 0.09]. The AUROC of the AI output detector was 0.94. Generated abstracts scored very high on originality using the plagiarism detector (100% [100, 100] originality). Generated abstracts had a similar patient cohort size as original abstracts, though the exact numbers were fabricated. When given a mixture of original and general abstracts, blinded human reviewers correctly identified 68% of generated abstracts as being generated by ChatGPT, but incorrectly identified 14% of original abstracts as being generated. Reviewers indicated that it was surprisingly difficult to differentiate between the two, but that the generated abstracts were vaguer and had a formulaic feel to the writing. Conclusion ChatGPT writes believable scientific abstracts, though with completely generated data. These are original without any plagiarism detected but are often identifiable using an AI output detector and skeptical human reviewers. Abstract evaluation for journals and medical conferences must adapt policy and practice to maintain rigorous scientific standards; we suggest inclusion of AI output detectors in the editorial process and clear disclosure if these technologies are used. The boundaries of ethical and acceptable use of large language models to help scientific writing remain to be determined.
Formatted from the source metadata, not typed by hand. For a whole reference list, use the citation generator or ask your assistant through Find Research.
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E., Ramesh, S., Luo, Y., & Pearson, A. T. (2023). Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. npj Digital Medicine, 6(1), 75. https://doi.org/10.1038/s41746-023-00819-6
Gao, Catherine A., et al. "Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers." npj Digital Medicine, vol. 6, no. 1, 2023, pp. 75, https://doi.org/10.1038/s41746-023-00819-6.
Gao, Catherine A., Frederick Matthew Howard, Nikolay S. Markov, Emma Dyer, Siddhi Ramesh, Yuan Luo, and Alexander T. Pearson. "Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers." npj Digital Medicine 6, no. 1 (2023): 75. https://doi.org/10.1038/s41746-023-00819-6.
Gao C. A., Howard F. M., Markov N. S. [и др.] Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers // npj Digital Medicine. — 2023. — Vol. 6, № 1. — P. 75. — URL: https://doi.org/10.1038/s41746-023-00819-6
@article{gao2023comparing,
title = {{Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers}},
author = {Catherine A. Gao and Frederick Matthew Howard and Nikolay S. Markov and Emma Dyer and Siddhi Ramesh and Yuan Luo and Alexander T. Pearson},
journal = {npj Digital Medicine},
year = {2023},
volume = {6},
number = {1},
pages = {75},
doi = {10.1038/s41746-023-00819-6},
url = {https://doi.org/10.1038/s41746-023-00819-6}
}TY - JOUR AU - Gao, Catherine A. AU - Howard, Frederick Matthew AU - Markov, Nikolay S. AU - Dyer, Emma AU - Ramesh, Siddhi AU - Luo, Yuan AU - Pearson, Alexander T. TI - Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers JO - npj Digital Medicine VL - 6 IS - 1 SP - 75 PY - 2023 DA - 2023/04/26 DO - 10.1038/s41746-023-00819-6 UR - https://doi.org/10.1038/s41746-023-00819-6 ER -
Add Find Research to Claude or ChatGPT and ask in plain language. The assistant reads this record, finds the papers around it and cites only what it actually found:
Setup takes about a minute: add the connector, then start the free trial.
Record from OpenAlex; open-access location from Unpaywall and the indexes. Find Research links only to legal copies.