One connector, four scholarly indexes, real DOIs and legal free PDFs. Works in Claude and ChatGPT.
Paste it into Claude or ChatGPT as a custom connector. 3-day free trial.
Fabrizio Gilardi, Meysam Alizadeh, Maël Kubli
Many NLP applications require manual text annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using four samples of tweets and news articles ( n = 6,183), we show that ChatGPT outperforms crowd workers for several annotation tasks, including relevance, stance, topics, and frame detection. Across the four datasets, the zero-shot accuracy of ChatGPT exceeds that of crowd workers by about 25 percentage points on average, while ChatGPT’s intercoder agreement exceeds that of both crowd workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003—about thirty times cheaper than MTurk. These results demonstrate the potential of large language models to drastically increase the efficiency of text classification.
Formatted from the source metadata, not typed by hand. For a whole reference list, use the citation generator or ask your assistant through Find Research.
Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30), e2305016120. https://doi.org/10.1073/pnas.2305016120
Gilardi, Fabrizio, et al. "ChatGPT outperforms crowd workers for text-annotation tasks." Proceedings of the National Academy of Sciences, vol. 120, no. 30, 2023, pp. e2305016120, https://doi.org/10.1073/pnas.2305016120.
Gilardi, Fabrizio, Meysam Alizadeh, and Maël Kubli. "ChatGPT outperforms crowd workers for text-annotation tasks." Proceedings of the National Academy of Sciences 120, no. 30 (2023): e2305016120. https://doi.org/10.1073/pnas.2305016120.
Gilardi F., Alizadeh M., Kubli M. ChatGPT outperforms crowd workers for text-annotation tasks // Proceedings of the National Academy of Sciences. — 2023. — Vol. 120, № 30. — P. e2305016120. — URL: https://doi.org/10.1073/pnas.2305016120
@article{gilardi2023chatgpt,
title = {{ChatGPT outperforms crowd workers for text-annotation tasks}},
author = {Fabrizio Gilardi and Meysam Alizadeh and Maël Kubli},
journal = {Proceedings of the National Academy of Sciences},
year = {2023},
volume = {120},
number = {30},
pages = {e2305016120},
doi = {10.1073/pnas.2305016120},
url = {https://doi.org/10.1073/pnas.2305016120}
}TY - JOUR AU - Gilardi, Fabrizio AU - Alizadeh, Meysam AU - Kubli, Maël TI - ChatGPT outperforms crowd workers for text-annotation tasks JO - Proceedings of the National Academy of Sciences VL - 120 IS - 30 SP - e2305016120 PY - 2023 DA - 2023/07/18 DO - 10.1073/pnas.2305016120 UR - https://doi.org/10.1073/pnas.2305016120 ER -
Add Find Research to Claude or ChatGPT and ask in plain language. The assistant reads this record, finds the papers around it and cites only what it actually found:
Setup takes about a minute: add the connector, then start the free trial.
Record from OpenAlex; open-access location from Unpaywall and the indexes. Find Research links only to legal copies.