Can ChatGPT Understand Causal Language in Science Claims?

Yuheun Kim, Lu Guo, Bei Yu, Yingya Li

Research output: Chapter in Book/Entry/PoemConference contribution

1 Scopus citations

Abstract

This study evaluated ChatGPT’s ability to understand causal language in science papers and news by testing its accuracy in a task of labeling the strength of a claim as causal, conditional causal, correlational, or no relationship. The results show that ChatGPT is still behind the existing fine-tuned BERT models by a large margin. ChatGPT also had difficulty understanding conditional causal claims mitigated by hedges. However, its weakness may be utilized to improve the clarity of human annotation guideline. Chain-of-thought prompting was faithful and helpful for improving prompt performance, but finding the optimal prompt is difficult with inconsistent results and the lack of effective method to establish cause-effect between prompts and outcomes, suggesting caution when generalizing prompt engineering results across tasks or models.

Original languageEnglish (US)
Title of host publicationWASSA 2023 - 13th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, Proceedings of the Workshop
EditorsJeremy Barnes, Orphee De Clercq, Roman Klinger
PublisherAssociation for Computational Linguistics (ACL)
Pages379-389
Number of pages11
ISBN (Electronic)9781959429876
StatePublished - 2023
Externally publishedYes
Event13th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, WASSA 2023 - Toronto, Canada
Duration: Jul 14 2023 → …

Publication series

NameProceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)0736-587X

Conference

Conference13th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, WASSA 2023
Country/TerritoryCanada
CityToronto
Period7/14/23 → …

ASJC Scopus subject areas

  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Fingerprint

Dive into the research topics of 'Can ChatGPT Understand Causal Language in Science Claims?'. Together they form a unique fingerprint.

Cite this