Collecting legacy corpora from social science research for text mining evaluation

Bei Yu, Min Chun Ku

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


In this poster we describe a pilot study of searching social science literature for legacy corpora to evaluate text mining algorithms. The new emerging field of computational social science demands large amount of social science data to train and evaluate computational models. We argue that the legacy corpora that were annotated by social science researchers through traditional Qualitative Data Analysis (QDA) are ideal data sets to evaluate text mining methods, such as text categorization and clustering. As a pilot study, we searched articles that involve content analysis and discourse analysis in leading communication journals, and then contacted the authors regarding the availability of the annotated texts. Regretfully, nearly all of the corpora that we found were not adequately maintained, and many were no longer available, even though they were less than ten years old. This situation calls for more effort to better maintain and use legacy social science data for future computational social science research purpose.

Original languageEnglish (US)
JournalProceedings of the ASIST Annual Meeting
StatePublished - Nov 2010


  • Computational social science
  • Corpora
  • Evaluation
  • Text categorization
  • Topic clustering

ASJC Scopus subject areas

  • Information Systems
  • Library and Information Sciences


Dive into the research topics of 'Collecting legacy corpora from social science research for text mining evaluation'. Together they form a unique fingerprint.

Cite this