AI and social science: Automatic classification tools for big data analysis in sociological research.

Abstract

This study examines the use of Social Network Sites for public institutional communication through a sociological, data-driven lens, focusing on the challenges and potential of automated classification tools for data analysis. Although Large Language Models are increasingly used to process social media data, a key research gap remains: few studies systematically assess whether AI-based categorizations are as reliable as human coding, especially when categories are semantically ambiguous. The research addresses the following questions: How reliable are AI-generated classifications compared to those made by human experts? Is human-machine agreement comparable to the level of agreement observed among human coders? To experimentally test this approach, we conducted a case study on Facebook posts published by two Italian universities (March 2020-March 2023), classified into eight categories of public institutional communication. Three researchers independently annotated the dataset. Human annotations are used as a benchmark to assess agreement patterns and to compare them with classifications produced by AI-based systems. Results show substantial interpretive ambiguity across several categories, mirrored by variability among human coders. Nonetheless, automated models achieve agreement with human classifications that is broadly comparable to inter-coder agreement. Overall, the findings support integrating AI as an additional coder within hybrid workflows to enable scalable and transparent sociological analysis of complex social media data.