Agreement between human qualitative coding processes and ai-based automated qualitative coding
DOI:
https://doi.org/10.47058/joa12.8Keywords:
Qualitative Research, coding, artificial intelligence, language modelsAbstract
Artificial intelligence offers significant enhancements to traditional qualitative research methods, particularly in handling large volumes of data and improving result reliability. This study explores the potential of AI-driven automated qualitative analysis by comparing two parallel coding processes of unstructured data composed of open-ended textual responses: one automated using artificial intelligence and the other conducted traditionally through human cognition. An open-ended questionnaire was administered to a sample of 263 Disney fans to understand their perceptions of what the brand represents to them, through a free-response question. The automated coding process employed Python and a language model called Llama 3.2-1b-Instruct. The results showed that while the coding outcomes were highly similar across the dataset, there was only moderate agreement at the individual case level. It is concluded that artificial intelligence demonstrates strong potential in terms of analytical efficiency and scalability, but also reveals limitations by introducing inconsistencies and redundancies in coding, underscoring the need for oversight through human cognitive processes.
Downloads
References
Alvesson, M. y Karreman, D. (2000). Varieties of discourse: On the study of organizations through discourse analysis. Human relations, 53(9), 1125-1149. https://doi.org/10.1177/0018726700539002
Bird, S., Klein, E. y Loper, E. (2009). Natural language processing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc.
Chollet, F. (2021). Deep learning with Python. Simon and Schuster.
Christou, P. (2023a). A critical perspective over whether and how to acknowledge the use of artificial intelligence (AI) in qualitative studies. The Qualitative Report, 28(7), 1981-1991. https://doi.org/10.46743/2160-3715/2023.6407
Christou, P. (2023b). How to use artificial intelligence (AI) as a resource, methodological and analysis tool in qualitative research? Qualitative Report, 28(7), 1968-1980. https://doi.org/10.46743/2160-3715/2023.6406
Christou, P. A. (2024). Thematic Analysis through Artificial Intelligence (AI). The Qualitative Report, 29(2), 560-576. https://doi.org/10.46743/2160-3715/2024.7046
Filieri, R., Lin, Z., Li, Y., Lu, X. y Yang, X. (2022). Customer emotions in service robot encounters: A hybrid machine-human intelligence approach. Journal of Service Research, 25(4), 614-629. https://doi.org/10.1177/10946705221103937
Gao, J., Choo, K. T. W., Cao, J., Lee, R. K. W. y Perrault, S. (2023). CoAIcoder: Examining the effectiveness of AI-assisted human-to-human collaboration in qualitative analysis. ACM Transactions on Computer-Human Interaction, 31(1), 1-38. https://doi.org/10.1145/3617362
Gebreegziabher, S. A., Zhang, Z., Tang, X., Meng, Y., Glassman, E. L. y Li, T. J. J. (2023). Patat: Human-ai collaborative qualitative coding with explainable interactive rule synthesis. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (pp. 1-19). https://doi.org/10.1145/3544548.3581352
Gibson, A. F. y Beattie, A. (2024). More or less than human? Evaluating the role of AI-as-participant in online qualitative research. Qualitative Research in Psychology, 21(2), 175-199. https://doi.org/10.1080/14780887.2024.2311427
Gioia, D. (2021). A systematic methodology for doing qualitative research. The Journal of Applied Behavioral Science, 57(1), 20-29. https://doi.org/10.1177/0021886320982715
Gioia, D. A., Corley, K. G. y Hamilton, A. L. (2013). Seeking qualitative rigor in inductive research: Notes on the Gioia methodology. Organizational research methods, 16(1), 15-31. https://doi.org/10.1177/1094428112452151
Hamilton, L., Elliott, D., Quick, A., Smith, S. y Choplin, V. (2023). Exploring the use of AI in qualitative analysis: A comparative study of guaranteed income data. International Journal of Qualitative Methods, 22. https://doi.org/10.1177/16094069231201504
Jeldes-Delgado, F., Ferreira, T. A., Diaz, D. y Ortiz, R. (2024). Exploring gender stereotypes in financial reporting: An aspect-level sentiment analysis using big data and deep learning. Heliyon, 10(20). https://doi.org/10.1016/j.heliyon.2024.e38915
Jiménez-Partearroyo, M., Medina-López, A. y Rana, S. (2024). Business intelligence and business analytics in tourism: insights through Gioia methodology. International Entrepreneurship and Management Journal, 20, 2287-2321. https://doi.org/10.1007/s11365-024-00973-7
Kozinets, R. V. (1998). On netnography: Initial reflections on consumer research investigations of cyberculture. Advances in consumer research, 25(1), 366-371.
Kozinets, R. V. (2002). The field behind the screen: Using netnography for marketing research in online communities. Journal of marketing research, 39(1), 61-72. https://doi.org/10.1509/jmkr.39.1.61.18935
Kozinets, R. V. (2006). Click to connect: Netnography and tribal advertising. Journal of advertising research, 46(3), 279-288. https://doi.org/10.2501/S0021849906060338
Kozinets, R. V., Scaraboto, D. y Parmentier, M. A. (2018). Evolving netnography: How brand auto-netnography, a netnographic sensibility, and more-than-human netnography can transform your research. Journal of Marketing Management, 34(3-4), 231-242. https://doi.org/10.1080/0267257X.2018.1446488
Kull, K. (2020). Codes: Necessary, but not sufficient for meaning-making. Constructivist Foundations, 15(2), 137-139.
Kumbure, M. M., Lohrmann, C., Luukka, P. y Porras, J. (2022). Machine learning techniques and data for stock market forecasting: A literature review. Expert Systems with Applications, 197. https://doi.org/10.1016/j.eswa.2022.116659
Lexman, R. R., Krishnan, G., Baral, R. y Thomas, S. C. (2024). Navigating the digital classroom: a qualitative content analysis of MOOC discourses in Indian e-newspapers. Journal of Information, Communication and Ethics in Society, 22(4), 494-516. https://doi.org/10.1108/JICES-03-2023-0027
Magnani, G. y Gioia, D. (2023). Using the Gioia Methodology in international business and entrepreneurship research. International Business Review, 32(2). https://doi.org/10.1016/j.ibusrev.2022.102097
Marcolin, C. B., Diniz, E. H., Becker, J. L. y de Oliveira, H. P. G. (2023). Who knows it better? Reassessing human qualitative analysis with text mining. Qualitative Research in Organizations and Management: An International Journal, 18(2), 181-198. https://doi.org/10.1108/QROM-07-2021-2173
Mees-Buss, J., Welch, C. y Piekkari, R. (2022). From templates to heuristics: How and why to move beyond the Gioia methodology. Organizational Research Methods, 25(2), 405-429. https://doi.org/10.1177/1094428120967716
Palea, D., Vadhul, G. y Lee, D. T. (2024). Annota: Peer-based AI Hints Towards Learning Qualitative Coding at Scale. En Proceedings of the 29th International Conference on Intelligent User Interfaces (pp. 455-470). https://doi.org/10.1145/3640543.3645168
Schmitt, B. (2024). Transforming qualitative research in phygital settings: the role of generative AI. Qualitative Market Research: An International Journal, 27(3), 523-526. https://doi.org/10.1108/QMR-08-2023-0107
Sinha, R., Solola, I., Nguyen, H., Swanson, H. y Lawrence, L. (2024). The Role of Generative AI in Qualitative Research: GPT-4's Contributions to a Grounded Theory Analysis. En Proceedings of the Symposium on Learning, Design and Technology (pp. 17-25). https://doi.org/10.1145/3663433.3663456
Published
Issue
Section
License
Copyright (c) 2025 David Alejandro Álvarez Maldonado, Anna Milano-Meneses
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
