IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v71y2025i9p7888-7906.html

ChatGPT for Textual Analysis? How to Use Generative LLMs in Accounting Research

Author

Listed:
  • Ties de Kok

    (University of Washington, Seattle, Washington 98195)

Abstract

Generative large language models (GLLMs), such as ChatGPT and GPT-4 by OpenAI, are emerging as powerful tools for textual analysis tasks in accounting research. GLLMs can solve any textual analysis task solvable using nongenerative methods as well as tasks previously only solvable using human coding. Whereas GLLMs are new and powerful, they also come with limitations and present new challenges that require care and due diligence. This paper highlights the applications of GLLMs for accounting research and compares them with existing methods. It also provides a framework on how to effectively use GLLMs by addressing key considerations, such as model selection, prompt engineering, and ensuring construct validity. In a case study, I demonstrate the capabilities of GLLMs by detecting nonanswers in earnings conference calls, a traditionally challenging task to automate. The new GPT method achieves an accuracy of 96% and reduces the nonanswer error rate by 70% relative to the existing Gow et al. (2021) method. Finally, I discuss the importance of addressing bias, replicability, and data sharing concerns when using GLLMs. Taken together, this paper provides researchers, reviewers, and editors with the knowledge and tools to effectively use and evaluate GLLMs for academic research.

Suggested Citation

  • Ties de Kok, 2025. "ChatGPT for Textual Analysis? How to Use Generative LLMs in Accounting Research," Management Science, INFORMS, vol. 71(9), pages 7888-7906, September.
    • Handle: RePEc:inm:ormnsc:v:71:y:2025:i:9:p:7888-7906
    DOI: 10.1287/mnsc.2023.03253
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.2023.03253
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mnsc.2023.03253?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Stephan Hollander & Maarten Pronk & Erik Roelofsen, 2010. "Does Silence Speak? An Empirical Analysis of Disclosure Choices During Conference Calls," Journal of Accounting Research, John Wiley & Sons, Ltd., vol. 48(3), pages 531-563, June.
    2. Chuk, Elizabeth & Matsumoto, Dawn & Miller, Gregory S., 2013. "Assessing methods of identifying management forecasts: CIG vs. researcher collected," Journal of Accounting and Economics, Elsevier, vol. 55(1), pages 23-42.
    3. Stephen V. Brown & Xiaoli (Shaolee) Tian & Jennifer Wu Tucker, 2018. "The Spillover Effect of SEC Comment Letters on Qualitative Corporate Disclosure: Evidence from the Risk Factor Disclosure," Contemporary Accounting Research, John Wiley & Sons, vol. 35(2), pages 622-656, June.
    4. Anand, Vic & Bochkay, Khrystyna & Chychyla, Roman & Leone, Andrew, 2020. "Using Python for Text Analysis in Accounting Research," Foundations and Trends(R) in Accounting, now publishers, vol. 14(3-4), pages 128-359, December.
    5. Ian D. Gow & David F. Larcker & Anastasia A. Zakolyukina, 2021. "Non‐Answers During Conference Calls," Journal of Accounting Research, John Wiley & Sons, Ltd., vol. 59(4), pages 1349-1384, September.
    6. Luzi Hail & Ahmed Tahoun & Clare Wang, 2018. "Corporate Scandals and Regulation," Journal of Accounting Research, John Wiley & Sons, Ltd., vol. 56(2), pages 617-671, May.
    7. Andrew C. Call & Rachel W. Flam & Joshua A. Lee & Nathan Y. Sharp, 2024. "Managers’ use of humor on public earnings conference calls," Review of Accounting Studies, Springer, vol. 29(3), pages 2650-2687, September.
    8. Tarek A Hassan & Stephan Hollander & Laurence van Lent & Ahmed Tahoun, 2019. "Firm-Level Political Risk: Measurement and Effects," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 134(4), pages 2135-2202.
    9. Manish Jha & Jialin Qian & Michael Weber & Baozhong Yang, 2024. "ChatGPT and Corporate Policies," Papers 2409.17933, arXiv.org, revised Feb 2025.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cheng, Qiang & Lin, Pengkai & Zhao, Yue, 2025. "Does generative AI facilitate investor Trading? Early evidence from ChatGPT outages," Journal of Accounting and Economics, Elsevier, vol. 80(2).
    2. Ziyuan Xia & Saixing Zeng & Anchen Sun & Huabin Sun & Xiaodong Cai, 2026. "Global Evolution of Social Responsibility in Smart‐Service Industries: Insights From a Cross‐Sector Hybrid Large Language Models Approach," Corporate Social Responsibility and Environmental Management, John Wiley & Sons, vol. 33(2), pages 1815-1831, March.
    3. Shen, Yi & Giroud, Axèle & Han, Xia, 2025. "Navigating turbulent waters: A literature review of socio-political risks and multinational enterprises’ nonmarket strategies," International Business Review, Elsevier, vol. 34(6).
    4. Dou, Runliang & Nan, Guofang & Pan, Yueming & Liu, Xin, 2026. "Balancing privacy considerations and customization preferences for consumer: LLMs adoption and coordination strategies in supply chains," International Journal of Production Economics, Elsevier, vol. 292(C).
    5. Blondeel, Eva & Bullock, Taylor & Gaskin, James & Schuetzler, Ryan & Serre, Rachel & Steffen, Jacob & Wells, Taylor M. & Wood, David A., 2025. "The effects of generative artificial intelligence (GenAI) on learning in an accounting data analytics course," Journal of Accounting Education, Elsevier, vol. 72(C).
    6. Blondeel, Eva & Everaert, Patricia & Opdecam, Evelien, 2025. "A practical guide to implementing ChatGPT as a secondary coder in qualitative research," International Journal of Accounting Information Systems, Elsevier, vol. 56(C).
    7. Stratopoulos, Theophanis C. & Wang, Victor Xiaoqi, 2025. "Artificial intelligence and accounting research: a framework and agenda," International Journal of Accounting Information Systems, Elsevier, vol. 56(C).
    8. Zhao, Nenggui & Wu, Youqing & Li, Kai, 2026. "Human vs. Generative AI: Strategic content creation mode choices for competing creators," International Journal of Production Economics, Elsevier, vol. 292(C).
    9. Fan, Siyu & Kong, Dongmin & Wu, Yifei & Yu, Honghai, 2025. "Digital innovation and supply chain risk: A large language model-based analysis," Pacific-Basin Finance Journal, Elsevier, vol. 92(C).
    10. Liu, Zhen-yuan Ralph & Dong, Shuqi Kyra & Zeng, Wenjuan & Wang, Yu-ting & Niu, Dong-fang, 2025. "Exploring the impact of human-centred AI on firms’ social and operational performance: A large language model approach," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 203(C).
    11. Li, Yang, 2025. "Can large language models (LLMs) replace human reading? Empirical evidence from sustainability reports," Finance Research Letters, Elsevier, vol. 85(PC).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jessica L. Darby & David J. Ketchen & Brent D. Williams & Travis Tokar, 2020. "The Implications of Firm‐Specific Policy Risk, Policy Uncertainty, and Industry Factors for Inventory: A Resource Dependence Perspective," Journal of Supply Chain Management, Institute for Supply Management, vol. 56(4), pages 3-24, October.
    2. Hao, Mengshu & Xu, Yang & Yuan, Peiyao & Chen, Kecai, 2025. "Unveiling the impact of irrelevant answers on analyst forecast errors: A topic modeling approach," International Review of Financial Analysis, Elsevier, vol. 102(C).
    3. Armstrong, Daphne M. & Glaeser, Stephen & Hoopes, Jeffrey L., 2025. "Measuring firm exposure to government agencies," Journal of Accounting and Economics, Elsevier, vol. 79(1).
    4. Tarek A. Hassan & Stephan Hollander & Aakash Kalyani & Markus Schwedeler & Ahmed Tahoun & Laurence van Lent, 2024. "Text as Data in Economic Analysis," Working Papers 2024-022, Federal Reserve Bank of St. Louis, revised 11 Sep 2025.
    5. Barth, Andreas & Mansouri, Sasan & Wöbbeking, Carl Fabian, 2025. "Information flow and market efficiency: The economic impact of precise language," IWH Discussion Papers 13/2025, Halle Institute for Economic Research (IWH).
    6. Shibo Bian & Iftekhar Hasan & Xunxiao Wang & Zhipeng Yan, 2024. "Do markets value manager-investor interaction quality? Evidence from IPO returns," Review of Quantitative Finance and Accounting, Springer, vol. 63(2), pages 599-632, August.
    7. Xie, Nan & Xu, Qifa & Jiang, Cuixia & Yang, Shanlin, 2023. "The determinants and consequences of firms’ boilerplate answers: Evidence from online interactive platforms in China," Finance Research Letters, Elsevier, vol. 58(PD).
    8. Mansouri, Sasan, 2021. "Does firm's silence drive media's attention away?," VfS Annual Conference 2021 (Virtual Conference): Climate Economics 242433, Verein für Socialpolitik / German Economic Association.
    9. Bozanic, Zahn & Roulstone, Darren T. & Van Buskirk, Andrew, 2018. "Management earnings forecasts and other forward-looking statements," Journal of Accounting and Economics, Elsevier, vol. 65(1), pages 1-20.
    10. Miguel Duro & Jonas Heese & Gaizka Ormazabal, 2019. "The effect of enforcement transparency: Evidence from SEC comment-letter reviews," Review of Accounting Studies, Springer, vol. 24(3), pages 780-823, September.
    11. Xiaoyang Li & Xinyue Zhang, 2025. "Climate Change Exposure and Cash Holdings," Sustainability, MDPI, vol. 17(1), pages 1-35, January.
    12. Chourou, Lamia & Hossain, Ashrafee T. & Jha, Anand, 2024. "US political corruption and quarterly conference calls," Journal of Banking & Finance, Elsevier, vol. 161(C).
    13. Rahman, Sohanur & Sinnewe, Elisabeth & Chapple, Larelle, 2024. "Environment-specific political risk discourse and expected crash risk: The role of political activism," International Review of Financial Analysis, Elsevier, vol. 95(PB).
    14. Zacharias Sautner & Laurence van Lent & Grigory Vilkov & Ruishen Zhang, 2023. "Pricing Climate Change Exposure," Management Science, INFORMS, vol. 69(12), pages 7540-7561, December.
    15. Sean Shun Cao & Guojin Gong & Yongtae Kim & Hanzhong Shi & Angie Wang, 2025. "Site Visits and Corporate Investment Efficiency," Management Science, INFORMS, vol. 71(1), pages 635-658, January.
    16. Tarek Alexander Hassan & Stephan Hollander & Laurence van Lent & Ahmed Tahoun, 2020. "The Global Impact of Brexit Uncertainty," Boston University - Department of Economics - The Institute for Economic Development Working Papers Series dp-332, Boston University - Department of Economics.
    17. Hassan, Tarek & Hollander, Stephan & van Lent, Laurence & Schwedeler, Markus & Tahoun, Ahmed, 2020. "Firm-Level Exposure to Epidemic Diseases: Covid-19, SARS, and H1N1," CEPR Discussion Papers 14573, Centre for Economic Policy Research.
    18. Francois Brochet & Maria Loumioti & George Serafeim, 2015. "Speaking of the short-term: disclosure horizon and managerial myopia," Review of Accounting Studies, Springer, vol. 20(3), pages 1122-1163, September.
    19. Gang Li & Dandan Qiao & Mingxuan Zheng, 2025. "Structured Event Representation and Stock Return Predictability," Papers 2512.19484, arXiv.org.
    20. Andreas Barth & Sasan Mansouri & Fabian Wöbbeking, 2023. "“Let Me Get Back to You”—A Machine Learning Approach to Measuring NonAnswers," Management Science, INFORMS, vol. 69(10), pages 6333-6348, October.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:71:y:2025:i:9:p:7888-7906. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.