ACM Transactions on Management Information Systems

The emergence of Large Language Models (LLM), such as ChatGPT, is considered a productivity revolution in many areas of business and society. For a classroom setting, especially, it would be useful to understand whether, and how, to incorporate ChatGPT, similar to any other productivity revolution technology, such as calculators or a Google search engine. Although there are concerns regarding the use of LLMs in business education, the positive or negative impact of LLM use is not well-understood. In this research, we examine the substitution and complementarity effects of using ChatGPT in business curricula on learning outcomes and well-being in a socially supportive learning environment. Specifically, we examine whether technology anchors impact students’ goal orientation, learning outcomes, and well-being by conducting an empirical study with students majoring in Information Systems. Our analysis reveals that a technology anchor (computer playfulness) can complement the effects of social support on learning outcomes, while enhancing well-being for simple tasks. Students’ well-being and learning outcomes are hindered by LLM use (specifically, the computer anxiety anchor), substituting social support for simple and difficult tasks. These findings have implications for educational institutions that are assessing how to incorporate LLMs into business curricula.

Amrita George

Veda Catherine Storey

Shuguang Hong

Unraveling the Impact of ChatGPT as a Knowledge Anchor in Business Education

揭示ChatGPT作为商业教育知识锚点的影响

Large Language Models (LLMs) have demonstrated impressive capabilities across diverse Natural Language Processing (NLP) tasks, including language understanding, reasoning, and generation. However, general-domain LLMs often struggle with financial tasks due to the technical and specialized nature of financial texts. This study investigates the efficacy of instruction fine-tuning smaller-scale LLMs, including Mistral-7B, Llama3-8B, and Phi3-mini, to enhance their performance in financial text classification tasks. We fine-tuned both instruction-tuned and base models across four financial classification tasks, achieving significant improvements in task-specific performance. Furthermore, we evaluated the zero-shot capabilities of these fine-tuned models on three unseen complex financial tasks, including argument classification, deal completeness classification, and causal classification. Our results indicate while base model fine-tuning led to greater degradation, instruction-tuned models maintained more robust performance. To address this degradation, we employed model merging techniques, integrating single-task domain-specific fine-tuned models with the base model. Using this merging method resulted in significant enhancements in zero-shot performance, even exceeding the original model’s accuracy on certain datasets. Our findings underscore the effectiveness of instruction fine-tuning and model merging for adapting LLMs to specialized financial text classification tasks.

Sorouralsadat Fatemi

Yuheng Hu

Maryam Mousavi

A Comparative Analysis of Instruction Fine-Tuning Large Language Models for Financial Text Classification

金融文本分类中微调大型语言模型的比较分析

More efforts are being put into improving Large Language Models’ (LLM) capabilities than into dealing with their implications. Current LLMs are able to generate high quality texts seemingly indistinguishable from those written by human experts. While offering great potentials, such breakthroughs also pose new challenges for safe and ethical uses of LLMs in education, science, and a multitude of other areas. To add up, the majority of current approaches in LLM text detection are either computationally expensive or need access to the LLMs’ internal computations, both of which hinder their public accessibility. With such motivation, this paper presents a novel metric learning paradigm for detection of LLM-generated texts that is able to balance among computational costs, accessibility, and performances. Specifically, the detection is based on learning a similarity function between a given text and an equivalent example generated by LLMs that outputs high values for LLM-LLM text pairs and low values for LLM-human text pairs. In terms of architecture, the detection framework includes a pretrained language model for the text embedding task and a newly designed deep metric model. The metric component can be trained on triplets or pairs of same-context instances to signify the distances between human texts and LLM ones while reducing that among LLM texts. Next, we develop five datasets totalling over 95,000 contexts and triplets of responses in which one from human and two from GPT-3.5 TURBO or GPT-4 TURBO for benchmarking. Experiment studies show that our best architectures maintain F1 scores in between 0.87 to 0.95 across the tested corpora in multiple experiment settings. The metric framework also demands significantly less time in training and inference compared to the RoBERTa, LLaMA 3, Mistral v0.3, and Ghostbuster, while keeping 90% to 150% performances of the best benchmark.

Linh Le

Dung Tran

A Metric-Based Detection System for Large Language Model Texts

基于度量的检测系统用于大型语言模型文本

Digital supply chains are increasingly interconnected and vulnerable to disruption, causing service interruptions impacting many firms and their customers. Combating threats to the digital supply chain is the top challenge for leaders in most supply chain industries, demonstrated by the tacit approval of nation-states for cyber-attacks on corporate supply chains to disrupt downstream firms. Disruptions to digital supply chains are not new. In April 2019, hundreds of flights in the United States were delayed when a critical service provider, AeroData, had a computer systems failure. AeroData delivers flight planning services to many airlines including Southwest, United, American, and Delta. All flight operations for AeroData’s over 100 clients simultaneously ceased and thousands of customers were stranded at airports across the country. In an increasingly connected business environment, competitors may be simultaneously disrupted due to a common service provider, impacting all affected firms’ demand. The synchronization of disruptions for firms that use a common service provider has implications for service provider choice and investment. We use a two-stage game to model how a firm’s customer demand is impacted by disruptions at a service provider, and how this subsequently affects the firms’ choices in managing service provider risk. Considering downstream demand effects from upstream service disruptions, the contribution of this paper is the examination of how risk synchronization impacts provider choice decisions and profits. In addition, we illustrate how these choices impact upstream industry concentration.

M. Lisa Yeo

Hooman Hidaji

Erik Rolland

Raymond A. Patterson

Barrie Nault

Bora Kolfal

IT Service Disruptions and Provider Choice

IT服务中断与供应商选择

The global rise in mental disorders, particularly in workplaces, necessitated innovative and scalable solutions for delivering therapy. Large Language Model (LLM)-based mental health chatbots have rapidly emerged as a promising tool for overcoming the time, cost, and accessibility constraints often associated with traditional mental health therapy. However, LLM-based mental health chatbots are in their nascency, with significant opportunities to enhance their capabilities to operate within organizational contexts. To this end, this research seeks to examine the role and development of LLMs in mental health chatbots over the past half-decade. Through our review, we identified over 50 mental health-related chatbots, including 22 LLM-based models targeting general mental health, depression, anxiety, stress, and suicide ideation. These chatbots are primarily used for emotional support and guidance but often lack capabilities specifically designed for workplace mental health, where such issues are increasingly prevalent. The review covers their development, applications, evaluation, ethical concerns, integration with traditional services, LLM-as-a-service, and various other business implications in organizational settings. We provide a research illustration of how LLM-based approaches could overcome the identified limitations and also offer a system that could help facilitate systematic evaluation of LLM-based mental health chatbots. We offer suggestions for future research tailored to workplace mental health needs.

Aijia Yuan

Edlin Garcia Colato

Bernice Pescosolido

Hyunju Song

Sagar Samtani

Improving Workplace Well-being in Modern Organizations: A Review of Large Language Model-based Mental Health Chatbots

现代组织中提升职场福祉：基于大型语言模型的心理健康聊天机器人综述

Abbreviation	ACM Trans. Manage. Inf. Syst.
Journal Impact	2.59
Quartiles(Global)	COMPUTER SCIENCE, INFORMATION SYSTEMS(Q2)

ISSN	2158-656X, 2158-6578
h-index	38

Publication Information	Publisher: Association for Computing Machinery (ACM)，Publishing cycle: ，Journal Type: journal，Open Access Journals: No
Basic data	Year of publication: 2010，Proportion of original research papers: 100.00%，Self Citation Rate:8.00%， Gold OA Rate: 4.59%

ACM Transactions on Management Information Systems

Journal Citation Format

A journal article with 1 author

A journal article with 2 authors

A journal article with 3 authors

A journal article with 5 or more authors

Books Citation Format

Thesis Citation Format

Web sites Citation Format

Patent Citation Format

Staying up late manually editing references? ivySCI automatically matches journals and helps you generate references with a single click.

Share Submission Experience

Unraveling the Impact of ChatGPT as a Knowledge Anchor in Business Education

A Comparative Analysis of Instruction Fine-Tuning Large Language Models for Financial Text Classification

A Metric-Based Detection System for Large Language Model Texts

IT Service Disruptions and Provider Choice

Improving Workplace Well-being in Modern Organizations: A Review of Large Language Model-based Mental Health Chatbots

A Comparative Analysis of Instruction Fine-Tuning Large Language Models for Financial Text Classification

Unraveling the Impact of ChatGPT as a Knowledge Anchor in Business Education

A Metric-Based Detection System for Large Language Model Texts

Improving Workplace Well-being in Modern Organizations: A Review of Large Language Model-based Mental Health Chatbots

IT Service Disruptions and Provider Choice