ACM Transactions on Management Information Systems

More efforts are being put into improving Large Language Models’ (LLM) capabilities than into dealing with their implications. Current LLMs are able to generate high quality texts seemingly indistinguishable from those written by human experts. While offering great potentials, such breakthroughs also pose new challenges for safe and ethical uses of LLMs in education, science, and a multitude of other areas. To add up, the majority of current approaches in LLM text detection are either computationally expensive or need access to the LLMs’ internal computations, both of which hinder their public accessibility. With such motivation, this paper presents a novel metric learning paradigm for detection of LLM-generated texts that is able to balance among computational costs, accessibility, and performances. Specifically, the detection is based on learning a similarity function between a given text and an equivalent example generated by LLMs that outputs high values for LLM-LLM text pairs and low values for LLM-human text pairs. In terms of architecture, the detection framework includes a pretrained language model for the text embedding task and a newly designed deep metric model. The metric component can be trained on triplets or pairs of same-context instances to signify the distances between human texts and LLM ones while reducing that among LLM texts. Next, we develop five datasets totalling over 95,000 contexts and triplets of responses in which one from human and two from GPT-3.5 TURBO or GPT-4 TURBO for benchmarking. Experiment studies show that our best architectures maintain F1 scores in between 0.87 to 0.95 across the tested corpora in multiple experiment settings. The metric framework also demands significantly less time in training and inference compared to the RoBERTa, LLaMA 3, Mistral v0.3, and Ghostbuster, while keeping 90% to 150% performances of the best benchmark.

Linh Le

Dung Tran

A Metric-Based Detection System for Large Language Model Texts

基于度量的检测系统用于大型语言模型文本

Digital supply chains are increasingly interconnected and vulnerable to disruption, causing service interruptions impacting many firms and their customers. Combating threats to the digital supply chain is the top challenge for leaders in most supply chain industries, demonstrated by the tacit approval of nation-states for cyber-attacks on corporate supply chains to disrupt downstream firms. Disruptions to digital supply chains are not new. In April 2019, hundreds of flights in the United States were delayed when a critical service provider, AeroData, had a computer systems failure. AeroData delivers flight planning services to many airlines including Southwest, United, American, and Delta. All flight operations for AeroData’s over 100 clients simultaneously ceased and thousands of customers were stranded at airports across the country. In an increasingly connected business environment, competitors may be simultaneously disrupted due to a common service provider, impacting all affected firms’ demand. The synchronization of disruptions for firms that use a common service provider has implications for service provider choice and investment. We use a two-stage game to model how a firm’s customer demand is impacted by disruptions at a service provider, and how this subsequently affects the firms’ choices in managing service provider risk. Considering downstream demand effects from upstream service disruptions, the contribution of this paper is the examination of how risk synchronization impacts provider choice decisions and profits. In addition, we illustrate how these choices impact upstream industry concentration.

M. Lisa Yeo

Hooman Hidaji

Erik Rolland

Raymond A. Patterson

Barrie Nault

Bora Kolfal

IT Service Disruptions and Provider Choice

IT服务中断与供应商选择

The global rise in mental disorders, particularly in workplaces, necessitated innovative and scalable solutions for delivering therapy. Large Language Model (LLM)-based mental health chatbots have rapidly emerged as a promising tool for overcoming the time, cost, and accessibility constraints often associated with traditional mental health therapy. However, LLM-based mental health chatbots are in their nascency, with significant opportunities to enhance their capabilities to operate within organizational contexts. To this end, this research seeks to examine the role and development of LLMs in mental health chatbots over the past half-decade. Through our review, we identified over 50 mental health-related chatbots, including 22 LLM-based models targeting general mental health, depression, anxiety, stress, and suicide ideation. These chatbots are primarily used for emotional support and guidance but often lack capabilities specifically designed for workplace mental health, where such issues are increasingly prevalent. The review covers their development, applications, evaluation, ethical concerns, integration with traditional services, LLM-as-a-service, and various other business implications in organizational settings. We provide a research illustration of how LLM-based approaches could overcome the identified limitations and also offer a system that could help facilitate systematic evaluation of LLM-based mental health chatbots. We offer suggestions for future research tailored to workplace mental health needs.

Aijia Yuan

Edlin Garcia Colato

Bernice Pescosolido

Hyunju Song

Sagar Samtani

Improving Workplace Well-being in Modern Organizations: A Review of Large Language Model-based Mental Health Chatbots

现代组织中提升职场福祉：基于大型语言模型的心理健康聊天机器人综述

This study examines the novel integration of Large Language Models (LLMs) into the survey development process in business and research through the development and evaluation of the Behavioral Research ASSistant (BRASS) Bot. We first analyzed the traditional scale development process to identify tasks suitable for LLM integration, including both human-in-the-loop and automated LLM data collection methods. Following this analysis, we developed the details of BRASS Bot, incorporating design principles of falsifiability and reproducibility. We then conducted a comprehensive evaluation of the BRASS Bot across a diverse set of LLMs, including GPT, Claude, Gemini, and Llama, to assess its usability, validity, and reliability. We further demonstrated the practical utility of the BRASS Bot by conducting a user study and a predictive validity simulation. Our research presents both theoretical and practical implications. The augmentation approach of the BRASS Bot enriches the theoretical foundations of behavioral constructs by identifying previously overlooked patterns. Additionally, the BRASS Bot offers significant time and resource efficiency gains while enhancing scale validity. Our work lays the foundation for future research on the broader application of LLMs as both assistants and collaborators in survey analysis and behavioral research design and execution, highlighting their potential for a transformative impact on the field.

Ping Fan Ke

Ka Chung Ng

Human-AI Synergy in Survey Development: Implications from Large Language Models in Business and Research

调查开发中的人工智能协同作用：大型语言模型在商业和研究中的启示

A business process (workflow) is an assembly of tasks to accomplish a business goal. Real-world workflow models often demanded to change due to new laws and policies, changes in the environment, and so on. To understand the inner workings of a business process to facilitate changes, workflow logs have the potential to enable inspecting, monitoring, diagnosing, analyzing, and improving the design of a complex workflow. Querying workflow logs, however, is still mostly an ad hoc practice by workflow managers. In this article, we focus on the problem of querying workflow log concerning both control flow and dataflow properties. We develop a query language based on “incident patterns” to allow the user to directly query workflow logs instead of having to transform such queries into database operations. We provide the formal semantics and a query evaluation algorithm of our language. By deriving an accurate cost model, we develop an optimization mechanism to accelerate query evaluation. Our experiment results demonstrate the effectiveness of the optimization and achieves up to 50× speedup over an adaption of existing evaluation method.

Yan Tang

Weilong Cui

Jianwen Su

A Query Language for Workflow Logs

一种用于工作流日志的查询语言

The primary goal of software defect prediction (SDP) is to predict the software defects for a specific software using historical data or data from past releases of software projects. The existing state of arts on SDP primarily discusses two prediction scenarios: Within Project Defect Prediction (WPDP) and Cross Project Defect Prediction (CPDP). The prediction model belongs to the WPDP scenario, which means that the model is trained and tested on different parts of the same dataset or trained on the dataset belonging to the previous version of the same project. While in the CPDP scenario, training and testing occur on different software project datasets. Due to the unavailability of historical datasets or prior releases of software defect datasets, CPDP is more useful in real-life scenarios. So, CPDP analysis is a very challenging issue in the SDP domain. Sometimes, machine learning (ML) models perform poorly due to inadequate training in the CPDP scenario. To support better CPDP performance, we must carefully build an ML model focusing on lower training error and overfitting issues. To address these issues, we have proposed a cross-project data preprocessing method to correlate the metrics of different project datasets, namely Unique Selection of Matched Metrics (USMM), using the KS test and Hungarian method. To further improve the CPDP performance, we have also used the dropout regularized deep learning (DRDL) model. We have deployed 34 software defect datasets to validate the DRDL model and USMM method. The experimental results demonstrate that the DRDL model using the USMM method (DRDL-USMM) is a promising model to enhance the prediction accuracy, and an improvement in the range of 3.3% to 8.5% as compared to the existing works in the CPDP scenario has been found.

Pravas Bal

Sandeep Kumar

Cross Project Defect Prediction using Dropout Regularized Deep Learning and Unique Matched Metrics

基于Dropout正则化深度学习和唯一匹配指标的跨项目缺陷预测

短名	ACM Trans. Manage. Inf. Syst.
Journal Impact	2.59
国际分区	COMPUTER SCIENCE, INFORMATION SYSTEMS(Q2)

ISSN	2158-656X, 2158-6578
h-index	38

涉及主题	计算机科学数学经济人工智能操作系统工程类政治学法学业务程序设计语言生物机器学习万维网哲学数据挖掘物理统计心理学地质学数据科学计算机安全营销量子力学地理医学认识论知识管理
出版信息	出版商: Association for Computing Machinery (ACM)，出版周期: ，期刊类型: journal，开源期刊: 否
基本数据	创刊年份: 2010，原创研究文献占比: 100.00%，自引率:8.00%， Gold OA占比: 4.59%

ACM Transactions on Management Information Systems

期刊引文格式

有 1 位以上作者的期刊

有 2 位作者的期刊

有 3 位作者的期刊

有 5 位以上作者的期刊

书籍引用格式

学位论文引用格式

网页引用格式

专利引用格式

手工熬夜修改参考文献？研飞自动匹配期刊，一键轻松成稿，支持 Word/WPS

投稿经验分享

A Metric-Based Detection System for Large Language Model Texts

IT Service Disruptions and Provider Choice

Improving Workplace Well-being in Modern Organizations: A Review of Large Language Model-based Mental Health Chatbots

Human-AI Synergy in Survey Development: Implications from Large Language Models in Business and Research

A Query Language for Workflow Logs

A Metric-Based Detection System for Large Language Model Texts

Improving Workplace Well-being in Modern Organizations: A Review of Large Language Model-based Mental Health Chatbots

IT Service Disruptions and Provider Choice

Human-AI Synergy in Survey Development: Implications from Large Language Models in Business and Research

Cross Project Defect Prediction using Dropout Regularized Deep Learning and Unique Matched Metrics