ACM Transactions on Computing for Healthcare

Using extensive data sources to create foundation language models has revolutionized the performance of deep learning-based architectures. This remarkable improvement has led to state-of-the-art results for various downstream NLP tasks, including clinical tasks. However, more research is needed to measure model performance intrinsically, especially in the clinical domain. We revisit the use of analogy questions as an effective method to measure the intrinsic performance of language models for the clinical domain in English. We tested multiple Transformers-based language models over analogy questions constructed from the Unified Medical Language System (UMLS), a massive knowledge graph of clinical concepts. Our results show that large language models are significantly more performant for analogy resolution than small language models. Similarly, domain-specific language models perform better than general domain language models. We also found a correlation between intrinsic and extrinsic performance, validated through PubMedQA extrinsic task. Creating clinical-specific and language-specific language models is essential for advancing biomedical and clinical NLP and will ensure a valid application in clinical practice. Finally, given that our proposed intrinsic test is based on a term graph available in multiple languages, the dataset can be built to measure the performance of models in languages other than English.

Fabián Villena

Tamara Quiroga

Jocelyn Dunstan

Clinical analogy resolution performance for foundation language models

基础语言模型的临床类比解析性能

Mental fatigue is a crucial aspect that has gained attention across various disciplines due to its impact on overall well-being. While previous research has explored the use of wearable devices for detecting mental fatigue, limited investigation has been conducted into the effectiveness of these devices in different body positions or in multi-device setups. To address this, our study utilizes a unique public dataset containing over 13 hours of sensor data collected across 36 sessions, with four wearable devices (Earable, Chestband, Wristband, and Headband). We propose several machine learning-based approaches to assess both psychological and physiological mental fatigue levels in a multimodal and multi-device environment. Specifically, we introduce device type-specific approaches (trained and tested on a single device) and multi-device approaches (trained and tested on multiple devices) for mental fatigue inference tasks. Our findings show that device type-specific models perform well, with AUC scores ranging from 0.63 to 0.69 for psychological and from 0.74 to 0.80 for physiological mental fatigue. The multi-device approach shows improved performance for psychological mental fatigue (AUC of 0.69 to 0.74) and physiological mental fatigue (AUC of 0.81 to 0.88). Hence, this study presents a unique and in-depth analysis of detecting mental fatigue with wearables, demonstrating the potential of machine learning-based approaches in multi-device and multimodal setups that are prevalent in today’s emerging lifestyles.

Chalindu Kodikara

Sapumal Wijekoon

Lakmal Meegahapola

FatigueSense: Multi-Device and Multi-Modal Wearable Sensing for Detecting Mental Fatigue

疲劳感知：多设备与多模态可穿戴传感用于检测精神疲劳

This paper explores video analysis methods for monitoring eating behaviors, a critical factor in approximately 70% of global deaths due to illnesses like cancer, diabetes, and heart disease. Automated monitoring quantifies aspects such as meal duration, food types, and intake gestures (bite and drink gestures). Previous deep-learning methods segment videos into short clips (e.g., 16 frames at 8 Hz) for analysis, but this approach overlooks common meal-length patterns in gesture distribution across different individuals and sessions, which can enhance detection accuracy. Our study introduces a novel pipeline that analyzes the entire meal context (5-40 minutes). We propose a framework allowing a global detector to learn meal-length patterns with manageable computational demands. Additionally, we introduced a new augmentation technique to generate hundreds of meal-length feature samples per video, facilitating effective training of a global detector with limited video availability. Experimental results on two datasets (Clemson Cafeteria and EatSense) demonstrate that our pipeline significantly enhances the performance of state-of-the-art window-based networks, particularly in reducing false positives in gesture detection. On the Clemson Cafeteria dataset of 486 meal videos (the largest dataset to date), our method achieves F1 scores of 0.93 for bite gestures and 0.88 for drink gestures, substantially outperforming existing methodologies.

Zeyu Tang

Adam Hoover

Video-based Intake Gesture Recognition using Meal-length Context

基于视频的进食动作识别：利用用餐时长上下文

Providing contextual and comprehensive medical information tailored to individual patients is critical for enabling effective care in the healthcare domain. However, existing approaches often struggle to deliver personalized responses due to the distributed nature of medical data across multiple sources such as patient records, medical literature, and online resources. To address this challenge, we present MedInsight , a multi-source context augmentation framework that leverages Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) techniques to augment patient-specific information from medical transcripts with trusted knowledge from textbooks and web resources to generate personalized and contextually relevant responses. Our framework consists of three phases: patient context retrieval, medical knowledge retrieval, and response generation. By augmenting patient context with relevant external knowledge, MedInsight generates contextually relevant responses, empowering patients and caregivers with actionable insights. Experiments on the MTSamples dataset validate MedInsight ’s effectiveness in generating contextually appropriate medical responses, using a comprehensive set of metrics including RAGAs, TruLens, ROUGE, and BertScore. Additionally, qualitative evaluations by Subject-Matter Experts (SMEs) further confirm the relevance and factual correctness of the generated responses.

Subash Neupane

Shaswata Mitra

Sudip Mittal

Manas Gaur

Noorbakhsh Amiri Golilarz

Shahram Rahimi

Amin Amirlatifi

<scp>MedInsight</scp> : A Multi-Source Context Augmentation Framework for Generating Patient-Centric Medical Responses using Large Language Models

<scp>MedInsight</scp>：基于多源上下文增强框架的以患者为中心的医疗响应生成系统——利用大型语言模型

The continuous development of artificial intelligence (AI) theory has propelled this field to unprecedented heights, owing to the relentless efforts of scholars and researchers. In the medical realm, AI takes a pivotal role, leveraging robust machine learning (ML) algorithms. AI technology in medical imaging aids physicians in X-ray, computed tomography (CT) scans, and magnetic resonance imaging (MRI) diagnoses, conducts pattern recognition and disease prediction based on acoustic data, delivers prognoses on disease types and developmental trends for patients, and employs intelligent health management wearable devices with human-computer interaction technology to name but a few. While these well-established applications have significantly assisted in medical field diagnoses, clinical decision-making, and management, collaboration between the medical and AI sectors faces an urgent challenge: How to substantiate the reliability of decision-making? The underlying issue stems from the conflict between the demand for accountability and result transparency in medical scenarios and the black-box model traits of AI. This article reviews recent research grounded in explainable artificial intelligence (XAI), with an emphasis on medical practices within the visual, audio, and multimodal perspectives. We endeavour to categorise and synthesise these practices, aiming to provide support and guidance for future researchers and healthcare professionals.

Qiyang Sun

Alican Akman

Björn W. Schuller

Explainable Artificial Intelligence for Medical Applications: A Review

可解释人工智能在医疗应用中的综述

Forecasting future health status is beneficial for understanding health patterns and providing anticipatory support for cognitive and physical health difficulties. In recent years, generative large language models (LLMs) have shown promise as forecasters. Though not traditionally considered strong candidates for numeric tasks, LLMs demonstrate emerging abilities to address various forecasting problems. They also provide the ability to incorporate unstructured information and explain their reasoning process. In this paper, we explore whether LLMs can effectively forecast future self-reported health state. To do this, we utilized in-the-moment assessments of mental sharpness, fatigue, and stress from multiple studies, utilizing daily responses ( N =106 participants) and responses that are accompanied by text descriptions of activities ( N =32 participants). With these data, we constructed prompt/response pairs to predict a participant’s next answer. We fine-tuned several LLMs and applied chain-of-thought prompting evaluating forecasting accuracy and prediction explainability. Notably, we found that LLMs achieved the lowest mean absolute error (MAE) overall (0.851), while gradient boosting achieved the lowest overall root mean squared error (RMSE) (1.356). When additional text context was provided, LLM forecasts achieved the lowest MAE for predicting mental sharpness (0.862), fatigue (1.000), and stress (0.414). These multimodal LLMs further outperformed the numeric baselines in terms of RMSE when predicting stress (0.947), although numeric algorithms achieved the best RMSE results for mental sharpness (1.246) and fatigue (1.587). This study offers valuable insights for future applications of LLMs in health-based forecasting. The findings suggest that LLMs, when supplemented with additional text information, can be effective tools for improving health forecasting accuracy.

Gina Sprint

Maureen Schmitter-Edgecombe

Raven Weaver

Lisa Wiese

Diane J. Cook

CogProg: Utilizing Large Language Models to Forecast In-the-moment Health Assessment

CogProg：利用大型语言模型进行即时健康评估预测

ISSN	2637-8051, 2691-1957
h-index	16

Publication Information	Publisher: Association for Computing Machinery (ACM)，Publishing cycle: ，Journal Type: journal，Open Access Journals: No
Basic data	Year of publication: 2020，Proportion of original research papers: ，Self Citation Rate:， Gold OA Rate:

ACM Transactions on Computing for Healthcare

Journal Citation Format

A journal article with 1 author

A journal article with 2 authors

A journal article with 3 authors

A journal article with 5 or more authors

Books Citation Format

Thesis Citation Format

Web sites Citation Format

Patent Citation Format

Staying up late manually editing references? ivySCI automatically matches journals and helps you generate references with a single click.

Share Submission Experience

Clinical analogy resolution performance for foundation language models

FatigueSense: Multi-Device and Multi-Modal Wearable Sensing for Detecting Mental Fatigue

Video-based Intake Gesture Recognition using Meal-length Context

MedInsight : A Multi-Source Context Augmentation Framework for Generating Patient-Centric Medical Responses using Large Language Models

Explainable Artificial Intelligence for Medical Applications: A Review

CogProg: Utilizing Large Language Models to Forecast In-the-moment Health Assessment

Clinical analogy resolution performance for foundation language models

FatigueSense: Multi-Device and Multi-Modal Wearable Sensing for Detecting Mental Fatigue

Video-based Intake Gesture Recognition using Meal-length Context

MedInsight : A Multi-Source Context Augmentation Framework for Generating Patient-Centric Medical Responses using Large Language Models