ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY 是一本学术期刊，以多学科的视角发表关于智能系统、适用算法和技术的最高质量论文。智能系统是使用人工智能 (AI) 技术提供重要服务（例如，作为更大系统的组件）以允许集成系统在现实世界中感知、推理、学习和智能行动的系统。ACM TIST 是每季出版（一年六期）。每期有 8-11 篇常规论文，大约有 20 个已发表的期刊页面或每篇论文 10,000 字。额外的参考文献、证明、图表或详细的实验结果可以作为单独的附录提交，而过长的论文将被自动拒绝。作者可以在其已发表论文的其他内容中包含仅在线附录，并鼓励与其他读者共享其代码和/或数据。

ACM Transactions on Intelligent Systems and Technology is an academic journal that publishes high-quality papers on intelligent systems, applicable algorithms, and technologies from a multidisciplinary perspective. Intelligent systems use artificial intelligence (AI) technologies to provide essential services, functioning as components of larger systems, enabling integrated systems to perceive, reason, learn, and act intelligently in the real world. ACM TIST is published quarterly, with six issues per year, each containing 8 to 11 regular papers, approximately 20 published journal pages, or up to 10,000 words per paper. Additional references, proofs, charts, or detailed experimental results may be submitted as separate appendices, and excessively long papers will be automatically rejected. Authors are allowed to include online-only appendices in their published papers and are encouraged to share their code and/or data with other readers.

《ACM Transactions on Intelligent Systems and Technology》是一本学术期刊，采用多学科的视角，发表关于智能系统、相关算法和技术的高质量论文。智能系统利用人工智能（AI）技术提供重要服务，作为更大系统的组成部分，使得集成系统能够在现实世界中进行感知、推理、学习和智能行动。该期刊每季度出版六期，每期包含8到11篇常规论文，约20页的已发表期刊内容，或每篇论文不超过10,000字。额外的参考文献、证明、图表或详细的实验结果可以作为附录提交，过长的论文将被自动拒绝。作者可以在已发表论文中包含仅在线附录，并鼓励与其他读者共享其代码和/或数据。

ACM Transactions on Intelligent Systems and Technology

Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate , where to evaluate , and how to evaluate . Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, education, natural and social sciences, agent applications, and other areas. Secondly, we answer the ‘where’ and ‘how’ questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing the performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey

Yupeng Chang

Xu Wang

Jindong Wang

Yuan Wu

Linyi Yang

Kaijie Zhu

Hao Chen

Xiaoyuan Yi

Cunxiang Wang

Yidong Wang

Wei Ye

Yue Zhang

Yi Chang

Philip S. Yu

Qiang Yang

Xing Xie

ACM Trans. Intell. Syst. Technol.

A Survey on Evaluation of Large Language Models

大型语言模型评估综述

Traditional recommender systems aim to estimate a user’s rating to an item based on observed ratings from the population. As with all observational studies, hidden confounders, which are factors that affect both item exposures and user ratings, lead to a systematic bias in the estimation. Consequently, causal inference has been introduced in recommendations to address the influence of unobserved confounders. Observing that confounders in recommendations are usually shared among items and are therefore multi-cause confounders, we model the recommendation as a multi-cause multi-outcome (MCMO) inference problem. Specifically, to remedy the confounding bias, we estimate user-specific latent variables that render the item exposures independent Bernoulli trials. The generative distribution is parameterized by a DNN with factorized logistic likelihood and the intractable posteriors are estimated by variational inference. Controlling these factors as substitute confounders, under mild assumptions, can eliminate the bias incurred by multi-cause confounders. Furthermore, we show that MCMO modeling may lead to high variance due to scarce observations associated with the high-dimensional treatment space. Therefore, we theoretically demonstrate that controlling user features as pre-treatment variables can substantially improve sample efficiency and alleviate overfitting. Empirical studies on both simulated and real-world datasets demonstrate that the proposed deep causal recommender shows more robustness to unobserved confounders than state-of-the-art causal recommenders. Codes and datasets are released at https://github.com/yaochenzhu/Deep-Deconf.

Yaochen Zhu

Jing Yi

Jiayi Xie

Zhenzhong Chen

Deep Causal Reasoning for Recommendations

Wangsheng Zhang

Guande Qi

Gang Pan

Hua Lu

Shijian Li

Zhaohui Wu

City-Scale Social Event Detection and Evaluation with Taxi Traces

城市规模社交事件检测与评估：基于出租车轨迹数据

Session-based recommendation (SBR) systems, traditionally reliant on complex graph neural networks (GNNs), often face challenges with marginal performance improvements despite increased model complexity. In this paper, we dissect the classical GNN-based SBR models and empirically find that the sophisticated GNN propagations might be redundant, given the readout module plays a significant role in GNN-based models. Based on this observation, we introduce Atten-Mixer+, an advanced iteration of our previously developed Multi-Level Attention Mixture Network (Atten-Mixer). Atten-Mixer+ forgoes GNN propagation in favor of a dynamic and adaptive readout process, tailored to the unique characteristics of each session. Different from the vanilla version, Atten-Mixer+ features the Adaptive Intent Scaler (AIS) layer, which dynamically determines the depth of multi-level user intent analysis, and a soft allocation approach for generating user intent queries across entire user interaction sequences. This innovative design allows Atten-Mixer+ to capture a nuanced and comprehensive understanding of user behaviors, overcoming the limitations of fixed-length analysis. Empirical evaluations on benchmark datasets highlight Atten-Mixer+'s superior efficiency and effectiveness, marking a significant step forward in the predictive accuracy of session-based recommendation systems.

Peiyan Zhang

Jiayan Guo

Chaozhuo Li

Liying Kang

Jaeboum Kim

Jie Xu

Xi Zhang

Yan Zhang

Haohan Wang

Sunghun Kim

Advancing Session-Based Recommendations with Atten-Mixer+: Dynamic and Adaptive Multi-Level Intent Mining

推进基于会话的推荐系统：Atten-Mixer+——动态自适应多层次意图挖掘

Tables are powerful and popular tools for organizing and manipulating data. A vast number of tables can be found on the Web, which represent a valuable knowledge resource. The objective of this survey is to synthesize and present two decades of research on web tables. In particular, we organize existing literature into six main categories of information access tasks: table extraction, table interpretation, table search, question answering, knowledge base augmentation, and table augmentation. For each of these tasks, we identify and describe seminal approaches, present relevant resources, and point out interdependencies among the different tasks.

Shuo Zhang

Krisztian Balog

Web Table Extraction, Retrieval, and Augmentation

网页表格提取、检索与增强

In recent years, session-based recommender systems (SRSs) have emerged as a significant research focus within the recommendation field. Capturing user intentions to infer user interest accordingly has proven to be effective in enhancing the accuracy of SRSs. However, existing techniques assume that all sessions have the same number of intentions or that the items in one category belonging to the same session reflect the same intention. In real applications, such as e-commerce, sessions may have different numbers of intentions, and the same type of items in a session may correspond to different intentions. As a result, existing techniques cannot guarantee high-quality user interest prediction. In this paper, we propose a novel A daptive I ntention L earning N etwork (AILN) to capture an adaptive number of intentions for each session, thereby enhancing the accuracy of user interest inference. Specifically, we design an intention evaluation network (IEN) to evaluate whether a subsequence of a session corresponds to a valid intention, and an intention generation network (IGN) to learn the representation of a valid intention. By checking each subsequence of a session, IEN and IGN enable the incremental learning of a session-specific intention hierarchy (IH) to store valid intentions of the session. To reduce the cost of building the IH, we propose a pruning strategy that exploits the intention validity to avoid unnecessary evaluation. The representative intentions are selected from IH and input into a designed interest predictor to infer the user interest. Experimental results on two real-world datasets demonstrate the superiority of our proposed AILN.

Qingbo Zhang

Xiaochun Yang

Bin Wang

Zhu Sun

Xiangmin Zhou

Adaptive Intention Learning for Session-based Recommendation

基于会话的自适应意图学习推荐

Feature noise and label noise are ubiquitous in practical scenarios, which pose great challenges for training a robust machine learning model. Most previous approaches usually deal with only a single problem of either feature noise or label noise. However, in real-world applications, hybrid noise, which contains both feature noise and label noise, is very common due to the unreliable data collection and annotation processes. Although some results have been achieved by a few representation learning based attempts, this issue is still far from being addressed with promising performance and guaranteed theoretical analyses. To address the challenge, we propose a novel unified learning framework called “ F eature and L abel R ecovery” (FLR) to combat the hybrid noise from the perspective of data recovery, where we concurrently reconstruct both the feature matrix and the label matrix of input data. Specifically, the clean feature matrix is discovered by the low-rank approximation, and the ground-truth label matrix is embedded based on the recovered features with a nuclear norm regularization. Meanwhile, the feature noise and label noise are characterized by their respective adaptive matrix norms to satisfy the corresponding maximum likelihood. As this framework leads to a non-convex optimization problem, we develop the non-convex Alternating Direction Method of Multipliers (ADMM) with the convergence guarantee to solve our learning objective. We also provide the theoretical analysis to show that the generalization error of FLR can be upper-bounded in the presence of hybrid noise. Experimental results on several typical benchmark datasets clearly demonstrate the superiority of our proposed method over the state-of-the-art robust learning approaches for various noises.

Yang Wei

Shuo Chen

Shanshan Ye

Bo Han

Chen Gong

Robust Learning under Hybrid Noise

混合噪声下的鲁棒学习

Feature selection aims to identify the optimal feature subset for enhancing downstream models. Effective feature selection can remove redundant features, save computational resources, accelerate the model learning process, and improve the model overall performance. However, existing works are often time-intensive to identify the effective feature subset within high-dimensional feature spaces. Meanwhile, these methods mainly utilize a single downstream task performance as the selection criterion, leading to the selected subsets that are not only redundant but also lack generalizability. To bridge these gaps, we reformulate feature selection through a neuro-symbolic lens and introduce a novel generative framework aimed at identifying short and effective feature subsets. More specifically, we found that feature ID tokens of the selected subset can be formulated as symbols to reflect the intricate correlations among features. Thus, in this framework, we first create a data collector to automatically collect numerous feature selection samples consisting of feature ID tokens, model performance, and the measurement of feature subset redundancy. Building on the collected data, an encoder-decoder-evaluator learning paradigm is developed to preserve the intelligence of feature selection into a continuous embedding space for efficient search. Within the learned embedding space, we leverage a multi-gradient search algorithm to find more robust and generalized embeddings with the objective of improving model performance and reducing feature subset redundancy. These embeddings are then utilized to reconstruct the feature ID tokens for executing the final feature selection. Ultimately, comprehensive experiments and case studies are conducted to validate the effectiveness of the proposed framework. The associated data and code are publicly available 1 .

Nanxu Gong

Wangyang Ying

Dongjie Wang

Yanjie Fu

Neuro-Symbolic Embedding for Short and Effective Feature Selection via Autoregressive Generation

神经符号嵌入用于通过自回归生成实现短小有效的特征选择

Large language models (LLMs) are sophisticated artificial intelligence systems designed to process and understand natural language at a complex level. The recent progress of these models, culminating in chat-based LLMs, has democratized the accessibility of these sophisticated intelligent systems, showcasing how machine learning methods can help humans in daily tasks. This research addresses the growing interest in understanding the mechanisms of LLMs and in evaluating their alignment with human cognition. We introduce an innovative alignment assessment strategy in the realm of LLMs that diverges from traditional approaches, utilizing the odd-one-out triplet-based task to investigate the alignment of LLMs’ representations with human object concept mental organization. Our methodology, which incorporates image captioning and zero/few-shot learning accuracy scoring, is designed to evaluate language models’ ability to predict similarities and differences in object concepts. A comprehensive experimental evaluation was conducted, involving four captioning strategies, twenty-four LLMs across eight model families, and three scoring procedures, utilizing a significantly large dataset for enhanced understanding of LLM comprehensibility. Finally, our study explores the impact of object description comprehensiveness on model-human representation alignment and analyzes a subset of randomly selected triplets to assess how LLMs are able to represent different levels of human judgment patterns.

Anastasiia Hrytsyna

Rodrigo Alves

From Representation to Response: Assessing the Alignment of Large Language Models with Human Judgment Patterns

从表达到回应：评估大型语言模型与人类判断模式的一致性

Conventional biomedical image segmentation heavily relies on substantial annotations, which demand significant human and financial resources for collection. Consequently, learning a model with excellent performance using limited medical image data becomes a challenging problem. Upliftingly, the advent of few-shot medical image segmentation (FSMIS) offers a potential solution. Although prototypical networks are commonly employed in existing FSMIS tasks, the prototypes derived from support features often induce significant bias issues caused by intra-class variations. To this end, we propose a method called Cascaded Altering Refinement Transformer (CART) to iteratively calibrate the prototypes with both support and query features. This method focuses on capturing the commonality between foreground information of the support and query features using the Alterating Refinement Transformer (ART) module, which includes two Multi-head Cross Attention (MCA) modules. Furthermore, we cascade ART modules to refine the class prototypes, resulting in representative prototypes. This process ultimately contributes to a more accurate predicted mask. Besides, to preserve more valid information in each cascaded ART module and achieve better performance, we propose a novel inference method that accumulates the predicted segmentation masks in all ART modules by applying the Rounding-Up strategy. Extensive experiments on three public medical image datasets demonstrate that our model outperforms the state-of-the-art methods, and detailed analysis also validates the reasonableness of this design. Code is available at: https://github.com/zmcheng9/CART .

Ziming Cheng

Yazhou Zhu

Shidong Wang

Tong Xin

Haofeng Zhang

Cascaded Alternating Refinement Transformer for Few-shot Medical Image Segmentation

用于少样本医学图像分割的级联交替精炼Transformer

Abbreviation	ACM Trans. Intell. Syst. Technol.
Journal Impact	7.08
Quartiles(Global)	COMPUTER SCIENCE, INFORMATION SYSTEMS(Q1)

ISSN	2157-6904, 2157-6912
h-index	78

Publication Information	Publisher: Association for Computing Machinery (ACM)，Publishing cycle: ，Journal Type: journal，Open Access Journals: No
Basic data	Year of publication: 2010，Proportion of original research papers: 99.00%，Self Citation Rate:4.20%， Gold OA Rate: 1.69%

ACM Transactions on Intelligent Systems and Technology

Journal Citation Format

A journal article with 1 author

A journal article with 2 authors

A journal article with 3 authors

A journal article with 5 or more authors

Books Citation Format

Thesis Citation Format

Web sites Citation Format

Patent Citation Format

Staying up late manually editing references? ivySCI automatically matches journals and helps you generate references with a single click.

Share Submission Experience