ACM Transactions on Office Information Systems

Recommender models are hard to evaluate, particularly under offline setting. In this article, we provide a comprehensive and critical analysis of the data leakage issue in recommender system offline evaluation. Data leakage is caused by not observing global timeline in evaluating recommenders e.g., train/test data split does not follow global timeline. As a result, a model learns from the user-item interactions that are not expected to be available at the prediction time. We first show the temporal dynamics of user-item interactions along global timeline, then explain why data leakage exists for collaborative filtering models. Through carefully designed experiments, we show that all models indeed recommend future items that are not available at the time point of a test instance, as the result of data leakage. The experiments are conducted with four widely used baseline models—BPR, NeuMF, SASRec, and LightGCN, on four popular offline datasets—MovieLens-25M, Yelp, Amazon-music, and Amazon-electronic, adopting leave-last-one-out data split. 1 We further show that data leakage does impact models’ recommendation accuracy. Their relative performance orders thus become unpredictable with different amount of leaked future data in training. To evaluate recommendation systems in a realistic manner in offline setting, we propose a timeline scheme, which calls for a revisit of the recommendation model design.

Yitong Ji

Aixin Sun

Jie Zhang

Chenliang Li

ACM transactions on office information systems

ACM Trans. Inf. Syst.

A Critical Study on Data Leakage in Recommender System Offline Evaluation

Yuanjie Lyu

Zhiyu Li

Simin Niu

Feiyu Xiong

Bo Tang

Wenjin Wang

Hao Wu

Huanyong Liu

Tong Xu

Enhong Chen

CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models

CRUD-RAG：用于大语言模型检索增强生成的综合中文基准

Multimodal information (e.g., visual, acoustic, and textual) has been widely used to enhance representation learning for micro-video recommendation. For integrating multimodal information into a joint representation of micro-video, multimodal fusion plays a vital role in the existing micro-video recommendation approaches. However, the static multimodal fusion used in previous studies is insufficient to model the various relationships among multimodal information of different micro-videos. In this article, we develop a novel meta-learning-based multimodal fusion framework called Meta Multimodal Fusion (MetaMMF), which dynamically assigns parameters to the multimodal fusion function for each micro-video during its representation learning. Specifically, MetaMMF regards the multimodal fusion of each micro-video as an independent task. Based on the meta information extracted from the multimodal features of the input task, MetaMMF parameterizes a neural network as the item-specific fusion function via a meta learner. We perform extensive experiments on three benchmark datasets, demonstrating the significant improvements over several state-of-the-art multimodal recommendation models, like MMGCN, LATTICE, and InvRL. Furthermore, we lighten our model by adopting canonical polyadic decomposition to improve the training efficiency, and validate its effectiveness through experimental results. Codes are available at https://github.com/hanliu95/MetaMMF .

Han Liu

Yinwei Wei

Fan Liu

Wenjie Wang

Liqiang Nie

Tat-Seng Chua

Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation

基于元学习的动态多模态融合微视频推荐

Knowledge Tracing (KT) is the task of modeling a learner’s knowledge state to predict future performance in e-learning systems based on past performance. Deep learning-based methods, such as recurrent neural networks, memory-augmented neural networks, and attention-based neural networks, have recently been used in KT. Such methods have demonstrated excellent performance in capturing the latent dependencies of a learner’s knowledge state on recent exercises. However, these methods have limitations when it comes to dealing with the so-called Skill Switching Phenomenon (SSP), i.e., when learners respond to exercises in an e-learning system, the latent skills in the exercises typically switch irregularly. SSP will deteriorate the performance of deep learning-based approaches for simulating the learner’s knowledge state during skill switching, particularly when the association between the switching skills and the previously learned skills is weak. To address this problem, we propose the Memory-augmented Attentive Network (MAN), which combines the advantages of memory-augmented neural networks and attention-based neural networks. Specifically, in MAN, memory-augmented neural networks are used to model learners’ longer term memory knowledge, while attention-based neural networks are used to model learners’ recent term knowledge. In addition, we design a context-aware attention mechanism that automatically weighs the tradeoff between these two types of knowledge. With extensive experiments on several e-learning datasets, we show that MAN effectively improve predictive accuracies of existing state-of-the-art DLKT methods.

Liangliang He

Xiao Li

Pancheng Wang

Jintao Tang

Ting Wang

MAN: Memory-augmented Attentive Networks for Deep Learning-based Knowledge Tracing

MAN：基于深度学习的知识追踪的记忆增强注意力网络

A valuable small subset strategically selected from massive online reviews is beneficial to improve consumers’ decision-making efficiency in e-commerce. Existing review selection methods primarily concentrate on the informativeness of reviews and aim to find a subset of reviews that can reflect the informational properties of the original review set. However, changes in consumers’ review diets during the two-phase decision process are not fully considered. In this study, we propose a novel review selection problem of finding a diet-matched review subset with high diversity and representativeness, which can better adapt to consumers’ review-diet conversion from attribute-oriented to experience-oriented reviews between two decision phases. A novel decision-phase-based review selection method named DPRS is further proposed, which involves two steps: review classification and review selection. In the review classification step, the probability of a review being attribute-oriented or experience-oriented is estimated by prior knowledge-aware attentive neural network. In the second step, a novel heuristic algorithm, namely, stepwise non-dominated selection with superiority strategy, is introduced to seek the solution to the review selection problem. Extensive experiments on a real-world dataset demonstrate that DPRS outperforms state-of-the-art methods in terms of both review classification and review selection.

Jin Zhang

Xinrui Li

Liye Wang

A Review Selection Method Based on Consumer Decision Phases in E-commerce

基于电子商务消费者决策阶段的文献综述选择方法

Large Language Models (LLMs) have recently garnered significant attention in various domains, including recommendation systems. Recent research leverages the capabilities of LLMs to improve the performance and user modeling aspects of recommender systems. These studies primarily focus on utilizing LLMs to interpret textual data in recommendation tasks. However, it's worth noting that in ID-based recommendations, textual data is absent, and only ID data is available. The untapped potential of LLMs for ID data within the ID-based recommendation paradigm remains relatively unexplored. To this end, we introduce a pioneering approach called “LLM for ID-based Recommendation” (LLM4IDRec). This innovative approach integrates the capabilities of LLMs while exclusively relying on ID data, thus diverging from the previous reliance on textual data. The basic idea of LLM4IDRec is that by employing LLM to augment ID data, if augmented ID data can improve recommendation performance, it demonstrates the ability of LLM to interpret ID data effectively, exploring an innovative way for the integration of LLM in ID-based recommendation. Specifically, we first define a prompt template to enhance LLM's ability to comprehend ID data and the ID-based recommendation task. Next, during the process of generating training data using this prompt template, we develop two efficient methods to capture both the local and global structure of ID data. We feed this generated training data into the LLM and employ LoRA for fine-tuning LLM. Following the fine-tuning phase, we utilize the fine-tuned LLM to generate ID data that aligns with users’ preferences. We design two filtering strategies to eliminate invalid generated data. Thirdly, we can merge the original ID data with the generated ID data, creating augmented data. Finally, we input this augmented data into the existing ID-based recommendation models without any modifications to the recommendation model itself. We evaluate the effectiveness of our LLM4IDRec approach using three widely-used datasets. Our results demonstrate a notable improvement in recommendation performance, with our approach consistently outperforming existing methods in ID-based recommendation by solely augmenting input data.

Lei Chen

Chen Gao

Xiaoyi Du

Hengliang Luo

Depeng Jin

Yong Li

Meng Wang

Enhancing ID-based Recommendation with Large Language Models

基于ID的推荐系统在大语言模型中的增强

With the growing privacy concerns in recommender systems, recommendation unlearning is getting increasing attention. Existing studies predominantly use training data, i.e., model inputs, as unlearning target. However, attackers can extract private information from the model even if it has not been explicitly encountered during training. We name this unseen information as attribute and treat it as unlearning target. To protect the sensitive attribute of users, Attribute Unlearning (AU) aims to make target attributes indistinguishable. In this paper, we focus on a strict but practical setting of AU, namely Post-Training Attribute Unlearning (PoT-AU), where unlearning can only be performed after the training of the recommendation model is completed. To address the PoT-AU problem in recommender systems, we propose a two-component loss function. The first component is distinguishability loss, where we design a distribution-based measurement to make attribute labels indistinguishable from attackers. We further extend this measurement to handle multi-class attribute cases with efficient computational overhead. The second component is regularization loss, where we explore a function-space measurement that effectively maintains recommendation performance compared to parameter-space regularization. We use stochastic gradient descent algorithm to optimize our proposed loss. Extensive experiments on four real-world datasets demonstrate the effectiveness of our proposed methods.

Chaochao Chen

Yizhao Zhang

Yuyuan Li

Jun Wang

Lianyong Qi

Xiaolong Xu

Xiaolin Zheng

Jianwei Yin

Post-Training Attribute Unlearning in Recommender Systems

Marialena Bevilacqua

Kezia Oketch

Ruiyang Qin

Will Stamey

Xinyuan Zhang

Yi Gan

Kai Yang

Ahmed Abbasi

When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTs

当自动化评估遇上自动化内容生成：GPT时代文本质量的审视

Recommendations for events play a pivotal role in facilitating the discovery of upcoming intriguing events within Event-Based Social Networks (EBSNs). Previous research has established the crucial significance of mining contextual features and implicit relationships to enhance recommendation performance and alleviate data sparsity issues. However, the noise inherent in contextual features exacerbates data sparsity and hampers the ability of previous methods to explore implicit relationships for mitigating data sparsity. To address this challenge, we propose a variational type graph autoencoder model that attenuates the influence of noise in different types of context features by introducing type-specific latent variables. Firstly, we introduce a heterogeneous denoising convolution module composed of two components: 1) Denoising attention aggregation is proposed to mitigate the influence of noisy structures and uncover implicit relationships. 2) A heterogeneous normalization module leverages context features within the same type to alleviate the effects of noise in context features and data sparsity. Furthermore, we propose a learnable heterogeneous mixture prior that assists in assigning different priors to distinct types of latent variables, effectively modeling different types of contextual features. Through comprehensive experiments conducted on real-world datasets, we demonstrate the compelling performance of our model compared to state-of-the-art competitive approaches.

Shuo Zhang

Xiangwu Meng

Yujie Zhang

Variational Type Graph Autoencoder For Denoising On Event Recommendation

用于事件推荐去噪的变分类型图自编码器

Sequential recommendation, leveraging user-item interaction histories to provide personalized and timely suggestions, has drawn significant research interest recently. With the power of exploiting spatio-temporal dynamics, dynamic graph neural networks (DyGNNs) show great potential in sequential recommendation by modeling the dynamic relationship between users and items. However, spatio-temporal distribution shifts naturally exist in out-of-distribution sequential recommendation, where both user-item relationships and temporal sequences demonstrate pattern shifts. The out-of-distribution scenarios may lead to the failure of existing DyGNNs in handling spatio-temporal distribution shifts in sequential recommendation, given that the patterns they exploit tend to be variant w.r.t labels under distribution shifts. In this paper, we propose D isentangled I ntervention-based D ynamic graph A ttention networks with I nvariance Promotion ( I-DIDA ) to handle spatio-temporal distribution shifts in sequential recommendation by discovering and utilizing invariant patterns, ie , structures and features whose predictive abilities are stable across distribution shifts. Specifically, we first propose a disentangled spatio-temporal attention network to capture the variant and invariant patterns. By utilizing the disentangled patterns, we design a spatio-temporal intervention mechanism to create multiple interventional distributions and an environment inference module to infer the latent spatio-temporal environments, and minimize the invariance loss to leverage the invariant patterns with stable predictive abilities under distribution shifts. Extensive experiments demonstrate the superiority of our method over state-of-the-art sequential recommendation baselines under distribution shifts.

Zeyang Zhang

Xin Wang

Haibo Chen

Haoyang Li

Wenwu Zhu

Disentangled Dynamic Graph Attention Network for Out-of-distribution Sequential Recommendation

解耦动态图注意力网络用于分布外序列推荐

出版信息	出版商: Association for Computing Machinery (ACM)，出版周期: ，期刊类型: journal，开源期刊: 否
基本数据	创刊年份: 1986，原创研究文献占比: ，自引率:， Gold OA占比:

ACM Transactions on Office Information Systems

期刊引文格式

有 1 位以上作者的期刊

有 2 位作者的期刊

有 3 位作者的期刊

有 5 位以上作者的期刊

书籍引用格式

学位论文引用格式

网页引用格式

专利引用格式

手工熬夜修改参考文献？研飞自动匹配期刊，一键轻松成稿，支持 Word/WPS

投稿经验分享