ACM TRANSACTIONS ON MULTIMEDIA COMPUTING, COMMUNICATIONS, AND APPLICATIONS 是 ACM 多媒体特别兴趣小组 (SIGMM) 的旗舰出版物。它正在征集有关多媒体各个方面的论文提交。也欢迎有关单一媒体（例如，音频、视频、动画）及其处理的论文。TOMM 是一份经过同行评审的档案期刊，有印刷和数字两种形式。该杂志每季度出版一次；每期大约有 7 篇 23 页的文章。此外，所有特刊均仅在线发布，以确保及时发布。交易主要包括研究论文。这是一份档案期刊，旨在使论文随着时间的推移具有持久的重要性和价值。一般而言，不包括主要关注特定多媒体产品或行业现状的论文。

ACM Transactions on Multimedia Computing, Communications and Applications is the flagship publication of the ACM Special Interest Group on Multimedia (SIGMM). It invites submissions on all aspects of multimedia, as well as papers focused on individual media (e.g., audio, video, animation) and their processing. TOMM is a peer-reviewed archival journal available in both print and digital formats, published quarterly with approximately 7 articles per issue, totaling around 23 pages. Additionally, all special issues are published online only to ensure timely dissemination. The journal primarily includes research papers and aims to maintain the enduring significance and value of the papers over time. Generally, papers focusing primarily on specific multimedia products or the current state of the industry are not included.

《ACM Transactions on Multimedia Computing, Communications and Applications》是ACM多媒体特别兴趣小组（SIGMM）的旗舰期刊，欢迎提交关于多媒体各个方面的论文。我们也接受关于单一媒体（如音频、视频、动画）及其处理的研究。该期刊经过同行评审，提供印刷和数字两种形式，每季度出版一次，每期约有7篇、23页的文章。此外，所有特刊均仅在线发布，以确保及时性。期刊主要发表研究论文，旨在使论文在时间的推移中保持持久的重要性和价值。一般而言，不包括主要关注特定多媒体产品或行业现状的论文。

ACM Transactions on Multimedia Computing, Communications and Applications

As training a high-performance deep neural network (DNN) model requires a large amount of data, powerful computing resources and expert knowledge, protecting well-trained DNN models from Intellectual Property (IP) infringement has raised serious concerns in recent years. Most existing methods using DNN watermarks to verify the ownership of the models after IP infringement occurs, which is reactive in the sense that they cannot prevent unauthorized users from using the model in the first place. Different from these methods, in this paper, we propose an active authorization control and user’s fingerprint tracking method for the IP protection of DNN models by utilizing sample-specific backdoor attack. The proposed method inversely and multiplely exploits sample-specific trigger as the key to implement authorization control for DNN model, in which the generated triggers are imperceptible and sample-specific for clean images. Specifically, a U-Net model is used to generate backdoor instances. Then, the target model is trained on the clean images and backdoor instances, which are inversely labelled as wrong classes and correct classes, respectively. Only authorized users can use the target model normally by pre-processing the clean images through the U-Net model. Moreover, the images processed by the U-Net model will contain unique fingerprint that can be extracted to verify and track the corresponding user’s identity. This paper is the first work that utilizes the sample-specific backdoor attack to implement active authorization control and user’s fingerprint management for DNN model under black-box scenarios. Extensive experimental results on ImageNet dataset and YouTube Aligned Face dataset demonstrate that the proposed method is effective in protecting the DNN model from unauthorized usage. Specifically, the protected model has a low inference accuracy (1.00%) for unauthorized users, while maintaining a normal inference accuracy (97.67%) for authorized users. Besides, the proposed method can achieve 100% fingerprint tracking success rates on both the ImageNet and YouTube Aligned Face datasets. Moreover, it is demonstrated that the proposed method is robust against fine-tuning attack, pruning attack, pruning attack with retraining, reverse-engineering attack, adaptive attack, and JPEG compression attack. The code is available at https://github.com/nuaaaisec/SSAT .

Mingfu Xue

Yinghao Wu

Leo Yu Zhang

Dujuan Gu

Yushu Zhang

Weiqiang Liu

ACM Transactions on Multimedia Computing Communications and Applications

SSAT: Active Authorization Control and User’s Fingerprint Tracking Framework for DNN IP Protection

SSAT：DNN IP保护的主动授权控制与用户指纹追踪框架

Text-based person re-identification aims to find the target person from a large pedestrian gallery with the given natural language description. Previous works mainly focus on embedding salient textual and visual representations in a common latent space by utilizing the dual-path structure or parameter-shared network. However, they still lack the ability to effectively extract fine-grained unimodal features as well as fuse the cross-modal data, leading to the increase of misaligned cases. To settle these issues, we propose a text-and-image implicit learning Transformer (TILT) to eliminate textual anisotropy and enhance the cross-modal alignment from both domains based on the bi-direction multimodal encoders. Specifically, we apply the pre-trained multimodal embedding module to overcome the unimodal anisotropy problem with contrastive learning, and map fine-grained features with dual encoder in bidirectional masking. Then, we design the cross-modal interaction encoder to comprehensively mine implicit cross-modal relations by reconstructing masked tokens, and fuse rich multi-modal knowledge in a common space. In addition, the cross-modal similarity matching module is proposed to optimize the intra-domain classification and decrease the inter-domain divergence. Extensive experiments are conducted on three public benchmarks CUHK-PEDES, ICFG-PEDES and RSTPReid to verify the effectiveness of our proposed framework. And results prove that our model outperforms state-of-the-art methods on all metrics.

Tinghui Wu

Shuhe Zhang

Dihu Chen

Haifeng Hu

Text-and-Image Learning Transformer for Cross-modal Person Re-identification

文本与图像学习变换器用于跨模态行人重识别

In this article, we focus on fake news detection task and aim to automatically identify the fake news from vast amount of social media posts. To date, many approaches have been proposed to detect fake news, which includes traditional learning methods and deep learning-based models. However, there are three existing challenges: (i) How to represent social media posts effectively, since the post content is various and highly complicated; (ii) how to propose a data-driven method to increase the flexibility of the model to deal with the samples in different contexts and news backgrounds; and (iii) how to fully utilize the additional auxiliary information (the background knowledge and multi-modal information) of posts for better representation learning. To tackle the above challenges, we propose a novel Knowledge-aware Multi-modal Adaptive Graph Convolutional Networks (KMAGCN) to capture the semantic representations by jointly modeling the textual information, knowledge concepts, and visual information into a unified framework for fake news detection. We model posts as graphs and use a knowledge-aware multi-modal adaptive graph learning principal for the effective feature learning. Compared with existing methods, the proposed KMAGCN addresses challenges from three aspects: (1) It models posts as graphs to capture the non-consecutive and long-range semantic relations; (2) it proposes a novel adaptive graph convolutional network to handle the variability of graph data; and (3) it leverages textual information, knowledge concepts and visual information jointly for model learning. We have conducted extensive experiments on three public real-world datasets and superior results demonstrate the effectiveness of KMAGCN compared with other state-of-the-art algorithms.

Shengsheng Qian

Jun Hu

Quan Fang

Changsheng Xu

ACM Trans. Multimedia Comput. Commun. Appl.

Knowledge-aware Multi-modal Adaptive Graph Convolutional Networks for Fake News Detection

知识感知多模态自适应图卷积网络用于虚假新闻检测

Skeleton-based action recognition is beneficial for understanding human behavior in videos, and thus has received much attention in recent years as an important research area in action recognition. Current research focuses on designing more advanced algorithms to better extract spatio-temporal information from skeleton data. However, due to the small amount of data in the existing skeleton dataset and the lack of effective data augmentation methods, it is easy to lead to overfitting in model training. To address this challenge, we propose a mix-based data augmentation method, Joint Mixing Data Augmentation (JMDA), which can generally improve the effectiveness and robustness of various skeleton-based action recognition algorithms. In terms of spatial information, we introduce SpatialMix (SM), a method that projects the original 3D skeleton discrete information into a 2D space. Then, SM mixes the projected spatial information between two random samples during the training process to achieve the spatial-based mixing data augmentation. Concerning temporal information, we propose TemporalMix (TM). Leveraging the temporal continuity in skeleton data, we perform a temporal resize operation on the original skeleton data, and then merge two random samples during training to achieve the temporal-based mixed data augmentation. Additionally, we analyze the Feature Mismatch (FM) problem caused by introducing mix-based data augmentation into skeleton data. Then we propose a new data preprocessing method called Feature Alignment (FA) to effectively address this problem and improve model performance. Moreover, we propose a novel training pipeline, Joint Training Strategy (JTS), which combines multiple mix-based data augmentation methods for further improvement of model performance. Specifically, our proposed JMDA is plug-and-play and widely applicable to skeleton-based action recognition models. At the same time, the application of JMDA does not increase the model parameters and there is almost no additional training cost. We conduct extensive experiments on NTU RGB+D 60 and NTU RGB+D 120 datasets to demonstrate the effectiveness and robustness of the proposed JMDA on several mainstream skeleton-based action recognition algorithms.

Linhua Xiang

Zengfu Wang

Joint Mixing Data Augmentation for Skeleton-based Action Recognition

基于骨架的动作识别联合混合数据增强方法

Smart healthcare systems are generating a large scale of heterogenous high-dimensional data with complex relationships. It is hard for current methods to analyze such high-dimensional healthcare data. Specifically, the traditional data reduction methods can not keep the correlation among different modalities of data objects, while the latest methods based on tensor singular value decomposition are not effective for data reduction, although they can keep the correlation. This article presents a tensor train-tensor singular value decomposition (TT-TSVD) algorithm for data reduction. Particularly, the presented algorithm balances the correlation-preservation ability of modalities and data reduction ability by combining the advantages of the train structure of the tensor train decomposition and the association relationship between the tensor singular value decomposition retention mode. Extensive experiments are conducted on the convolutional neural network and the results clearly show that the presented algorithm performs effectively for data reduction with a low-loss classification accuracy; what is more, classification accuracy on medical image dataset has been improved a little.

Debin Liu

Laurence T. Yang

Puming Wang

Ruonan Zhao

Qingchen Zhang

TT-TSVD: A Multi-modal Tensor Train Decomposition with Its Application in Convolutional Neural Networks for Smart Healthcare

TT-TSVD：一种多模态张量列车分解方法及其在智能医疗卷积神经网络中的应用

Multi-label recognition with partial labels (MLR-PL), in which only some labels are known while others are unknown for each image, is a practical task in computer vision, since collecting large-scale and complete multi-label datasets is difficult in real application scenarios. Recently, vision language models (e.g. CLIP) have demonstrated impressive transferability to downstream tasks in data limited or label limited settings. However, current CLIP-based methods suffer from semantic confusion in MLR task due to the lack of fine-grained information in the single global visual and textual representation for all categories. In this work, we address this problem by introducing a semantic decoupling module and a category-specific prompt optimization method in CLIP-based framework. Specifically, the semantic decoupling module following the visual encoder learns category-specific feature maps by utilizing the semantic-guided spatial attention mechanism. Moreover, the category-specific prompt optimization method is introduced to learn text representations aligned with category semantics. Therefore, the prediction of each category is independent, which alleviate the semantic confusion problem. Extensive experiments on Microsoft COCO 2014 and Pascal VOC 2007 datasets demonstrate that the proposed framework significantly outperforms current state-of-art methods with a simpler model structure. Additionally, visual analysis shows that our method effectively separates information from different categories and achieves better performance compared to CLIP-based baseline method.

Haoxian Ruan

Zhihua Xu

Zhijing Yang

Yongyi Lu

Jinghui Qin

Tianshui Chen

Learning Semantic-Aware Representation in Visual-Language Models for Multi-Label Recognition with Partial Labels

学习语义感知表示在视觉-语言模型中用于带部分标签的多标签识别

Non-annotated visual description (NaVD) aims to describe generic visuals without human-annotated pairwise data. The generic visuals refer to images and videos. Existing works mainly focus on one specific visual modality, i.e. , image or video. In this paper, we propose a new framework for this task, which can directly be applied to both image and video with the pipeline unchanged. Essentially, it is a unified framework that flexibly adapts to images and videos. Recently, contrastive visual-language pre-training models (CVLPs) have experienced rapid development, demonstrating powerful abilities to align vision and language. To continuously leverage advanced CVLPs, our framework is designed to work well with general CVLPs. It can easily use image-language CVLPs for image input and switch to video-language CVLPs for video input. Specifically, we propose a CVLP-based framework for NaVD, named CVLP-NaVD. It follows the paradigm of adversarial learning, containing a generator and a discriminator. The generator takes an image or a video as input and produces a corresponding language description, while the discriminator evaluates the generated sentence for its naturalness in human-like language. Apart from the naturalness, CVLPs play a crucial role in enhancing the alignment between visual and language signals during generation. Particularly, we explore three rewarding strategies to compute the alignment score, including directly calculating cosine similarity ( i.e. , VL-cross), projecting visual embeddings into the textual domain ( i.e. , VL-project) and their combination ( i.e. , VL-mix). The three strategies are fully examined in different scenarios. Finally, we conduct extensive experiments with various unpaired and unsupervised setups in both image and video captioning tasks. The experimental results demonstrate that our CVLP-NaVD outperforms the state-of-the-art methods significantly.

Haoran Li

Yanbin Hao

Jiarui Yu

Bin Zhu

Shuo Wang

Tong Xu

CVLP-NaVD: Contrastive Visual-Language Pre-training Models for Non-annotated Visual Description

CVLP-NaVD：用于非注释视觉描述的对比视觉-语言预训练模型

Deepfake technology presents critical cybersecurity challenges that have become more popular since easily accessible applications have become more widely available. The proliferation of fake portrait videos constitutes a serious risk to the legal system, society, and personal privacy. The publication of fraudulent explicit content starring celebrities, the circulation of fake political videos, and the use of faked impersonated videos as proof in court of law are all examples of the effects of deepfakes in the real-world. In reaction to this growing threat, we propose a simple yet efficient Person of Interest PoI Siamese network-based model to detect deepfake synthetic content in portrait images, providing a preventative measure against the growing danger of deepfakes. On one side and unlike traditional neural networks, which process inputs independently, our approach leverages a Siamese network that processes two inputs simultaneously using identical sub-networks with shared weights and parameters. This twin structure is particularly effective for our adopted Person-of-Interest (PoI) methodology, where one input is a reference image of a specific individual, and the other is an image that needs to be verified as either real or fake. By ensuring that both the reference and the suspect image are processed in the same way, the network can accurately learn and detect subtle differences, enabling it to determine whether the second image is a genuine representation of the individual or a deepfake. This makes our method particularly relevant for targeted forensic investigations or security applications where an individual’s media integrity is paramount. On the other side, our proposed method does not require any additional complex biological feature extraction. Despite this simplification, our method achieves comparable accuracy to more complex models that rely on biological feature extraction. This efficiency makes our approach practical for implementation on resource-constrained devices, such as mobile phones and Internet of Things (IoT) systems.

Khouloud Samrouth

Pia El Housseini

Olivier Deforges

Siamese Network-Based Detection of DeepFake Impersonation Attacks with a Person of Interest Approach

基于孪生网络的深度伪造冒充攻击检测与目标人物方法

Scene graph generation (SGG) aims to perceive objects and their relations in images, which can bridge the gap between upstream detection tasks and downstream high-level visual understanding tasks. For scene graph generation models, over-fitting the head predicates can lead to bias in the generated scene graph, which has become a consensus. A series of debiasing methods have been proposed to solve the problem. However, some existing debiasing SGG methods have a tendency to over-fit the tail predicates, which is another type of bias. In order to eliminate the one-way over-fitting of head or tail predicates, this paper proposes a balanced relation prediction (BRP) module which is model-agnostic and compatible with existed re-balancing methods. Moreover, because the relation prediction is based on object feature representation, this paper proposes a scene adaptive context fusion (SACF) module to refine the object feature representation. Specifically, SACF models the context based on a chain structure, where the order of objects in the chain structure is adaptively arranged according to the scene content, achieving visual information fusion that adapts to the scene where the objects are located. Experiments on VG and GQA datasets show that the proposed method achieves competitive results on the comprehensive metric across R@K and mR@K.

Kai Xu

Lichun Wang

Shuang Li

Tong Gao

Baocai Yin

Scene Adaptive Context Modeling and Balanced Relation Prediction for Scene Graph Generation

场景自适应上下文建模与平衡关系预测用于场景图生成

Facing enormous data generated at the network edge, Edge Intelligence (EI) emerges as the fusion of Edge Computing and Artificial Intelligence, revolutionizing edge data processing and intelligent decision-making. Nonetheless, this emergent mode presents a complex array of security challenges, particularly prominent in image-centric applications due to the sheer volume of visual data and its direct connection to user privacy. These challenges include safeguarding model/image privacy and ensuring model integrity against various security threats, such as model poisoning. Essentially, those threats originate from data attacks, suggesting data protection as a promising solution. Although data protection measures are well-established in other domains, image-centric EI necessitates focused research. This survey examines the security issues inherent to image-centric EI and outlines the protection efforts, providing a comprehensive overview of the landscape. We begin by introducing EI, detailing its operational mechanics and associated security issues. We then explore the technologies facilitating security enhancement (e.g., differential privacy) and edge intelligence (e.g., compact networks and distributed learning frameworks). Next, we categorize security strategies by their application in data preparation, training, and inference, with a focus on image-based contexts. Despite these efforts on security, our investigation identifies research gaps. We also outline promising research directions to bridge these gaps, bolstering security frameworks in image-centric EI applications.

Abbreviation
Journal Impact	5.35
Quartiles(Global)	COMPUTER SCIENCE, THEORY & METHODS(Q1)

ISSN	1551-6857, 1551-6865
h-index	63

Publication Information	Publisher: Association for Computing Machinery (ACM)，Publishing cycle: Quarterly，Journal Type: journal，Open Access Journals: No
Basic data	Year of publication: 2005，Proportion of original research papers: 98.61%，Self Citation Rate:13.50%， Gold OA Rate: 0.57%
Average review cycle	网友分享经验：>12周，或约稿
Average recruitment ratio	网友分享经验：容易

ACM Transactions on Multimedia Computing, Communications and Applications

Journal Citation Format

A journal article with 1 author

A journal article with 2 authors

A journal article with 3 authors

A journal article with 5 or more authors

Books Citation Format

Thesis Citation Format

Web sites Citation Format

Patent Citation Format

Staying up late manually editing references? ivySCI automatically matches journals and helps you generate references with a single click.

Share Submission Experience