期刊的范围是多媒体技术和多媒体应用研究的各个方面，包括但不限于电路、网络、信号处理、系统、软件和系统集成，以“感兴趣的领域”为代表。赞助商。

The IEEE Transactions on Multimedia journal covers various aspects of multimedia technology and application research, including but not limited to circuits, networks, signal processing, systems, software, and system integration. The journal is sponsored by stakeholders in the field.

《IEEE Transactions on Multimedia》期刊涵盖多媒体技术及其应用研究的各个方面，包括但不限于电路、网络、信号处理、系统、软件和系统集成等领域。该期刊由相关领域的赞助商支持。

IEEE Transactions on Multimedia

Named Entity Recognition (NER) in social media posts is challenging since texts are usually short and contexts are lacking. Most recent works show that visual information can boost the NER performance since images can provide complementary contextual information for texts. However, the image-level features ignore the mapping relations between fine-grained visual objects and textual entities, which results in error detection in entities with different types. To better exploit visual and textual information in NER, we propose an adversarial gated bilinear attention neural network (AGBAN). The model jointly extracts entity-related features from both visual objects and texts, and leverages an adversarial training to map two different representations into a shared representation. As a result, domain information contained in an image can be transferred and applied for extracting named entities in the text associated with the image. Experimental results on Tweets dataset demonstrate that our model outperforms the state-of-the-art methods. Moreover, we systematically evaluate the effectiveness of the proposed gated bilinear attention network in capturing the interactions of mutimodal features visual objects and textual words. Our results indicate that the adversarial training can effectively exploit commonalities across heterogeneous data sources, which leads to improved performance in NER when compared to models purely exploiting text data or combining the image-level visual features.

Changmeng Zheng

Zhiwei Wu

Tao Wang

Yi Cai

Qing Li

IEEE Trans. Multimedia

Object-Aware Multimodal Named Entity Recognition in Social Media Posts With Adversarial Learning

面向对象的多模态社交媒体帖子命名实体识别与对抗学习

Image-text retrieval (ITR) is a challenging task in the field of multimodal information processing due to the semantic gap between different modalities. In recent years, researchers have made great progress in exploring the accurate alignment between image and text. However, existing works mainly focus on the fine-grained alignment between image regions and sentence fragments, which ignores the guiding significance of context background information. Actually, integrating the local fine-grained information and global context background information can provide more semantic clues for retrieval. In this paper, we propose a novel Hierarchical Graph Alignment Network (HGAN) for image-text retrieval. First, to capture the comprehensive multimodal features, we construct the feature graphs for the image and text modality respectively. Then, a multi-granularity shared space is established with a designed Multi-granularity Feature Aggregation and Rearrangement (MFAR) module, which enhances the semantic corresponding relations between the local and global information, and obtains more accurate feature representations for the image and text modalities. Finally, the ultimate image and text features are further refined through three-level similarity functions to achieve the hierarchical alignment. To justify the proposed model, we perform extensive experiments on MS-COCO and Flickr30 K datasets. Experimental results show that the proposed HGAN outperforms the state-of-the-art methods on both datasets, which demonstrates the effectiveness and superiority of our model.

Jie Guo

Meiting Wang

Yan Zhou

Bin Song

Yuhao Chi

Wei Fan

Jianglong Chang

HGAN: Hierarchical Graph Alignment Network for Image-Text Retrieval

In this paper, we study the problem of human action recognition from multiple feature modalities. We propose bimodal hybrid centroid canonical correlation analysis (BHCCCA) and multimodal hybrid centroid canonical correlation analysis (MHCCCA) to learn the discriminative and informative shared space, by considering the correlation among different classes across two modalities (BHCCCA) and three or more modalities (MHCCCA). We then introduce a new human action recognition framework by using BHCCCA/MHCCCA for fusing different modalities (RGB, depth, skeleton, and accelerometer data). Performance evaluation on four publicly accessible data sets (MSR Action3D, UTD-MHAD, UTD-MHAD-Kinect V2, and Berkeley MHAD) demonstrated the effectiveness of the proposed framework.

Nour El Din Elmadany

Yifeng He

Ling Guan

Multimodal Learning for Human Action Recognition Via Bimodal/Multimodal Hybrid Centroid Canonical Correlation Analysis

Emotion Recognition in Conversation (ERC) plays a significant part in Human-Computer Interaction (HCI) systems since it can provide empathetic services. Multimodal ERC can mitigate the drawbacks of uni-modal approaches. Recently, Graph Neural Networks (GNNs) have been widely used in a variety of fields due to their superior performance in relation modeling. In multimodal ERC, GNNs are capable of extracting both long-distance contextual information and inter-modal interactive information. Unfortunately, since existing methods such as MMGCN directly fuse multiple modalities, redundant information may be generated and diverse information may be lost. In this work, we present a directed Graph based Cross-modal Feature Complementation (GraphCFC) module that can efficiently model contextual and interactive information. GraphCFC alleviates the problem of heterogeneity gap in multimodal fusion by utilizing multiple subspace extractors and Pair-wise Cross-modal Complementary (PairCC) strategy. We extract various types of edges from the constructed graph for encoding, thus enabling GNNs to extract crucial contextual and interactive information more accurately when performing message passing. Furthermore, we design a GNN structure called GAT-MLP, which can provide a new unified network framework for multimodal learning. The experimental results on two benchmark datasets show that our GraphCFC outperforms the state-of-the-art (SOTA) approaches.

Jiang Li

Xiaoping Wang

Guoqing Lv

Zhigang Zeng

GraphCFC: A Directed Graph Based Cross-Modal Feature Complementation Approach for Multimodal Conversational Emotion Recognition

Image stitching aims at generating high-quality panoramas with the lowest computational cost. In this paper, we propose a parallax-tolerant image stitching method based on robust elastic warping, which could achieve accurate alignment and efficient processing simultaneously. Given a group of point matches between images, an analytical warping function is constructed to eliminate the parallax errors. Then, the input images are warped according to the computed deformations over the meshed image plane. The seamless panorama is composed by directly reprojecting the warped images. As an important complement to the proposed method, a Bayesian model of feature refinement is proposed to adaptively remove the incorrect local matches. This ensures a more robust alignment than existing approaches. Moreover, our warp is highly compatible with different transformation types. A flexible strategy of combining it with the global similarity transformation is provided as an example. The performance of the proposed approach is demonstrated using several challenging cases.

Jing Li

Zhengming Wang

Shiming Lai

Yongping Zhai

Maojun Zhang

Parallax-Tolerant Image Stitching Based on Robust Elastic Warping

基于鲁棒弹性变形的视差容忍图像拼接

Lingru Zhou

Yiqi Gao

Manqing Zhang

Peng Wu

Peng Wang

Yanning Zhang

Human-centric Behavior Description in Videos: New Benchmark and Model

以人为中心的行为描述：新基准与模型

In this paper, we propose a novel optical flow guided network with progressive frequencies learning, achieving promising dynamic multi-exposure image fusion. Specifically, the proposed method consists of the optical flow alignment block and the progressive frequencies fusion block, where the former is to alleviate the ghost caused by the camera and object motions, and the latter is dedicated to synthesizing the desired color and details. First, in the optical flow alignment block, we estimate the optical flow between two source images and utilize the deformable convolutional network to achieve spatial alignment guided by the estimated optical flow. Second, in the progressive frequencies fusion block, color correction and details preservation are implemented in two gradual phases, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , low and high frequencies. For the low frequency fusion phase, we combine the convolutional neural network and swin transformer to capture local and global features, so as to consider the color correction from a complete perspective. For the high frequency fusion phase, an attention gate is designed to evaluate the important details from source images, bringing fewer artifacts and halos. Finally, the low and high frequency fusion phases are connected through a residual mapping strategy to generate a desired image with reasonable colors and rich details. Extensive experiments on publicly available datasets reveal that our method outperforms the state-of-the-art for both static and dynamic scenarios. Moreover, our method is superior in running efficiency over most of the state-of-the-art methods.

Wenhui Hong

Hao Zhang

Jiayi Ma

OFPF-MEF: An Optical Flow Guided Dynamic Multi-Exposure Image Fusion Network With Progressive Frequencies Learning

OFPF-MEF：一种基于光流引导的动态多曝光图像融合网络，具有渐进频率学习功能

We present an automatic generation pipeline of interactive nonlinear video for online apparel shopping navigation. Our approach was inspired by Google's "Messy Middle" theory, which suggests that people mentally are faced with two tasks—exploration and evaluation—before purchasing online. Given a set of apparel product presentation videos, our navigation UI organizes them to optimize users' product exploration and automatically generates interactive videos for users' product evaluation. To support automatic methods, we proposed a video clustering similarity ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\operatorname{CSIM}$</tex-math></inline-formula> ) and a camera movement similarity ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\operatorname{MSIM}$</tex-math></inline-formula> ), as well as a comparative video generation algorithm for product recommendation, presentation, and comparison. To evaluate our pipeline's effectiveness, we conducted several user studies. The results showed that our pipeline can help users complete the consumption process more efficiently, making it easier for them to understand and choose a product.

Weitao You

Juntao Ji

Lingyun Sun

Changyuan Yang

Mi Yu

Shi Chen

Jiayi Yao

Automatic Generation of Interactive Nonlinear Video for Online Apparel Shopping Navigation

在线服装购物导航的交互式非线性视频自动生成

Infrared target recognition and anti-interference in complex battlefields is one of the key technologies enabling the precise strike capability of aircraft. Currently, infrared-guided aircraft face complex interference such as natural backgrounds and artificial decoys, leading to a decrease in the performance of infrared target recognition. A particular challenge to infrared target recognition and anti-interference capabilities is the strong interference situation caused by the combination of target maneuvering and the dense, continuous, and coordinated deployment of infrared decoys. To address extreme issues such as complete loss of target feature information and inability to identify due to target occlusion, we develop an anti-interference recognition method based on a visually inspired Spatio-Temporal Semantic Reasoning Model (STSRM). Firstly, inspired by the functional characteristics of visual semantic reasoning, the STSRM is proposed to simplify the reasoning of relationships among multiple regions into modeling relationships between corresponding region node features in a graph-based module. Secondly, an anti-occlusion target recognition model based on STSRM is constructed, which introduces a reasoning graph module connecting node regions to infer semantic information and predict targets between regions. The test results on the infrared dataset established in this paper indicate that the proposed anti-interference recognition model can make accurate target predictions in large-scale or full-occlusion conditions, and we achieve 13.9% and 3.1% improvement on mAP scores and mIoU scores, compared to current advanced method on our simulated infrared dataset.

Xi Yang

Shaoyi Li

Saisai Niu

Binbin Yan

Zhongjie Meng

Graph-based Spatio-Temporal Semantic Reasoning Model for Anti-occlusion Infrared Aerial Target Recognition

基于图的时空语义抗遮挡红外空中目标识别推理模型

Incomplete multi-view clustering (IMVC) aims to leverage complementary information from multi-view data with missing instances to enhance clustering performance. Many existing IMVC methods exhibit limitations in effectively exploiting hidden information and addressing distribution differences between views and modules. To address these challenges, we present a novel IMVC framework that leverages the proposed stack feature-based matrix completion to impute the missing instances, enhancing the exploitation of underlying information. We also incorporate graph consensus to integrate graph structures learned from both completed and observed data. Additionally, we introduce correntropy-induced metric as a flexible measurement to adaptively assign different constraints to various views and modules. Furthermore, we derive an efficient iterative algorithm based on Fenchel conjugate and accelerated block coordinate update (BCU) to solve the joint learning problem. Experimental results on eight benchmark datasets demonstrate the superior performance of our method compared to state-of-the-art IMVC methods across various metrics.

Lei Xing

Yawen Song

Badong Chen

Changyuan Yu

Jing Qin

Incomplete Multi-View Clustering Via Correntropy and Complement Consensus Learning

基于相关熵和互补共识学习的不完全多视图聚类

Abbreviation	IEEE Trans. Multimedia
Journal Impact	8.56
Quartiles(Global)	TELECOMMUNICATIONS(Q1)

ISSN	1520-9210, 1941-0077
h-index	152
Top Journals	Yes

Publication Information	Publisher: Institute of Electrical and Electronics Engineers Inc.，Publishing cycle: Quarterly，Journal Type: journal，Open Access Journals: No
Basic data	Year of publication: 1999，Proportion of original research papers: 100.00%，Self Citation Rate:21.40%， Gold OA Rate: 3.34%
Average review cycle	网友分享经验：平均3.0个月
Average recruitment ratio	网友分享经验：较易

IEEE Transactions on Multimedia

Journal Citation Format

A journal article with 1 author

A journal article with 2 authors

A journal article with 3 authors

A journal article with 5 or more authors

Books Citation Format

Thesis Citation Format

Web sites Citation Format

Patent Citation Format

Staying up late manually editing references? ivySCI automatically matches journals and helps you generate references with a single click.

Share Submission Experience