Non-annotated visual description (NaVD) aims to describe generic visuals without human-annotated pairwise data. The generic visuals refer to images and videos. Existing works mainly focus on one specific visual modality, i.e. , image or video. In this paper, we propose a new framework for this task, which can directly be applied to both image and video with the pipeline unchanged. Essentially, it is a unified framework that flexibly adapts to images and videos. Recently, contrastive visual-language pre-training models (CVLPs) have experienced rapid development, demonstrating powerful abilities to align vision and language. To continuously leverage advanced CVLPs, our framework is designed to work well with general CVLPs. It can easily use image-language CVLPs for image input and switch to video-language CVLPs for video input. Specifically, we propose a CVLP-based framework for NaVD, named CVLP-NaVD. It follows the paradigm of adversarial learning, containing a generator and a discriminator. The generator takes an image or a video as input and produces a corresponding language description, while the discriminator evaluates the generated sentence for its naturalness in human-like language. Apart from the naturalness, CVLPs play a crucial role in enhancing the alignment between visual and language signals during generation. Particularly, we explore three rewarding strategies to compute the alignment score, including directly calculating cosine similarity ( i.e. , VL-cross), projecting visual embeddings into the textual domain ( i.e. , VL-project) and their combination ( i.e. , VL-mix). The three strategies are fully examined in different scenarios. Finally, we conduct extensive experiments with various unpaired and unsupervised setups in both image and video captioning tasks. The experimental results demonstrate that our CVLP-NaVD outperforms the state-of-the-art methods significantly.

Haoran Li

Yanbin Hao

Jiarui Yu

Bin Zhu

Shuo Wang

Tong Xu

ACM Transactions on Multimedia Computing Communications and Applications

ACM TRANSACTIONS ON MULTIMEDIA COMPUTING, COMMUNICATIONS, AND APPLICATIONS 是 ACM 多媒体特别兴趣小组 (SIGMM) 的旗舰出版物。它正在征集有关多媒体各个方面的论文提交。也欢迎有关单一媒体（例如，音频、视频、动画）及其处理的论文。TOMM 是一份经过同行评审的档案期刊，有印刷和数字两种形式。该杂志每季度出版一次；每期大约有 7 篇 23 页的文章。此外，所有特刊均仅在线发布，以确保及时发布。交易主要包括研究论文。这是一份档案期刊，旨在使论文随着时间的推移具有持久的重要性和价值。一般而言，不包括主要关注特定多媒体产品或行业现状的论文。

ACM Transactions on Multimedia Computing, Communications and Applications is the flagship publication of the ACM Special Interest Group on Multimedia (SIGMM). It invites submissions on all aspects of multimedia, as well as papers focused on individual media (e.g., audio, video, animation) and their processing. TOMM is a peer-reviewed archival journal available in both print and digital formats, published quarterly with approximately 7 articles per issue, totaling around 23 pages. Additionally, all special issues are published online only to ensure timely dissemination. The journal primarily includes research papers and aims to maintain the enduring significance and value of the papers over time. Generally, papers focusing primarily on specific multimedia products or the current state of the industry are not included.

《ACM Transactions on Multimedia Computing, Communications and Applications》是ACM多媒体特别兴趣小组（SIGMM）的旗舰期刊，欢迎提交关于多媒体各个方面的论文。我们也接受关于单一媒体（如音频、视频、动画）及其处理的研究。该期刊经过同行评审，提供印刷和数字两种形式，每季度出版一次，每期约有7篇、23页的文章。此外，所有特刊均仅在线发布，以确保及时性。期刊主要发表研究论文，旨在使论文在时间的推移中保持持久的重要性和价值。一般而言，不包括主要关注特定多媒体产品或行业现状的论文。

Category	Quartile
计算机科学	3
计算机科学, 计算机信息系统	3
计算机科学, 计算机软件工程	3
计算机科学, 计算机理论方法	3

CVLP-NaVD: Contrastive Visual-Language Pre-training Models for Non-annotated Visual Description

ivySCI AI Smartly Parses PDF, Answers Researchers' Questions, and Helps You Understand Papers in Seconds

Journal Info

Category	Quartile
COMPUTER SCIENCE, THEORY & METHODS	1