Two-step heterogeneous fusion hashing network for unpaired cross-modal retrieval

Runtian Song; Yun Liu; Shujuan Ji; Yongquan Liang; Jie Wang; Mengru Wang

doi:10.1117/12.2682247

26 May 2023 Two-step heterogeneous fusion hashing network for unpaired cross-modal retrieval

Runtian Song, Yun Liu, Shujuan Ji, Yongquan Liang, Jie Wang, Mengru Wang

Proceedings Volume 12700, International Conference on Electronic Information Engineering and Data Processing (EIEDP 2023); 127002W (2023) https://doi.org/10.1117/12.2682247
Event: International Conference on Electronic Information Engineering and Data Processing (EIEDP 2023), 2023, Nanchang, China

Abstract

Most existing cross-modal hashing methods take advantage of pairwise relationships to establish correlations be-tween multimodal data. However, the majority of methods employing cross-modal hashing ignore the incompleteness and imbalance of multimodal data. To cope with this issue, we propose a Two-Step Heterogeneous Fusion hashing network for unpaired cross-modal retrieval (TSHF). In the first step, we utilize zero padding to enable image-text stitching of unpaired data and construct a triple network to learn the stitching features together. In the second step, we deploy an attention fusion network in order to explore the hidden information in the data through the inter-modal enhancement and the intra-modal complementation. In addition, manifold learning is adopted to preserve semantic similarity of original features; adversarial learning is introduced to deal imbalance in the features; label prediction ensures that discrimination is preserved. Extensive experiments on WikiData and MIRFlickr datasets indicate that TSHF has excellent performance in paired and unpaired modal retrieval.

Citation Download Citation

Runtian Song, Yun Liu, Shujuan Ji, Yongquan Liang, Jie Wang, and Mengru Wang "Two-step heterogeneous fusion hashing network for unpaired cross-modal retrieval", Proc. SPIE 12700, International Conference on Electronic Information Engineering and Data Processing (EIEDP 2023), 127002W (26 May 2023); https://doi.org/10.1117/12.2682247

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
14 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Semantics

Education and training

Feature fusion

Data modeling

Matrices

Data fusion

Image fusion

Show All Keywords

Keywords/Phrases

Search In:

Publication Years