NATivE: Multi-modal Knowledge Graph Completion in the Wild

Authors: Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Binbin Hu, Ziqi Liu, Wen Zhang, Huajun Chen

Deep-Dive Summary:

NATivE: Multi-modal Knowledge Graph Completion in the Wild

Abstract

Multi-modal knowledge graph completion (MMKGC) aims to automatically discover unobserved factual knowledge from a given multi-modal knowledge graph by collaboratively modeling the triple structure and multi-modal information from entities. However, real-world multi-modal knowledge graphs (MMKGs) present challenges due to their diverse and imbalanced nature, which means that the modality information can span various types (e.g., image, text, numeric, audio, video) but its distribution among entities is uneven, leading to missing modalities for certain entities. Existing works usually focus on common modalities like image and text while neglecting the imbalanced distribution phenomenon of modal information. To address these issues, we propose a comprehensive framework NATivE to achieve MMKGC in the wild. NATivE proposes a relation-guided dual adaptive fusion module that enables adaptive fusion for any modalities and employs a collaborative modality adversarial training framework to augment the imbalanced modality information. We construct a new benchmark called WildKGC with five datasets to evaluate our method. The empirical results compared with 21 recent baselines confirm the superiority of our method, consistently achieving state-of-the-art performance across different datasets and various scenarios while keeping efficient and generalizable. Our code and data are released at https://github.com/zjukg/NATIVE.

CCS Concepts

• Computing methodologies → Knowledge representation and reasoning; Semantic networks.

Keywords

Multi-modal knowledge graph, Knowledge graph completion, Multi-modal fusion, Adversarial learning

ACM Reference Format

Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Binbin Hu, Ziqi Liu, Wen Zhang*, and Huajun Chen*. 2024. NATivE: Multi-modal Knowledge Graph Completion in the Wild. In Proceedings of The 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2024). ACM, New York, NY, USA, 11 pages. https://doi.org/XXXXXXX.XXXXXXX

1 Introduction

Multi-modal knowledge graphs (MMKGs) [7, 11, 32] are a rich extension of traditional knowledge graphs (KGs) [30, 50], significantly enriching the content of knowledge graphs by combining structured triples (head entity, relation, tail entity) with comprehensive multi-modal entity attributes. MMKGs are representative and have become an important infrastructure in many fields of artificial intelligence, such as recommendation systems [51], computer vision [18], and natural language processing [10, 60, 64].

Researching how to leverage structured knowledge in knowledge graphs and multi-modal knowledge graphs is significant for many areas of artificial intelligence research. However, knowledge graphs and multi-modal knowledge graphs often face severe completeness issues, as they contain many unobserved factual knowledge. This phenomenon makes knowledge graph completion (KGC) [30] an important task aimed at automatically discovering new knowledge in a given knowledge graph. Traditional KGC methods [3, 40, 43, 58] typically focus on learning structured embeddings to model the triple structure and measure the credibility of triples. Additionally, MMKGs complete tasks more comprehensively by integrating multi-modal information such as images and text into KGC models, extending the task to multi-modal knowledge graph completion (MMKGC) [5, 27, 57].

(a) Modality statistics in different MMKGs.

Existing MMKGC methods [27, 37, 48, 56] typically treat the multi-modal information of entities as auxiliary multi-modal embeddings and integrate them to enhance entity representations. However, these methods overlook two key issues in MMKGC in real-world scenarios: the diversity issue and the imbalance issue. The diversity issue arises from the various modalities in modern information media. As the use of various modal data such as numeric [46], audio [52], and video [34] increases in MMKG construction, Figure 1(a) summarizes the range of modalities contained in different MMKGs. However, existing MMKGC methods are primarily designed for common image and text modalities, limiting their generalization ability to diversified modalities. The imbalance issue arises from the uneven distribution of modality information [9], meaning that some important modality information may be missing in real-world knowledge graphs. Figure 1(b) shows the total amount and entity-level distribution of different modalities in TIVA [52], confirming the uneven distribution of modalities in the dataset. Existing research has not focused on this issue or has addressed it through simple initialization [56], leading to underutilization of modality information.

To address these key issues in MMKGC in the wild, we propose a novel framework NATivE, which can process and fuse numeric, audio, text, image, video, and any other modalities into an embedding space through adaptive fusion and adversarial enhancement. NATivE consists of two key modules, namely the relation-guided dual adaptive fusion (ReDAF) module and the collaborative modality adversarial training (CoMAT) module. The ReDAF module promotes diverse multi-modal fusion through relation guidance, adjusting the weights of each modality to support arbitrary input modalities. The CoMAT module designs an adversarial training strategy to enhance imbalanced modality information based on the Wasserstein distance [1]. Furthermore, we conduct theoretical analysis to prove the rationality of our design. We construct a new benchmark called WildKGC, which contains five MMKG datasets to evaluate our method against 21 recent baselines, and further exploration. Our contributions can be summarized as follows:

• Innovative Framework: We propose a new framework called NATivE to address the diversity and imbalance issues of MMKGs in the wild. NATivE enables adaptive fusion of arbitrary modalities through relation guidance, addressing the diversity issue of MMKGs while enhancing imbalanced modality information through collaborative modality adversarial training.
• Theoretical Analysis: We provide theoretical analysis to prove the legitimacy and rationality of our design.
• Comprehensive Experiments: We construct a new benchmark for evaluating MMKGC tasks in the wild and conduct comprehensive experiments to demonstrate the effectiveness, efficiency, and generalization ability of the NATivE framework.

2 Related Work

2.1 Knowledge Graph Completion

Knowledge graph completion (KGC) [30] is a fundamental task in the knowledge graph domain, aimed at discovering unobserved triples in a given knowledge graph. Traditional KGC methods are typically based on embedding techniques that map entities and relations in the knowledge graph to continuous vector spaces and learn these embeddings based on existing triple structures. This technique is also known as knowledge graph embedding (KGE) [50]. Generally, traditional KGE models design different scoring functions to measure the plausibility of triples, with the overall goal of assigning higher scores to positive triples and lower scores to negative triples.

Existing KGE models can be divided into two main categories: translation-based methods and semantic matching methods. Translation-based methods, such as TransE [3], TransD [21], RotatE [40], OTE [41], and PairRE [6], model the triple structure as a translation from the head entity to the tail entity, designing distance-based scoring functions as plausibility measures. Semantic matching methods, such as DistMult [58], ComplEx [43], and TuckER [2], adopt similarity scoring functions based on tensor decomposition. Some methods [13, 28, 29, 59, 66, 69] also attempt to use deep neural networks to extract structural semantics.

2.2 Multi-modal Knowledge Graph Completion

Multi-modal knowledge graph completion (MMKGC) further considers leveraging the complex multi-modal information in multi-modal knowledge graphs (MMKGs) to enhance KGC tasks. Current mainstream methods typically extend traditional KGE models by introducing more flexible multi-modal entity embeddings, such as visual embeddings and textual embeddings. These embeddings are extracted through pre-trained models and represent the multi-faceted features of entities. According to our classification, MMKGC methods can be further divided into three categories. The first category is modality fusion methods, which design elegant ways to achieve multi-modal fusion in the same representation space [5, 8, 25, 35, 37, 48, 54, 56, 65]. The second category is modality ensemble methods, which learn models for each modality separately and perform joint predictions through ensemble learning [27, 68]. The third category is negative sampling (NS) enhancement methods. As mentioned earlier, negative sampling is an important technique in KGE model training. Therefore, some methods [57, 63, 67] attempt to leverage multi-modal information to enhance the negative sampling process, generating high-quality negative samples. The problem scenarios of existing MMKGC methods are relatively simple, often considering only text and image modalities, with some works only considering numeric modalities. Moreover, they do not address the data imbalance issue in MMKGs. In our work, we aim to complete the MMKGC task from a more unified perspective, targeting more modalities and more complex wild multi-modal data distributions.

2.3 Generative Adversarial Networks

Generative adversarial networks (GANs) [17] are a milestone advancement in the field of deep learning. GANs propose to train both the discriminator and generator through a min-max game between them to achieve better performance, and this method has been widely applied in various fields such as computer vision [1, 22], natural language processing [12, 61], information retrieval [47, 62], and recommendation systems [53, 5, 61]. In the KGC domain, some methods [4, 33, 42, 49] adopt GAN-based frameworks to enhance the negative sampling [3] process. For example, KBGAN [4] combines reinforcement learning (RL) and GAN to learn better sampling strategies; MMKRL [33] designs adversarial training strategies for MMKGC. However, these methods often focus on traditional KGC and frequently require reinforcement learning, leading to limited effectiveness. Our work is the first to introduce multi-modal entity information into MMKGC and propose a unified framework to enhance MMKGC models.

3 Task Definition

A knowledge graph (KG) can typically be represented as , where and represent the set of entities and the set of relations, respectively. is the set of triples. Additionally, multi-modal knowledge graphs (MMKGs) have a set of modalities , which includes different modalities (image , text , numeric , audio , video ). For a given modality , the set of modality information is represented as . For an entity , its modality information is represented as , which is empty if the corresponding modality information is missing. The elements of different modalities vary in form; for example, when , can be a set of images; when , it can be some video clips. It is important to note that the graph structure () is also an inherent modality of each entity, and the structural information is already reflected in the set of triples .

The overall goal of the multi-modal knowledge graph completion (MMKGC) task is to learn a scoring function to distinguish the plausibility of a given triple . In this case, a higher score indicates that the triple is more plausible. To support gradient-based optimization for differentiable computation, entities and relations are embedded into continuous vector spaces, a process known as knowledge graph embedding (KGE). The embeddings of entities and relations are represented as and , where and are the embedding dimensions. Traditional KGE models focus solely on the structural information of triples, so these embeddings are referred to as structural embeddings. The scoring function utilizes structural embeddings to compute the scores of triples. For MMKGs, multi-modal entity information should also be considered in the scoring function, which means that additional embeddings are needed to represent the multi-modal features of entity under different modalities. This extension complicates the scoring function due to the integration of multi-modal embeddings. During the training phase, negative sampling (NS) [3] is widely used to construct artificial negative triples for contrastive learning, as knowledge graphs typically only contain observed positive triples. Given a positive triple , the NS process randomly replaces the head entity or tail entity with another entity . The set of negative triples can be represented as . In the inference phase, KGC models are typically evaluated through link prediction tasks [3]. The goal of link prediction is to predict the missing head entity or tail entity in a given query or . For example, in tail entity prediction, the entire set of entities will be evaluated as the candidate set. For each , the plausibility score of the triple is computed and ranked among the entire candidate set. The higher the ranking of the true triple , the better the model performance.

4 Methodology

In this section, we will detail the proposed MMKGC framework. We name the model NATivE, aiming to represent and combine various data modalities (numeric, audio, text, image, video, etc.) in MMKGs into multi-modal embeddings and achieve this through adversarial enhancement. This capability allows NATivE to provide robust predictive capabilities in real-world environments when facing diversity and imbalance issues. In NATivE, we design two new modules called relation-guided dual adaptive fusion and collaborative modality adversarial training to address the aforementioned issues.

4.1 Modality Encoding

To fully leverage modality information, we first perform modality encoding to capture modality features, which is a very common step in different MMKGC methods. The raw features of entity in modality are extracted as follows:

where is an element of entity in modality , and is the corresponding pre-trained encoder. These encoders are typically pre-trained on large-scale datasets to extract deep semantic features specific to each modality. For example, we can use BERT [14] as a text encoder and VGG16 [38] as an image encoder. Numeric information can also be extended to sequences and use BERT to capture its modality information. The pre-trained encoders are frozen to capture modality features and are not fine-tuned during training. Additionally, the modality embedding representation of entity in modality is represented as , where is the embedding dimension, and is the projection layer of the modality, aimed at projecting the embeddings of different modalities into the same vector space. Each contains a two-layer multi-layer perceptron (MLP) with ReLU [16] as the activation function.

The modality encoding process generates different modality embeddings for each entity . In the context of multi-modal knowledge graphs (MMKG), embeddings from other modalities serve as auxiliary information to enhance the structure of entities. To unify the representation of multi-modal information, we designate the structural modality as , and the structural embedding is represented as . However, the structural embeddings of entities differ from other modalities. They are set as learnable parameters to adapt to the structural information of triples without relying on the modality encoding provided by pre-trained encoders.

4.2 Relation-guided Dual Adaptive Fusion

To extract the rich semantic information contained in entities, it is often necessary to fuse these multi-modal embeddings. Existing methods often employ fusion mechanisms such as concatenation [37], dot product [27], or gating [48] to achieve modality fusion. However, these methods are primarily designed for specific modalities (e.g., text and image) and may not adequately serve scenarios involving richer modality mixtures. Additionally, given the inherent imbalance in knowledge graphs (KGs), different modalities need to play different roles in different contexts and provide diverse evidence for robust predictions.

To address the above issues, we propose a relation-guided dual adaptive fusion (ReDAF) module in the framework. ReDAF includes an adaptive fusion mechanism that uses a set of adaptive weights for different modalities, which can be dynamically adjusted when modality information is imbalanced. For example, when a certain modality is missing and the corresponding modality embedding is randomly initialized, the adaptive weight for that modality will be reduced. At the same time, we design a relation-related temperature parameter to further adjust the weight distribution, providing relational context for entity representations. This leads to dynamic adjustments of weights under different entities and different relational contexts, which is also why we name this module dual adaptive fusion. Specifically, the head entity of the triple has a relational context , and the adaptive weight of the head entity under each modality is represented as:

where is a learnable vector, is the pointwise operator, is the hyperbolic tangent function, and represents the sigmoid function, which aims to constrain the relation-related temperature within the range of , intending to amplify the differences between different modality weights. With adaptive weights, the joint embedding of the head entity is aggregated as follows:

The joint embedding of the tail entity can also be obtained in a similar manner.

Classic MMKGC methods typically adopt entity-centric modality fusion methods without considering their triple context. The advantage of our design lies in its ability to dynamically adjust the modality weights of different entities in different triples, allowing these weights to be dynamically adjusted based on their relational context. Therefore, ReDAF achieves dual adaptive multi-modal fusion while considering both the modality information of entities and the relational context. Furthermore, ReDAF is a universal modality fusion module capable of handling an unlimited number of input modalities, not just specific modalities.

After obtaining the joint embedding of entities, we adopt the RotatE [40] scoring function to distinguish the plausibility of triples, represented as:

where is the rotation operator in the complex space. A higher score indicates a higher plausibility of the triple. We choose RotatE as the scoring function because it can model the most common relational patterns. During training, we adopt a negative sampling-based loss function to optimize parameters, represented as:

where is the sigmoid function; is a fixed margin; is the negative samples of the triple . Additionally, is the self-adversarial weight proposed in RotatE [40], represented as:

where is a temperature parameter controlling the weight of negative triples. This is a common setting in knowledge graph completion models.

4.3 Collaborative Modality Adversarial Training

In the proposed ReDAF framework, although it achieves dual adaptive multi-modal fusion between imbalanced and diverse multi-modal data, its core functionality is essentially for prediction feature selection, while the original imbalanced modality information remains unchanged. Inspired by the idea of generative adversarial networks (GANs) [17, 44], we propose a collaborative modality adversarial training (CoMAT) module to enhance modality embeddings. The CoMAT module enhances multi-modal embeddings through adversarial training, leveraging the collaborative modality data of specific entities to balance the distribution of multi-modal information. Drawing on the design principles of classic Wasserstein GAN (WGAN) [1], our goal is to establish a min-max game between the discriminator D and the generator G, as follows:

where is the data sample. In this min-max game, the discriminator D is assigned to score the input data sample , while the generator G aims to generate a synthetic data sample . Through adversarial training, the generator G adapts to the actual distribution of data , while the discriminator D learns to judge the credibility of the input data.

In our scenario, data corresponds to the entity embeddings in the knowledge graph (KG). As mentioned earlier, each entity is represented by multiple modality embeddings, which can be represented as , where is each modality. We aim to design a pair of G and D to learn the joint distribution of multi-modal embeddings and enhance the imbalanced multi-modal information. Therefore, G attempts to generate a set of embeddings that can form a new synthetic entity, denoted as . By adversarial training between real entities and synthetic entities, we can enrich the multi-modal embeddings of entities.

4.3.1 Generator Design

In the design of CoMAT, the generator G is implemented by a two-layer MLP. G processes the input random noise and the concatenated real multi-modal embeddings to generate enhanced synthetic embeddings:

where is obtained by collaboratively concatenating different multi-modal embeddings (as shown in Figure 2), and the generated synthetic embedding has the same shape as . We can obtain the generated embedding of each modality generated by G through dimension splitting . The ultimate goal of this design is to simulate the probability distribution of multi-modal embeddings for each entity. Given a triple , synthetic embeddings can be generated for the head entity and tail entity, denoted as and , respectively. Additionally, the corresponding synthetic entities are denoted as and .

4.3.2 Discriminator Design

The next step is to design the discriminator D to score the synthetic embeddings. In existing work in other domains [1, 31], D is typically implemented by another two-layer MLP. However, in the KGC task, the scoring function mentioned in Equation 4 serves as a natural substitute for the discriminator. For a given triple and the synthetic entities generated by G, we can construct a series of synthetic triples, denoted as . The scores of these synthetic triples can be computed through the ReDAF module and the scoring function , indicating the credibility of the synthetic embeddings in the context of triples. We can denote these synthetic triples as a synthetic set . Therefore, in the design of CoMAT, the adversarial training loss can be represented as:

It is important to note that synthetic entities, like real entities, also have multiple multi-modal embeddings. They can be obtained through ReDAF to acquire joint embeddings and subsequently compute the final scores. Thus, the final min-max game between D and G can be represented as:

According to the previous settings, the parameters of G include all parameters in the two-layer MLP, while the parameters of D include existing real embeddings and additional parameters mentioned in ReDAF. These two parts are iteratively optimized through adversarial loss for convergence.

4.4 Overall Training Objective

The final training objective of NATivE combines the aforementioned knowledge graph completion (KGC) training loss and adversarial loss . The goal of the discriminator is to minimize and distinguish the synthetic entities generated by the generator . Conversely, the goal of the generator is to generate high-scoring synthetic entities. Therefore, the training objectives of and can be represented as:

where and are the loss weights, and is the gradient penalty term [19], commonly used to stabilize WGAN training, represented as:

During training, these two losses will be iteratively minimized.

4.5 Theoretical Analysis

In the previous sections, we provided intuitive motivation and design for the CoMAT module. We further provide theoretical analysis to discuss the rationality of our design. The success of WGAN relies on the scoring function of the discriminator satisfying the K-Lipschitz condition, typically achieved through a two-layer MLP with gradient penalty [1, 31]. However, our design adopts the RotatE scoring function as the discriminator. We can prove that the function is also a K-Lipschitz function for the head entity and tail entity .

Proposition: Let the function . Then for any in the domain of the function, there exists a scalar such that:

Proof: Based on the properties of the triangle inequality, we have the following derivation:

Moreover, in the RotatE scoring function, the relation embeddings are constrained to unit modulus [40], which means that for any , . Let , the proof is complete. This indicates that is K-Lipschitz for the head entity . A similar argument applies to the tail entity . From the above analysis, it can be seen that our WGAN-based design, utilizing the scoring function of the MMKGC model, is theoretically sound and reasonable for scoring synthetic entities. This makes the design of the entire framework more collaborative and natural. In fact, CoMAT, as a versatile framework, can also be extended to other embedding-based MMKGC models. We will further demonstrate its effectiveness and generalization ability through empirical evaluation.

5 Experiments and Evaluation

In this section, we first introduce the experimental process and settings, followed by a comprehensive discussion of extensive experiments to highlight the advantages of our method in various scenarios. The following six research questions (RQs) are key issues we explore in the experiments:

• RQ1: Can our model NATivE surpass existing baselines and make substantial progress in the MMKGC task?
• RQ2: Can NATivE maintain robust performance in the MMKGC task when modality information is imbalanced?
• RQ3: What is the contribution of each module in NATivE to the final results? Are these module designs reasonable?
• RQ4: Does our designed CoMAT strategy have sufficient universality and applicability to other MMKGC models?
• RQ5: How does the training efficiency of our model compare to existing methods?
• RQ6: Are there intuitive cases that can directly demonstrate the effectiveness of NATivE?

5.1 Experimental Setup

5.1.1 Datasets

To better explore the multi-modal knowledge graph completion (MMKGC) task in more complex and diverse environments, we constructed a new MMKGC benchmark dataset, including five different datasets. Four of these datasets are derived from previous studies, while one is a brand new dataset. Our benchmark dataset includes the following:

• **MKG-Y and MKG-W [57]**: These are two multi-modal knowledge graphs (MMKGs) derived from YAGO [39] and Wikidata [45], containing images and text, proposed by [57].
• **DB15K [32]**: This is an MMKG containing image, text, and numeric information, proposed by [32], which is a subset of DBpedia [26].
• **TIVA [52]**: This is an MMKG containing image, text, audio, and video information modalities, with 12K entities.
• **KVC16K [34]**: This dataset is adapted from KuaiPedia [34] (a video concept encyclopedia), and we reorganized it into an MMKG, utilizing the image/text/audio/video features provided by the original authors.

We named this benchmark dataset WildKGC, which contains the aforementioned five MMKGs. Detailed information about the datasets can be found in Table 1. The original modality features are inherited from the original datasets.

5.1.2 Tasks and Evaluation Protocols

We perform link prediction tasks [3] on the five datasets, which is an important task in knowledge graph completion (KGC). The setup for link prediction has already been introduced in Section 3. Following existing work, we use ranking-based evaluation metrics [40], such as Mean Reciprocal Rank (MRR) and Hit@K (K=1, 3, 10) to evaluate the results.

Additionally, to ensure fair comparisons, we adopt a filtering setting [3] to remove candidate triples that already exist in the training set.

5.1.3 Baseline Method Comparisons

In our experiments, we adopt 21 different state-of-the-art KGC and MMKGC models as baselines for comprehensive comparison and analysis. These baseline methods can be divided into four categories: single-modal (traditional) KGC methods, multi-modal KGC methods, negative sampling methods, and numeric-aware KGC methods.

1. Single-modal KGC methods: We select five state-of-the-art single-modal KGC methods, including TransE [3], DistMult [58], ComplEx [43], RotatE [40], and PairRE [6]. These methods design elegant scoring functions and learn structural embeddings of the given knowledge graph without using any multi-modal information.
2. Multi-modal KGC methods: We adopt ten different MMKGC methods that consider both multi-modal information and triple structure information, including IKRL [56], TBKGC [37], TransAE [54], RSME [48], MMKRL [33], VBKGC [67], OTKGE [5], IMF [27], QEB [52], and VISTA [25]. Comparing with these methods can validate the effectiveness of our model NATivE. Among these methods, MMKRL designs an adversarial training framework, which has some similarities with our CoMAT design.

1. Negative sampling methods: We adopt three different negative sampling methods for comparison with our method, including KBGAN [4], MANS [63], and MMRNS [57]. Among them, KBGAN [4] is an adversarial negative sampling method designed for traditional KGC, applying reinforcement learning to optimize the model. MANS [63] and MMRNS [57] are two negative sampling strategies designed for MMKGC, leveraging multi-modal information to enhance the negative sampling process.
2. Numeric-aware KGC methods: We adopt three popular numeric-aware KGC methods, including KBLRN [15], LiteralE [24], and KGA [46]. These methods consider numeric information to enhance KGC models. However, their designs cannot be generalized to other modalities (such as images and text) as they are specifically designed for numeric modality enhancement.

All selected baseline methods are embedding-based KGC and MMKGC methods. Other methods, such as text-based methods [36, 59] or GNN-based methods [69], were not considered as they are orthogonal to our design.

5.1.4 Implementation Details

We implemented our NATivE framework based on OpenKE [20] (a well-known open-source KGC library). We conducted experiments on a Linux server equipped with the Ubuntu 20.04.1 operating system, using a single NVIDIA A800 GPU.

In NATivE, we fixed the batch size to 1024 and set the number of training epochs to 1000. The embedding dimensions and were adjusted from {150, 200, 250}, and the number of negative samples was adjusted from {32, 64, 128}. The margin value was adjusted from {3, 6, 9, 12}, and the temperature was set to 2. In CoMAT, the random noise dimension was set to 64, and the coefficients and were adjusted from {1e-2, 1e-3, 1e-4, 1e-5}. We used the Adam optimizer [23] to optimize the model, with learning rates adjusted from {1e-3, 1e-4, 1e-5} for the generator (G) and discriminator (D) in the adversarial training setup. For baseline methods, we reproduced results on WildKGC according to the methods and parameter settings described in the original papers and their open-source official codes. Some baseline results referenced MMRNS [57].

5.2 Main Results (RQ1)

In the main results of the MMKGC experiments, we compared the proposed method NATivE with 18 different KGC baseline methods, with results shown in Table 2. To ensure fair comparisons, we considered all modalities utilized by each method and conducted two sets of different experiments on our model. The first set of experiments only considered the modalities common to all datasets (structural/image/text), while the second set of experiments considered all modalities inherent to each specific dataset.

Table 2 shows that NATivE outperforms all existing baseline methods and achieves new state-of-the-art (SOTA) results. Specifically, NATivE significantly surpasses baseline methods in Hit@1 metrics on the DB15K (15.75%), KVC16K (11.47%), and TIVA (10.05%) datasets, indicating a substantial improvement in accurate reasoning capabilities. Even when only using the common modalities (structural/image/text) considered by most mainstream methods, NATivE still outperforms baseline methods, demonstrating its ability to effectively utilize information from different modalities. The performance comparison with two adversarial-based methods, MMKRL [33] and KBGAN [4], further confirms the superiority of our designed adversarial module, which can fully leverage the potential of multi-modal information of entities.

Additionally, all MMKGC baseline methods in Table 2 did not consider the numeric information presented in the DB15K dataset. Therefore, we conducted another set of experiments comparing NATivE with other numeric KGC baseline methods. The design principles of these methods differ significantly from typical MMKGC methods and lack generalization capabilities to other modalities, so we only compared on the DB15K dataset. From Table 3, it can be seen that when only considering structural and numeric modalities, NATivE still outperforms all baseline methods. As the diversity of modality information increases, the effectiveness of the model further enhances.

Figure 3: Imbalance MMKGC results. We report the MRR and Hit@10 results on the DB15K dataset. Additionally, we categorize the test triples based on whether they have complete modality information, and we report the experimental results for each group, where: Group1 (both h and t have complete modalities); Group2 (either h or r has missing modalities); Group3 (both h and t have missing modalities).

5.3 Imbalanced Multi-modal Knowledge Graph Completion Experiments (RQ2)

To better demonstrate the performance of NATivE in complex modality imbalance scenarios, we conducted a series of multi-modal knowledge graph completion (MMKGC) experiments, particularly in imbalanced scenarios. We introduced a new parameter—the imbalance ratio (imbalanced proportion), denoted as , which indicates the degree of imbalance in the dataset, specifically the percentage of entities lacking modality information. For example, indicates that there is 30% modality information missing in the given multi-modal knowledge graph (MMKG).

Specifically, we randomly partition the original dataset, randomly selecting a proportion of of modality information to discard. Additionally, we distinguish between two types of imbalance stratification: entity-level imbalance and modality-level imbalance. For entity-level imbalance partitioning, we randomly discard all modality information of the selected entities; for modality-level imbalance, we randomly discard a proportion of modality information across all entities, meaning that a certain entity may exhibit different amounts of modality information. The missing modality information discarded in these settings will be randomly initialized, and the MMKGC experimental results on the imbalanced dataset are shown in Figure 3.

The results of the imbalance experiments indicate that NATivE can maintain its performance even when modality information is imbalanced. As the degree of imbalance increases, the performance of baseline models TBKGC [37] and QBE [52] significantly declines, while our method shows relative stability. Furthermore, the impact of imbalanced modalities is more pronounced on coarse-grained metrics (such as Hit@10), indicating that the completeness of modality information has a greater impact on coarse-grained entity ranking. Additionally, excessive irrelevant noise in modality information can hinder the model’s accurate reasoning, leading to a slight performance improvement of NATivE as the imbalance ratio increases. It is also noteworthy that the CoMAT design has a significant impact on model performance, maintaining its effectiveness even in imbalanced situations.

We evaluated the impact of imbalanced modalities on different triples by separately counting the results of triples with varying levels of missing entity modality information, with specific results shown in Figure ??(b).

In this figure, the imbalance ratio is . Interestingly, when the modality information is completely missing in the triple (Group 3), the model performance is not necessarily worse than when partially missing (Group 2). The reason for this result is that modality-level missing is less likely to lead to the complete loss of entity modality information, thus retaining some modality information that is helpful for the final prediction, resulting in relatively better model performance (MRR of Group 3). In contrast, entity-level missing scenarios deprive certain entities of all modality information, leading to the complete loss of effective information in modalities, resulting in poorer performance on coarse-grained metrics (such as Hit@10).

5.4 Ablation Studies (RQ3)

We conducted extensive ablation studies to verify the effectiveness of each module in our design. In the ablation studies, we conducted three sets of experiments to validate the contribution of each module by removing the corresponding key components in NATivE. These three sets of experiments focus on different modalities, the ReDAF module, and the CoMAT module. The results of the ablation studies are summarized in Table 4. From the results of G1, it can be concluded that although the role of the image modality seems smaller compared to the text modality, each modality contributes to the overall performance. The results of G2 show that key components in the ReDAF module play a crucial role, and removing these components leads to a significant drop in performance. Furthermore, compared to other widely used modality fusion strategies in MMKGC models (such as concat and dot-product), our NATivE module still performs excellently. From the results of G3, it can be seen that CoMAT brings a significant performance boost, especially in accuracy metrics like Hit@1. Additionally, when removing the gradient penalty loss or changing the adversarial loss to a vanilla GAN version with log-likelihood [17], the model performance still declines. We also attempted to replace the discriminator D with a two-layer MLP, but the final results were not as good as the original NATivE. This indicates that our adopted WGAN [1] framework performs better in the MMKGC scenario, confirming the correctness of our previous analysis and proofs in the MMKGC setup.

5.5 Further Analysis (RQ4 & RQ5)

5.5.1 Generalization Analysis (RQ4)

As previously mentioned, CoMAT is a general framework based on embeddings for enhancing MMKGC models. To demonstrate this, we applied CoMAT to more MMKGC models (IKRL [56], TBKGC [37], and VBKGC [67]) and presented the results in Figure 4. We can conclude that MMKGC models trained with CoMAT can achieve significant performance improvements. This indicates that CoMAT can serve as a universal adversarial enhancement framework in different MMKGC models.

5.5.2 Efficiency Analysis (RQ5)

Another important concern is the efficiency of adversarial methods. This is because iterative training strategies can lead to more computations. Therefore, we evaluated the efficiency of several MMKGC methods. The methods we selected cover non-adversarial methods (OTKGE [5], MMRNS [57], IMF [27]) and adversarial methods (KBGAN [4], MMKRL [33]). We evaluated the efficiency and performance of different models on the same device with the same batch size of 1024 and dimension of 250. As shown in Figure 5, we can observe that NATivE achieves a good balance between efficiency and performance, achieving the best results while remaining relatively fast and efficient. Although the adversarial training module CoMAT slightly slows down the training speed, this delay is within an acceptable range, and the model performance shows significant improvement.

5.6 Case Studies (RQ6)

This section further illustrates our design through clear case studies. We previously emphasized that the relation guidance in ReDAF can amplify the differences between different modalities and filter out useful parts, as the relation temperature is constrained within the range of (0, 1) through the sigmoid function. We first show the distribution of temperatures, as shown in Figure 6. We can observe that the distribution is diverse.

Figure 6: Distribution of relation temperatures and high-frequency relations that can enhance the weights of each modality. To better illustrate this, we filtered and listed the most common relations in each modality that would enhance the corresponding modality weights, as shown in Figure 6(b). We found that different relations have varying degrees of dependence on the modality information of entities, especially in some intuitive visual information. When predicting movie genres, such image information greatly improves the results. At the same time, personal affiliations, nationalities, and places of death can often be found in textual descriptions, making the text modality of significant value in predicting the outcomes of these relations.

6 Conclusion

In this paper, we propose a novel MMKGC framework NATivE to address the diversity and imbalance issues of MMKGs. NATivE includes two core designs: ReDAF and CoMAT. ReDAF proposes a relation-guided dual adaptive fusion method to integrate adaptive features of arbitrary modalities to obtain joint representations; while CoMAT aims to enhance the information of imbalanced modalities through a WGAN-based training strategy. We conducted in-depth theoretical analysis to prove the rationality of our design. We constructed a new MMKGC benchmark and conducted comprehensive experiments against 21 baselines to demonstrate the effectiveness, generalization ability, and efficiency of our framework. In the future, we believe that the MMKGC task and the downstream applications of MMKGs in more complex real-world scenarios remain significant challenges to be addressed.

Acknowledgments

This research was supported by the National Natural Science Foundation of China (NSFC62306276 / NSFCU23B2055 / NSFCU19B2027 / NSFC91846204), the Zhejiang Provincial Natural Science Foundation (No. LQ23F020017), the Ningbo Natural Science Foundation (2023J291), the Yongjiang Talent Project (2022A-238-G), and the Special Fund for Basic Scientific Research Business of Central Universities (226-2023-00138).

Original Abstract: Multi-modal knowledge graph completion (MMKGC) aims to automatically discover the unobserved factual knowledge from a given multi-modal knowledge graph by collaboratively modeling the triple structure and multi-modal information from entities. However, real-world MMKGs present challenges due to their diverse and imbalanced nature, which means that the modality information can span various types (e.g., image, text, numeric, audio, video) but its distribution among entities is uneven, leading to missing modalities for certain entities. Existing works usually focus on common modalities like image and text while neglecting the imbalanced distribution phenomenon of modal information. To address these issues, we propose a comprehensive framework NativE to achieve MMKGC in the wild. NativE proposes a relation-guided dual adaptive fusion module that enables adaptive fusion for any modalities and employs a collaborative modality adversarial training framework to augment the imbalanced modality information. We construct a new benchmark called WildKGC with five datasets to evaluate our method. The empirical results compared with 21 recent baselines confirm the superiority of our method, consistently achieving state-of-the-art performance across different datasets and various scenarios while keeping efficient and generalizable. Our code and data are released at https://github.com/zjukg/NATIVE

PDF Link:2406.17605v1