Abstract: Multi-Modal Learning (MML) aims to learn effective representations across modalities for accurate predictions. Existing methods typically focus on modality consistency and specificity to learn effective representations. However, from a causal perspective, they may lead to representations that contain insufficient and unnecessary information. To address this, we propose that effective MML representations should be causally sufficient and necessary. Considering practical issues like spurious correlations and modality conflicts, we relax the exogeneity and monotonicity assumptions prevalent in prior works and explore the concepts specific to MML, i.e., Causal Complete Cause ($C^3$). We begin by defining $C^3$, which quantifies the probability of representations being causally sufficient and necessary. We then discuss the identifiability of $C^3$ and introduce an instrumental variable to support identifying $C^3$ with non-exogeneity and non-monotonicity. Building on this, we conduct the $C^3$ measurement, i.e., $C^3$ risk. We propose a twin network to estimate it through (i) the real-world branch: utilizing the instrumental variable for sufficiency, and (ii) the hypothetical-world branch: applying gradient-based counterfactual modeling for necessity. Theoretical analyses confirm its reliability. Based on these results, we propose $C^3$ Regularization, a plug-and-play method that enforces the causal completeness of the learned representations by minimizing $C^3$ risk. Extensive experiments demonstrate its effectiveness.

Lay Summary: This work explores how to improve multi-modal learning (MML) from a causal perspective. Existing methods typically define good MML representations from two perspectives: modality consistency and modality specificity. However, this work points out that these methods may lead models to learn representations that are either incomplete or contain irrelevant information. To address this, this work proposes a new standard for high-quality representations: they should be causally complete, that is, both necessary and sufficient for making correct predictions. Simply put, the information learned by the model should be not only enough to make the right decision (sufficiency), but also essential, removing it would lead to wrong results (necessity). To make this possible in real-world settings where data can be noisy or conflicting, this work proposes a relaxed and more practical way to measure and enforce causal completeness. It defines a new concept called Causal Complete Cause ($C^3$), and shows how to measure the quality of learned representations using the proposed $C^3$ risk. A novel twin network structure is introduced to compute this risk through (i) a real-world branch, which utilizes the instrumental variable for sufficiency, and (ii) a hypothetical-world branch, which applies gradient-based counterfactual modeling for necessity. Finally, the work proposes a plug-and-play technique called $C^3$ Regularization that can be embedded into any MML model to help it learn more reliable and causally complete representations. Extensive experiments across various benchmarks demonstrate the advantages of this method.