Inertial confinement fusion (ICF) offers a pathway to sustainable energy production, but achieving controlled fusion requires precise modeling of complex structured and image data. Recent breakthroughs underscore the need for scalable methods to analyze multi-modal diagnostic data and simulations, which include scalar inputs, scalar outputs, and image outputs. In this work, we present a diffusion-based generative framework designed to model the joint and conditional distributions of these structured and image data. By leveraging simulation data for pretraining, our approach addresses the challenge of experimental data scarcity and enables robust conditional modeling tasks. This work represents a prototype towards an ICF foundation model, and its architecture is transferable to diverse multi-modal scientific problems.