Despite significant advancements in deep learning-based CSI compression, current approaches primarily view it as a source coding problem, neglecting transmission errors. Separate source and channel coding proves suboptimal in finite block length regimes, while autoencoder-based compression schemes struggle with complex channel distributions. We propose Residual-Diffusion Joint Source-Channel Coding (RD-JSCC), leveraging diffusion models to learn robust CSI representations. Our architecture combines a lightweight autoencoder with a residual diffusion module for iterative CSI reconstruction, enabling graceful performance degradation across variable SNR conditions and robust estimation under multipath fading in the uplink feedback channel. Our flexible decoding strategy dynamically selects between autoencoder decoding and diffusion-based refinement based on channel conditions, minimizing the overall computational complexity. Simulations demonstrate RD-JSCC significantly outperforms existing approaches in challenging wireless environments, without adding substantial decoding latency via a two-step inference, offering an efficient solution for next-generation wireless systems.