As Contrastive Language-Image Pre-training (CLIP) models are increasingly adopted for diverse downstream tasks and integrated into large vision-language models (VLMs), their susceptibility to adversarial perturbations has emerged as a critical concern. In this work, we introduce X-Transfer, a novel attack method that exposes a universal adversarial vulnerability in CLIP. X-Transfer generates a Universal Adversarial Perturbation (UAP) capable of deceiving various CLIP encoders and downstream VLMs across different samples, tasks, and domains. We refer to this property as super transferability—a single perturbation achieving cross-data, cross-domain, cross-model, and cross-task adversarial transferability simultaneously. This is achieved through surrogate scaling, a key innovation of our approach. Unlike existing methods that rely on fixed surrogate models, which are computationally intensive to scale, X-Transfer employs an efficient surrogate scaling strategy that dynamically selects a small subset of suitable surrogates from a large search space. Extensive evaluations demonstrate that X-Transfer significantly outperforms previous state-of-the-art UAP methods, establishing a new benchmark for adversarial transferability across CLIP models.

AI systems like CLIP learn to understand images and text together, and they’re now widely used in AI tools, such as chatbots. But these systems can be tricked. By slightly altering an image in a way that’s invisible to humans, attackers can make the AI misinterpret it completely.In this work, we introduce X-Transfer, a powerful new method that creates such a “visual trick”—a single, tiny change that confuses many different CLIP models across various image types, tasks, and applications. We call this rare and powerful ability super transferability because one small change works almost everywhere.X-Transfer achieves this using an efficient technique we call surrogate scaling, which smartly picks the right models to “train against” without needing to run expensive computations on all possible models.Our results show that X-Transfer is far more effective than past method. It sets a new standard for evaluating the security of AI systems that use CLIP, raising important concerns for their real-world deployment.