Semantic-aware Adversarial Fine-tuning for CLIP
arXiv:2602.12461v1 Announce Type: new Abstract: Recent studies have shown that CLIP model’s adversarial robustness in zero-shot classification tasks can be enhanced by adversarially fine-tuning its image encoder with adversarial examples (AEs), which are generated by minimizing the cosine similarity between images and a hand-crafted template (e.g., ”A photo of a {label}”). However, it has been shown that the cosine similarity between a single image and a single hand-crafted template is insufficient to measure the similarity for image-text pairs. […]