David Amebley and Sayanton Dibbo, The University of Alabama
The growing deployment of multi-modal models (MMs) has introduced new attack vectors that can leak sensitive training data, leading to privacy leakage. This paper investigates black-box membership inference attack (MIA) privacy attack on multi-modal vision-language models (VLMs). State-of-the-art research analyzes privacy vulnerabilities primarily to unimodal AI systems, while recent studies indicate MMs can also be vulnerable to privacy attacks. On the other hand, researchers have demonstrated that biologically inspired neural network representations can enhance the resilience of unimodal systems against adversarial attacks. However, the privacy vulnerability of neuro-inspired MMs remains unexplored. In this work, we introduce a systematic neuroscience-inspired topological regularization framework and expose the resilience of MM VLMs against image-text-based MIA vulnerabilities. We examine this MIA vulnerability using three different VLMs: BLIP, PaliGemma 2, and ViT-GPT2, across three benchmark datasets: COCO, CC3M, and NoCaps. We compare the baseline and neuro VLMs (with topological regularization) defined as neuro (τ =2) and another variant neuro++ (τ =3). Our results on the BLIP model using the COCO dataset demonstrate that neuro-inspired regularization significantly reduces MIA attack success, with neuro (τ=2) achieving an approximately 24% reduction in mean ROC-AUC, while preserving or improving utility as measured by MPNet and ROUGE-2. Additional evaluations on BLIP, PaliGemma 2, and ViT-GPT2 across CC3M and NoCaps indicate that privacy improvements are architecture and dataset-dependent, with stronger regularization (τ =3) generally required to achieve consistent reductions in membership inference risk. This research contributes to the growing understanding of privacy risks in MMs and a pathway to design privacy threat-resilient VLMs.
