← 返回列表

大模型安全技术综述

聚合站 SecWiki技术 入库 2026-11-18T05:05:14

This survey provides a systematic review of current safety research on large models, covering Vision Foundation Models (VFMs), Large Language Models (LLMs), Vision-Language Pre-training (VLP) models, Vision-Language Models (VLMs), Diffusion Models (DMs), and large-model-based Agents.

We will keep updating this survey. Please complete the following form to submit your paper for citation.

We appreciate your contributions and look forward to keeping this survey comprehensive and up to date!

⚠️ Note: We completed a major revision of the paper in August 2025. This version reflects our final planned update, and further substantial revisions may not be possible.

The survey have been structured with the following considerations for clarity and readability:

Figure 1: Left: The number of safety research papers published over the past four years. Middle: The distribution of research across
different models. Right: The distribution of research across different types of attacks and defenses.

Paper stats

First, we conducted a keyword-based search targeting specific model types and threat types to identify relevant papers. Next, we manually filtered out non-safety-related and non-technical papers. For each remaining paper, we categorized its proposed method or framework by analyzing its settings and attack/defense types, assigning them to appropriate categories and subcategories. We collected 390 technical papers, with their distribution across years, model types, and attack/defense strategies illustrated in the following Figures. As shown, safety research on large models has surged significantly since 2023, following the release of ChatGPT. Among the model types, LLMs and DMs have garnered the most attention, accounting for over 60% of the surveyed papers. Regarding attack types, jailbreak, adversarial, and backdoor attacks were the most extensively studied. On the defense side, jailbreak defenses received the highest focus, followed by adversarial defenses.

Figure 2: Paper statistics overview.

trends

Based on the survey, we identify several limitations and gaps in existing research and summarize them into the following topics. These open challenges reflect the evolving nature of large model safety, highlighting both technical and methodological barriers that must be overcome to ensure robustness and reliability across various AI systems.

The Purpose of Attack Is Not Just to Break the Model

While attacks are often designed to disrupt model functionality, they also serve as diagnostic tools to uncover unintended behaviors and reveal fundamental weaknesses. A deeper understanding of attack mechanisms can help address vulnerabilities at their root.

What Are the Fundamental Vulnerabilities of Language Models?

LLMs are prone to adversarial inputs, biases, and prompt manipulation. Research must explore how these vulnerabilities stem from model architectures and training data to develop effective defenses.

How Vulnerabilities Propagate Across Modalities?

As Multi-modal Large Language Models (MLLMs) integrate diverse modalities, new vulnerabilities arise. Vision encoders are known to be sensitive to subtle, continuous perturbations in pixel space, while language models are vulnerable to adversarial characters, words, or prompts. The interaction between modalities and how vulnerabilities in one modality propagate to others remains poorly understood.

Diffusion Models for Visual Content Generation Lack Language Capabilities

Despite their success in visual content creation, diffusion models struggle with language comprehension, leading to unintended outputs. Integrating linguistic capabilities remains an open challenge.

How Much Training Data Can a Model Memorize?

Deep neural networks exhibit memorization tendencies, raising privacy concerns. Understanding the mechanisms behind memorization is crucial for balancing performance and security.

Agent Vulnerabilities Grow with Their Abilities

Large-model-powered agents face increasing vulnerabilities as their capabilities expand. Ensuring security in evolving agents requires comprehensive defense frameworks.

Attack Success Rate Is Not All We Need

Metrics like attack success rate (ASR) fail to capture the full scope of safety risks. More comprehensive evaluation frameworks are needed to assess model resilience and ethical considerations.

Static Evaluations Create a False Sense of Safety

Current safety benchmarks are static and may not reflect real-world threats. Dynamic evaluation methods are required to assess evolving risks effectively.

Adversarial Evaluations Are a Necessity, Not an Option

Standard safety tests fail to capture adversarial risks. Adversarial evaluations are essential for understanding worst-case scenarios and improving robustness.

Open-Ended Evaluation

LLMs generate open-ended responses, complicating attack assessments. Improved evaluation metrics and constrained output spaces are needed for accurate safety testing.

Safety Alignment Is Not the Savior

Despite advances in safety alignment, models remain vulnerable to sophisticated attacks. Ensuring deeper, more robust alignment mechanisms is an ongoing challenge.

Jailbreak Attacks Are More Challenging to Defend Against Than Adversarial Attacks

Jailbreak attacks bypass safety mechanisms more flexibly than traditional adversarial attacks. Defense strategies must account for unconstrained perturbations.

The Need for More Practical Defenses

Effective defenses should be generalizable, efficient, black-box compatible, and adaptable to evolving threats.

The Lack of Proactive Defenses

Current defenses focus on mitigating attacks post-occurrence. Proactive defense strategies could deter threats before they succeed.

Detection Has Been Overlooked in Current Defenses

Detection mechanisms can enhance safety but are often neglected. Integrating detection into defense pipelines is a key research direction.

The Current Data Usage Practices Must Change

Ethical and sustainable data usage is critical. Addressing concerns like consent, data attribution, and memorization is necessary for responsible AI deployment.

Safe Embodied Agents

As AI extends into physical systems, ensuring safety in real-world interactions becomes paramount.

Safe Superintelligence

Building safety mechanisms into AGI and superintelligent models is a critical challenge. Oversight systems, adversarial safety, and safety-conscious architectures are potential solutions.

Defense-Oriented Research

The focus on attack strategies has overshadowed the development of robust defenses. Research efforts should prioritize integrated, layered defense mechanisms.

Dedicated Safety APIs

Commercial AI models should provide safety APIs to facilitate external safety evaluations and improvements.

Open-Source Platforms

Developing and sharing safety platforms would foster collaboration, benchmarking, and transparency in AI safety research.

Global Collaborations

AI safety is a global challenge requiring international cooperation in research, policy, and governance.

@article{ma2026safety,
  title={Safety at scale: A comprehensive survey of large model and agent safety},
  author={Ma, Xingjun and Gao, Yifeng and Wang, Yixu and Wang, Ruofan and Wang, Xin and Sun, Ye and Ding, Yifan and Xu, Hengyuan and Chen, Yunhao and Zhao, Yunhan and others},
  journal={Foundations and Trends in Privacy and Security},
  volume={8},
  number={3-4},
  pages={1--240},
  year={2026},
  publisher={Emerald Publishing Limited}
}
原文链接:https://github.com/xingjunm/Awesome-Large-Model-Safety