Where every claim in SecProve
comes from.

The definitive security risk list for LLM-powered applications. Covers prompt injection, insecure output handling, training data poisoning, and more.

Comprehensive taxonomy of adversarial ML attacks and mitigations. Covers evasion, poisoning, extraction, and inference attacks with standardized terminology.

Adversarial Threat Landscape for AI Systems. ATT&CK-style knowledge base of adversarial ML techniques, tactics, and real-world case studies.

Comprehensive guide to AI red teaming from Microsoft's dedicated AI security team. Covers methodology, tools, and findings.

The authoritative framework for managing AI risks. Defines four core functions: Govern, Map, Measure, Manage. Essential reading for anyone building or deploying AI systems.

Updated cybersecurity framework with six core functions: Govern, Identify, Protect, Detect, Respond, Recover.

Introduced DP-SGD for training neural networks with formal differential privacy guarantees. Foundation for private ML.

First practical membership inference attack against ML models. Showed that ML APIs leak information about their training data.

Introduced PGD-based adversarial training, currently the most reliable defense against adversarial examples. Established the robustness-accuracy tradeoff.

International standard for establishing and maintaining an AI management system. Includes 39 controls across 10 areas.

Seminal backdoor attack paper. Demonstrated trojaned models in transfer learning scenarios. Foundational for AI supply chain security questions.

Demonstrated that adversarial examples transfer between models, enabling black-box attacks via surrogate models. Key work on transferability.

Introduced the C&W attack, demonstrating that defensive distillation and other defenses could be reliably bypassed. Changed how robustness is evaluated.

Collection of Anthropic's published research on AI safety, alignment, interpretability, and security.

The European Union's comprehensive AI regulation. Classifies AI systems by risk level and sets requirements for high-risk systems.

Python Risk Identification Toolkit for generative AI. Automated red teaming framework for testing LLM applications.

Voluntary framework for improving privacy through enterprise risk management. Complements the Cybersecurity Framework.

The seminal paper introducing FGSM (Fast Gradient Sign Method). Established that adversarial examples are a fundamental property of neural networks, not a bug.

Demonstrated that LLMs memorize and can be prompted to regurgitate training data verbatim, including PII. Foundational work on LLM privacy risks.

Coalition for Content Provenance and Authenticity. Technical standard for digital content provenance and integrity.

Hugging Face's safe serialization format for ML models. Prevents arbitrary code execution from pickle-based attacks.

Showed that gradually escalating benign conversations can bypass safety filters over multiple turns. Defeats per-message safety checks.

Demonstrated indirect prompt injection attacks through RAG documents, emails, and web content. Essential reading for RAG security.

The GCG attack paper. Showed that adversarial suffixes can bypass safety alignment in LLMs, transferring across models.

CISA guidance on understanding, detecting, and defending against deepfake threats in organizational contexts.

Five practical safety problems: avoiding side effects, reward hacking, scalable oversight, safe exploration, distributional shift. Still the canonical taxonomy for AI safety research questions.

The largest model hub. Security features: malware scanning, pickle scanning, safetensors format. Questions on model provenance, serialization risks (pickle exploits), and model marketplace trust.

Security documentation for LangChain agent framework — sandboxing, tool permissions, prompt injection defenses, and deployment hardening.

Application container security guide covering image, registry, orchestrator, container, and host OS security.

NVIDIA's open-source LLM vulnerability scanner. Tests for prompt injection, jailbreaking, data leakage, and more.

Reports on state-affiliated actors using AI for influence operations. Documents actual observed misuse, not theoretical risks. Key for questions about real-world AI-enabled disinformation.

Research on propaganda techniques, cognitive security, and information warfare. The "firehose of falsehood" model explains high-volume, multi-channel disinformation. Good for strategic questions.

Security docs for major ML platforms. Covers authentication, authorization, experiment tracking security, model registry access controls. Practical infrastructure security questions.

Introduced SISA training for efficient machine unlearning — enabling models to "forget" specific training data without full retraining.

Standardized benchmark for evaluating adversarial robustness of ML models. Leaderboard of most robust models.

Benchmark measuring whether language models generate truthful answers. Tests for common misconceptions and falsehoods.

Industry coalition implementing C2PA. Open-source tools for content credentials. Practical implementation questions about provenance at scale.

Largest public AI red teaming event. 2,200+ participants testing multiple foundation models. Established community norms for responsible AI red teaming. Good for questions on practical red team methodology.

Analysis of risks specific to AI agents: tool use, chain-of-thought exploitation, multi-step task failures, delegation risks. Key for understanding why agents create new attack surfaces beyond single-turn interactions.

Crowdsourced red teaming methodology with 38,961 attacks across multiple models. Taxonomy of harmful outputs and effectiveness of different red teaming strategies. Key reference for structured AI red teaming.

Anthropic's framework for responsible AI development. Defines AI Safety Levels (ASL) and capability thresholds.

Anthropic's approach to AI alignment using a set of principles (a "constitution") to train helpful and harmless AI. Foundation of modern RLHF alternatives.

Demonstrated that long-context LLMs can be jailbroken by providing many examples of the desired behavior. Scales with context window size.

Anthropic's open protocol for connecting AI models to external tools and data sources. Critical reading for agentic AI security.

Technical standard for content provenance. Cryptographic binding of creation metadata to content. The leading technical approach to synthetic media authentication. Questions on architecture, limitations, and adoption challenges.

Comprehensive taxonomy of AI risks: weaponization, misinformation, power concentration, value lock-in, rogue AI. Good for strategic-level safety questions beyond technical alignment.

Official Kubernetes documentation on securing clusters, pods, and workloads. Essential for ML infrastructure security.

Framework for analyzing and countering disinformation. Provides a structured approach to information manipulation threats.

(See cross-cutting.md.) For C7 specifically: conformity assessments, technical documentation requirements, post-market monitoring, fundamental rights impact assessments. Detailed compliance questions.

Law enforcement perspective on deepfake threats: evidence tampering, identity fraud, CEO fraud, CSAM. Policy and response frameworks.

Annual trends report. AI trust, risk, and security management (AI TRiSM) has been featured prominently. Good for strategic-level questions about where the industry is heading.

Positions AI security technologies on the hype cycle. Useful for questions about technology maturity, adoption timelines, and distinguishing hype from operational readiness.

Analysis of how LLMs can amplify influence operations: cost reduction, scalability, personalization, multilingual content. Framework for assessing disinformation risk from generative AI.

Open-source DP libraries and practical guides. Bridges theory to implementation. Good for questions on real-world DP deployment challenges and privacy budget management.

Google's conceptual framework for securing AI systems. Covers supply chain, data governance, and deployment security.

Research on reward modeling, debate, recursive reward modeling, and interpretability. Provides an alternative perspective to Anthropic/OpenAI approaches.

Framework for evaluating dangerous capabilities: persuasion, deception, cyber operations, self-replication. Defines evaluation methodology for frontier model safety. Questions on what to test and how to interpret results.

Google DeepMind's watermarking technology for AI-generated content. Embeds imperceptible watermarks in images, audio, and text.

Extracted training data from ChatGPT (production model) using a divergence attack. Showed alignment doesn't prevent memorization. Questions on the gap between safety fine-tuning and data protection.

Security best practices for using Hugging Face Hub — model scanning, SafeTensors, access controls, and supply chain considerations.

Comprehensive library for adversarial ML. Supports attacks, defenses, and robustness evaluation across multiple ML frameworks.

Discovered 100+ malicious models on Hugging Face exploiting pickle deserialization for code execution. Real-world evidence of AI supply chain attacks. Good for scenario-based questions.

Microsoft's tool for assessing the security of ML models. Supports evasion, extraction, and inversion attacks.

Practical lessons from large-scale LLM red teaming across real products. Covers failure modes, testing methodologies, and organizational patterns. Rare insight into enterprise-scale AI security.

The theoretical foundation for differential privacy. Essential for questions on privacy-preserving ML training (DP-SGD) and the epsilon-delta framework.

Landmark study: false news spreads farther, faster, deeper than true news on social media. Not AI-specific but foundational for understanding why AI-generated disinformation is dangerous.

Companion to AI RMF 1.0 specifically for generative AI. Maps 12 GenAI risks to RMF actions. Covers CBRN, CSAM, confabulation, data privacy, environmental, human-AI interaction, information integrity, IP, obscenity, toxicity, value chain.

(See cross-cutting.md for details.) The primary AI governance framework for US context. Questions should test practical application of Govern/Map/Measure/Manage, not just recall.

Extending software bill of materials concepts to AI: model cards, data cards, training provenance. Emerging standard for AI supply chain transparency.

GPU cluster security, multi-tenant GPU isolation, model serving infrastructure hardening. Vendor-specific but covers unique infrastructure challenges (GPU memory isolation, CUDA vulnerabilities) not covered elsewhere.

Framework for agentic AI governance: scope control, human oversight, auditability, containment. Defines key properties agents should have and failure modes to prevent.

Description of external red teaming program and findings from GPT-4 pre-deployment testing. The system card details risk categories, testing methodology, and residual risks.

Research on the core alignment challenge: can weaker systems supervise stronger ones? Showed partial generalization is possible. Key for superalignment and scalable oversight questions.

Framework for ensuring the integrity of software artifacts throughout the supply chain. Applicable to ML model pipelines.

Extension of the LLM Top 10 specifically for agentic patterns. Covers excessive agency, insecure plugin/tool design, and multi-agent trust boundaries.

OWASP guidance on securing agentic AI systems — tool use, delegation chains, memory poisoning, and multi-agent architectures.

Top 10 security risks specific to machine learning systems, including supply chain attacks, data poisoning, and model theft.

Certification program for responsible AI. Assessment criteria across fairness, explainability, accountability, robustness. Emerging industry certification.

Research group studying abuse in information technologies, including AI-enabled disinformation, platform manipulation, and election interference.

Comprehensive annual data on AI progress: research output, investment, policy, public opinion, technical performance. The best source for quantitative AI landscape questions.

Security audit firm with deep AI/ML expertise. Published research on pickle deserialization attacks, model file format security, and ML pipeline vulnerabilities. Technical depth from a security-first perspective.

Large-scale benchmark dataset and tools for detecting facial manipulation in images and video. Used for deepfake detection research.

Historical survey tracing adversarial ML from 2004 spam filters through deep learning. Essential for questions on the evolution and taxonomy of adversarial attacks (evasion, poisoning, model extraction).

Extended training data extraction to image models. Showed Stable Diffusion memorizes and regurgitates training images. Important for multimodal AI data security questions.

The RLHF paper that enabled ChatGPT-style alignment. Reward model from human preferences + PPO. Foundational for understanding modern alignment approaches and their limitations.

Survey of tool-using, retrieval-augmented, and reasoning LMs. The architectural foundation for understanding agent capabilities and their security implications.

Comprehensive survey covering generation techniques (autoencoders, GANs, diffusion), detection approaches (visual artifacts, frequency analysis, physiological signals), and the arms race dynamic.

Largest prompt injection competition dataset. Taxonomy of prompt injection techniques: context ignoring, fake completion, payload splitting, obfuscation. Empirical data on attack success rates across models.

Benchmark dataset and detection methods for facial manipulation. Covers DeepFakes, Face2Face, FaceSwap, NeuralTextures. Standard reference for deepfake detection evaluation.

ToolEmu framework for evaluating agent risks in sandboxed environments. 36 risk categories across tool use failures. Practical methodology for agent security testing questions.

Systematic analysis of jailbreak techniques: competing objectives and mismatched generalization. Framework for understanding why safety training is inherently incomplete. Essential for nuanced jailbreak questions.