Where every claim in SecProve
comes from.

Evaluates model capabilities for autonomous cyber operations at each AI Safety Level (ASL). Defines thresholds where AI capability in offensive security requires additional safeguards. Key reference for responsible AI in offensive security.

NVIDIA's open-source toolkit for adding programmable guardrails to LLM applications. Supports input/output validation and topic control.

Comprehensive guide covering AI security threats, privacy risks, and practical controls for AI-powered applications.

Collection of Anthropic's published research on AI safety, alignment, interpretability, and security.

Five practical safety problems: avoiding side effects, reward hacking, scalable oversight, safe exploration, distributional shift. Still the canonical taxonomy for AI safety research questions.

Benchmark measuring whether language models generate truthful answers. Tests for common misconceptions and falsehoods.

Anthropic's framework for responsible AI development. Defines AI Safety Levels (ASL) and capability thresholds.

Anthropic's approach to AI alignment using a set of principles (a "constitution") to train helpful and harmless AI. Foundation of modern RLHF alternatives.

Comprehensive taxonomy of AI risks: weaponization, misinformation, power concentration, value lock-in, rogue AI. Good for strategic-level safety questions beyond technical alignment.

Google's conceptual framework for securing AI systems. Covers supply chain, data governance, and deployment security.

Research on reward modeling, debate, recursive reward modeling, and interpretability. Provides an alternative perspective to Anthropic/OpenAI approaches.

Framework for evaluating dangerous capabilities: persuasion, deception, cyber operations, self-replication. Defines evaluation methodology for frontier model safety. Questions on what to test and how to interpret results.

Description of external red teaming program and findings from GPT-4 pre-deployment testing. The system card details risk categories, testing methodology, and residual risks.

Research on the core alignment challenge: can weaker systems supervise stronger ones? Showed partial generalization is possible. Key for superalignment and scalable oversight questions.

The RLHF paper that enabled ChatGPT-style alignment. Reward model from human preferences + PPO. Foundational for understanding modern alignment approaches and their limitations.