Technical AI Safety Curriculum

How modern language models work, and why aligning them with what we actually want is hard: specification gaming, reward hacking, and how failures could play out.

• 3Blue1Brown — Large Language Models explained briefly
• Rational Animations — Specification Gaming: How AI Can Turn Your Wishes Against You
• Ajeya Cotra — Why AI Alignment could be hard with modern deep learning
• METR — Recent Reward Hacking
• AI In Context — If you remember one AI disaster, make it this one
• Paul Christiano — What Failure Looks Like

Inner alignment and mesa-optimizers, and Constitutional AI as an approach to training models to follow written principles.

• Robert Miles — The OTHER AI Alignment Problem: Mesa-Optimizers and Inner Alignment
• Anthropic — Constitutional AI: Harmlessness from AI Feedback

How models are trained on human feedback, whether that feedback can scale to hard tasks, and evidence of models faking alignment or scheming.

• A simple technical explanation of RLH(AI)F~12 min
• Can we scale human feedback for complex AI tasks?20-25 min
• Alignment faking in large language models15 min
• Detecting and reducing scheming in AI models~10-12 min
• Current AIs seem pretty misaligned to meOPTIONAL
• The anti-scheming paperOPTIONAL

An introduction to machine learning: neural networks and backpropagation, plus optional background on supervised, self-supervised and reinforcement learning, and transformers.

• 3Blue1Brown — Neural Networks, Chapters 1–62 hours
• Richard Ngo — A Short Introduction to Machine LearningOPTIONAL
• Maini and Sabri — Machine Learning for Humans, Part 2.1: Supervised LearningOPTIONAL
• CodeBasics — What Is Self-Supervised Learning?OPTIONAL
• Hado van Hasselt — Introduction to Reinforcement Learning (watch 2:00–1:02:10)OPTIONAL · 60 min
• Andrej Karpathy — The Spelled-Out Intro to Neural Networks and Backpropagation: Building MicrogradOPTIONAL · 150 min
• Brandon Rohrer — Transformers from ScratchOPTIONAL