Deep Reinforcement Learning from Human Preferences
Christiano et al. established preference-based reward modeling, the foundational method that RLHF alignment pipelines later built on to steer language model behavior.
Large language models
Key papers and explainers on large language models—how they work, what they can do, and why that matters for safety.
Browse the full interactive library →
Language Models are Unsupervised Multitask Learners (GPT-2)
GPT-2 showed that scaling data and parameters unlocks broad capabilities without task-specific supervision, raising early alarm about dual-use and misuse potential.
Language Models are Few-Shot Learners (GPT-3)
GPT-3 demonstrated in-context learning at scale, forcing the field to rethink assumptions about what pretrained models can do and compressing alignment timelines.
Scaling Laws for Neural Language Models
Kaplan et al. quantified predictable performance scaling with compute, data, and parameters, enabling labs to forecast capability jumps and estimate safety lead time.
Instruct-GPT-3
OpenAI showed that instruction tuning with RLHF can transform a raw next-token predictor into a helpful, more controllable assistant, proving alignment interventions work at scale.
TruthfulQA
TruthfulQA exposed how language models confidently repeat popular falsehoods, establishing a benchmark for measuring truthfulness as distinct from fluency.
Chain-of-Thought Prompting
Prompted intermediate reasoning unlocked substantial gains on complex tasks, but also revealed that reasoning chains can be unfaithful to the model's actual computation.
Training Compute-Optimal Large Language Models (Chinchilla)
Chinchilla reframed scaling laws by showing optimal performance requires balancing model size and training tokens, redirecting how labs plan capability and safety investment.
Emergent Abilities of LLMs
Wei et al. documented capability discontinuities appearing at key scale thresholds, raising concern that dangerous abilities could emerge unpredictably in larger models.
Discovering Latent Knowledge in Language Models Without Supervision
Burns et al. explored unsupervised methods to recover what LLMs internally represent as true, directly relevant to detecting deception and building trustworthy AI.
Red Teaming Language Models to Reduce Harms
Anthropic formalized red teaming for LLMs as a repeatable methodology, turning adversarial probing into a systematic process for discovering and cataloging misuse pathways.
Sparks of Artificial General Intelligence
Bubeck et al. documented broad GPT-4 capability jumps across domains, compressing alignment timelines and stress-testing whether current safety evaluations are sufficient.
Jailbroken
This paper catalogs prompt-based bypasses of LLM safety training, showing that many safeguards behave like brittle wrappers rather than deep behavioral changes.
Universal Adversarial Attacks
Simple adversarial suffixes can systematically bypass safety behavior across many models, revealing that current defenses are not robust against automated attack search.
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
Hubinger et al. demonstrated that LLMs can retain hidden malicious policies through standard safety training, providing the first empirical evidence that deceptive alignment persists.
ARENA (Alignment Research Engineer Accelerator)
Hands-on technical curriculum for skilling up in AI alignment research engineering, freely available online and covering deep learning fundamentals, transformer mechanistic interpretability, reinforcement learning, and LLM evaluations.
Co-Intelligence
Mollick offers a practical guide for working alongside current LLMs while understanding their jagged capability frontiers and failure modes.
The Diamond Age
Stephenson anticipated personalized AI tutors and their profound social effects decades before modern LLMs made them reality.
Robot & Frank
An elder-care robot builds a genuine bond with its user while following his instructions to commit crimes, showing what happens when the human directs the AI to break rules.
Eternal You
Startups use AI to resurrect the dead as chatbots and avatars, raising unsettling questions about consent, grief, and the consequences of deploying generative systems on the most vulnerable human moments.
Transformer Circuits
The home of mechanistic interpretability research, publishing detailed analyses of how transformer models represent and process information internally.
generative.ink
Essays on AI, alignment, and the philosophical implications of language models and generative systems.
EleutherAI Blog
Open-source ML research covering language model training, evaluation, and the safety considerations of making powerful models widely available.
Why AI Is Incredibly Smart and Shockingly Stupid | Yejin Choi | TED
Choi demystifies large language models by showing where they fail at basic reasoning and common sense, and argues for smaller systems trained on human norms and values.
[1hr Talk] Intro to Large Language Models
A widely praised technical primer on how LLMs work, ending with a clear tour of the security challenges—jailbreaks, prompt injection, and data poisoning—that make these systems hard to secure.