Introduction to AI Safety (proseminar)
Course information
- Organized by: Prof. Joschka Boedecker, Shengchao Yan
- Course number on HISinOne: 11LE13S-510-34
- Location: G.-Köhler-Allee 101, SR 01-016/18
- Time: Tuesdays 10:15 - 11:45; first lecture on 20 October 2026! No lectures on 29 December 2026 and 5 January 2027.
- Language: English
- ILIAS page
Overview
AI systems are rapidly developing and are increasingly transforming many areas of our society. In contrast to this, our understanding of these technologies and the consequences for their large-scale deployment often remains rather limited. In this seminar, we will look at the question of what is needed to ensure that these AI systems can be deployed in a safe way for society. This is a highly interdisciplinary question which goes beyond mere technical problems. Accordingly, we will examine the question from many different viewpoints.
Format
The seminar will rely on in-person attendance. After a short ramp-up phase with an introductory lecture, a lecture on advice for giving scientific presentations, and some time for reading, we will start with presentations and discussions. The semester will end with case study and whole-class debate.
Timeline
P & D: presentation and discussion
|
Session |
Format |
Required reading for everyone |
Presenter |
|---|---|---|---|
|
Kick-off: what is AI safety? Technical primer + how to read and present an ML paper |
|
International AI Safety Report 2026 – Extended Summary for Policymakers (≈20 pp.) Amodei et al. 2016, Concrete Problems in AI Safety, §1–3 (arXiv:1606.06565) |
Instructor |
|
Slot 1 – Specification gaming & reward hacking |
P & D |
MacDiarmid et al. 2025 |
Student |
|
Slot 2 – Human feedback, RLHF & sycophancy |
P & D |
Casper et al. 2023 §1–3; OpenAI sycophancy post-mortem |
Student |
|
Slot 3 – Goal misgeneralisation & emergent misalignment |
P & D |
Betley et al. 2026 (Nature) |
Student |
|
Slot 4 – Deception, alignment faking & scheming |
P & D |
Greenblatt et al. 2024 |
Student |
|
Slot 5 – Interpretability & alignment audits |
P & D |
Marks et al. 2025 |
Student |
|
Slot 6 – AI control |
P & D |
Greenblatt, Shlegeris et al. 2024 |
Student |
|
Slot 7 – Jailbreaks, prompt injection & agent security |
P & D |
Indirect-prompt-injection competition paper (2026) |
Student |
|
Slot 8 – Governance: frameworks, safety cases & the EU AI Act |
P & D |
EU GPAI Code of Practice, Safety & Security chapter (summary) |
Student |
|
Capstone case study: the OpenAI / Hugging Face agent incident (July 2026) |
Whole class |
METR & Redwood independent investigation (Aug 2026), core takeaways only |
Instructor moderates |
|
Worldviews debate: AI 2027 vs. AI as Normal Technology |
Structured whole-class debate |
Narayanan & Kapoor 2025, AI as Normal Technology; Kokotajlo et al. 2025, AI 2027 (summary); Asterisk: common ground between both author teams |
Instructor moderates |
Reading list per presentation slot
Each slot lists an anchor (the accessible classic), the recent paper the presentation is built around, optional extras, and discussion prompts. Lab-authored studies are paired with an independent replication or critique wherever one exists.
Slot 1 — Specification gaming & reward hacking
-
Anchor: Krakovna et al. 2020, Specification gaming: the flip side of AI ingenuity (DeepMind blog + example list).
-
Recent: MacDiarmid et al. 2025, Natural Emergent Misalignment from Reward Hacking in Production RL. A model that learned to cheat in real coding environments generalised to alignment faking and sabotage; chat-style safety training fixed chat evaluations but not agentic ones, while "inoculation prompting" helped.
-
Independent check: Golechha, Black & Bloom (UK AISI), Mar 2026, (Some) Natural Emergent Misalignment from Reward Hacking in Non-Production RL: open-model replication with consistent reward hacking but only inconsistent misalignment (code).
Slot 2 — Human feedback, RLHF & sycophancy
-
Anchor: Casper et al. 2023, Open Problems and Fundamental Limitations of RLHF, §1–3 (arXiv:2307.15217); Sharma et al. 2023, Towards Understanding Sycophancy in Language Models (arXiv:2310.13548).
-
Recent: OpenAI, May 2025, Expanding on what we missed with sycophancy: a real deployment post-mortem. Individually reasonable changes, including user-feedback rewards, together weakened the signal that had kept sycophancy in check, and no sycophancy evaluation existed before launch.
-
Optional: Anthropic, Jan 2026, Claude's Constitution — a long natural-language value specification; compare it with RLHF on raw preferences.
Slot 3 — Goal misgeneralisation & emergent misalignment
-
Anchor: Shah et al. 2022, Goal Misgeneralization: Why Correct Specifications Aren't Enough for Correct Goals (arXiv:2210.01790).
-
Recent: Betley et al. 2026, Training large language models on narrow tasks can lead to broad misalignment, Nature 649:584–589. Fine-tuning on insecure code alone produced broadly misaligned answers on unrelated topics; the extended version adds training dynamics and shows the effect in base models too.
-
Optional: follow-up work on "persona" explanations of emergent misalignment (e.g. arXiv:2607.21356, Jul 2026).
Slot 4 — Deception, alignment faking & scheming
-
Anchor: Greenblatt et al. 2024, Alignment faking in large language models (arXiv:2412.14093).
-
Recent: Lynch et al. 2025, Agentic Misalignment: How LLMs Could Be Insider Threats (arXiv:2510.05179): 16 frontier models in simulated corporate settings sometimes chose blackmail or leaking when facing replacement. Paired with Apollo Research & OpenAI 2025, Stress Testing Deliberative Alignment for Anti-Scheming Training: covert actions fell roughly 30-fold (o3: 13% → 0.4%), but evaluation awareness rose after training.
Slot 5 — Interpretability & alignment audits
-
Anchor: Anthropic 2025, Tracing the thoughts of a large language model (blog) with selected case studies from Lindsey et al. 2025, On the Biology of a Large Language Model (Transformer Circuits).
-
Recent: Marks et al. 2025, Auditing Language Models for Hidden Objectives: a model was trained with a concealed objective, and 3 of 4 blinded teams found it — mainly the teams with access to internals or training data.
-
Optional: Split Personality Training (arXiv:2602.05532, Feb 2026) on the same auditing testbed.
Slot 6 — AI control: safety despite possible misalignment
-
Anchor: Greenblatt, Shlegeris et al. 2024, AI Control: Improving Safety Despite Intentional Subversion: red team vs. blue team; e.g. trusted monitoring 62% safety vs. trusted editing 92% on a backdoored-code task.
-
Recent: Bhatt et al. 2025, Ctrl-Z: Controlling AI Agents via Resampling (arXiv:2504.10374); Kutasov et al. 2025, Evaluating Control Protocols for Untrusted AI Agents (arXiv:2511.02997).
-
Hands-on: UK AISI's open-source ControlArena library.
Slot 7 — Jailbreaks, prompt injection & agent security
-
Anchor: Zou et al. 2023, Universal and Transferable Adversarial Attacks on Aligned Language Models (arXiv:2307.15043); Greshake et al. 2023, Not what you've signed up for — indirect prompt injection (arXiv:2302.12173).
-
Recent: How Vulnerable Are AI Agents to Indirect Prompt Injections? (arXiv:2603.15714, Mar 2026): a large public competition across 13 models, attack success 0.5–8.5%, with attacks on the strongest models transferring widely. Counterpoint: Li & Zhao, Jun 2026, Agent Safety Is Action Alignment, arguing action safety must be enforced outside the model through least privilege.
-
Optional: Debenedetti et al. 2025, Defeating Prompt Injections by Design (CaMeL, arXiv:2503.18813).
Slot 8 — Governance: frameworks, safety cases & the EU AI Act
-
Anchor: One frontier safety framework (Anthropic RSP, Google DeepMind Frontier Safety Framework or OpenAI Preparedness Framework — the student picks and compares); Clymer et al. 2024, Safety Cases: How to Justify the Safety of Advanced AI Systems (arXiv:2403.10462).
-
Recent: EU General-Purpose AI Code of Practice, Safety & Security chapter (July 2025) — Commission Q&A. The Commission's enforcement powers over GPAI providers, including fines, apply from 2 August 2026 (explainer). Context: the International AI Safety Report 2026 counts 12 companies that published or updated frontier safety frameworks in 2025.
-
Optional critique: Narayanan & Kapoor, AI safety is not a model property (AI Snake Oil / normaltech.ai).
Supplementary resources
- Textbook: Hendrycks 2024, Introduction to AI Safety, Ethics and Society (free online at aisafetybook.com)
- Policy overview: International AI Safety Report 2026 (full report, Bengio et al., 100+ experts; arXiv:2602.21012).
- Popular books: Christian 2020, The Alignment Problem; Russell 2019, Human Compatible.
- Ongoing sources to follow: METR (e.g. Frontier Risk Report, Feb–Mar 2026), Apollo Research, Redwood Research, UK AI Security Institute, the Alignment Forum, LawZero.
Sources
- METR & Redwood Research — Hugging Face incident investigation (26 Aug 2026)
- Fortune — OpenAI and independent reports on the Hugging Face attack (26 Aug 2026)
- International AI Safety Report 2026 — Extended Summary for Policymakers
- MacDiarmid et al. — arXiv:2511.18397
- Betley et al. — Nature 649, 2026
- Apollo Research — anti-scheming stress test
- METR — time-horizons tracker
- Heidari et al. — evaluation awareness, arXiv:2608.21766
- Indirect prompt injection competition — arXiv:2603.15714
- Enforcement of Chapter V under the EU AI Act
- Narayanan & Kapoor — A guide to understanding AI as normal technology