Sie sind hier: Startseite › Teaching › WS2026/27 › Introduction to AI Safety …

Introduction to AI Safety (proseminar)

Course information

 

  • Organized by:  Prof. Joschka Boedecker, Shengchao Yan
  • Course number on HISinOne:  11LE13S-510-34
  • Location: G.-Köhler-Allee 101, SR 01-016/18
  • Time: Tuesdays 10:15 - 11:45; first lecture on 20 October 2026! No lectures on 29 December 2026 and 5 January 2027.
  • Language: English
  • ILIAS page

 

Overview

AI systems are rapidly developing and are increasingly transforming many areas of our society. In contrast to this, our understanding of these technologies and the consequences for their large-scale deployment often remains rather limited. In this seminar, we will look at the question of what is needed to ensure that these AI systems can be deployed in a safe way for society. This is a highly interdisciplinary question which goes beyond mere technical problems. Accordingly, we will examine the question from many different viewpoints.

Format

The seminar will rely on in-person attendance. After a short ramp-up phase with an introductory lecture, a lecture on advice for giving scientific presentations, and some time for reading, we will start with presentations and discussions. The semester will end with case study and whole-class debate.

 

Timeline

 P & D: presentation and discussion

Session

Format

Required reading for everyone

Presenter

Kick-off: what is AI safety? Technical primer  + how to read and present an ML paper

 

International AI Safety Report 2026 – Extended Summary for Policymakers (≈20 pp.)

Amodei et al. 2016, Concrete Problems in AI Safety, §1–3 (arXiv:1606.06565)

Instructor

Slot 1 – Specification gaming & reward hacking

P & D

MacDiarmid et al. 2025

 Student

Slot 2 – Human feedback, RLHF & sycophancy

P & D

Casper et al. 2023 §1–3; OpenAI sycophancy post-mortem

 Student

Slot 3 – Goal misgeneralisation & emergent misalignment

P & D

Betley et al. 2026 (Nature)

 Student

Slot 4 – Deception, alignment faking & scheming

P & D

Greenblatt et al. 2024

 Student

Slot 5 – Interpretability & alignment audits

P & D

Marks et al. 2025

 Student

Slot 6 – AI control

P & D

Greenblatt, Shlegeris et al. 2024

 Student

Slot 7 – Jailbreaks, prompt injection & agent security

P & D

Indirect-prompt-injection competition paper (2026)

 Student

Slot 8 – Governance: frameworks, safety cases & the EU AI Act

P & D

EU GPAI Code of Practice, Safety & Security chapter (summary)

 Student

Capstone case study: the OpenAI / Hugging Face agent incident (July 2026)

Whole class

METR & Redwood independent investigation (Aug 2026), core takeaways only

Instructor moderates

Worldviews debate: AI 2027 vs. AI as Normal Technology

Structured whole-class debate

Narayanan & Kapoor 2025, AI as Normal Technology; Kokotajlo et al. 2025, AI 2027 (summary); Asterisk: common ground between both author teams

Instructor moderates

Reading list per presentation slot

Each slot lists an anchor (the accessible classic), the recent paper the presentation is built around, optional extras, and discussion prompts. Lab-authored studies are paired with an independent replication or critique wherever one exists.

Slot 1 — Specification gaming & reward hacking

Slot 2 — Human feedback, RLHF & sycophancy

  • Anchor: Casper et al. 2023, Open Problems and Fundamental Limitations of RLHF, §1–3 (arXiv:2307.15217); Sharma et al. 2023, Towards Understanding Sycophancy in Language Models (arXiv:2310.13548).

  • Recent: OpenAI, May 2025, Expanding on what we missed with sycophancy: a real deployment post-mortem. Individually reasonable changes, including user-feedback rewards, together weakened the signal that had kept sycophancy in check, and no sycophancy evaluation existed before launch.

  • Optional: Anthropic, Jan 2026, Claude's Constitution — a long natural-language value specification; compare it with RLHF on raw preferences.

Slot 3 — Goal misgeneralisation & emergent misalignment

  • Anchor: Shah et al. 2022, Goal Misgeneralization: Why Correct Specifications Aren't Enough for Correct Goals (arXiv:2210.01790).

  • Recent: Betley et al. 2026, Training large language models on narrow tasks can lead to broad misalignment, Nature 649:584–589. Fine-tuning on insecure code alone produced broadly misaligned answers on unrelated topics; the extended version adds training dynamics and shows the effect in base models too.

  • Optional: follow-up work on "persona" explanations of emergent misalignment (e.g. arXiv:2607.21356, Jul 2026).

Slot 4 — Deception, alignment faking & scheming

  • Anchor: Greenblatt et al. 2024, Alignment faking in large language models (arXiv:2412.14093).

  • Recent: Lynch et al. 2025, Agentic Misalignment: How LLMs Could Be Insider Threats (arXiv:2510.05179): 16 frontier models in simulated corporate settings sometimes chose blackmail or leaking when facing replacement. Paired with Apollo Research & OpenAI 2025, Stress Testing Deliberative Alignment for Anti-Scheming Training: covert actions fell roughly 30-fold (o3: 13% → 0.4%), but evaluation awareness rose after training.

Slot 5 — Interpretability & alignment audits

  • Anchor: Anthropic 2025, Tracing the thoughts of a large language model (blog) with selected case studies from Lindsey et al. 2025, On the Biology of a Large Language Model (Transformer Circuits).

  • Recent: Marks et al. 2025, Auditing Language Models for Hidden Objectives: a model was trained with a concealed objective, and 3 of 4 blinded teams found it — mainly the teams with access to internals or training data.

  • Optional: Split Personality Training (arXiv:2602.05532, Feb 2026) on the same auditing testbed.

Slot 6 — AI control: safety despite possible misalignment

  • Anchor: Greenblatt, Shlegeris et al. 2024, AI Control: Improving Safety Despite Intentional Subversion: red team vs. blue team; e.g. trusted monitoring 62% safety vs. trusted editing 92% on a backdoored-code task.

  • Recent: Bhatt et al. 2025, Ctrl-Z: Controlling AI Agents via Resampling (arXiv:2504.10374); Kutasov et al. 2025, Evaluating Control Protocols for Untrusted AI Agents (arXiv:2511.02997).

  • Hands-on: UK AISI's open-source ControlArena library.

Slot 7 — Jailbreaks, prompt injection & agent security

  • Anchor: Zou et al. 2023, Universal and Transferable Adversarial Attacks on Aligned Language Models (arXiv:2307.15043); Greshake et al. 2023, Not what you've signed up for — indirect prompt injection (arXiv:2302.12173).

  • Recent: How Vulnerable Are AI Agents to Indirect Prompt Injections? (arXiv:2603.15714, Mar 2026): a large public competition across 13 models, attack success 0.5–8.5%, with attacks on the strongest models transferring widely. Counterpoint: Li & Zhao, Jun 2026, Agent Safety Is Action Alignment, arguing action safety must be enforced outside the model through least privilege.

  • Optional: Debenedetti et al. 2025, Defeating Prompt Injections by Design (CaMeL, arXiv:2503.18813).

Slot 8 — Governance: frameworks, safety cases & the EU AI Act

  • Anchor: One frontier safety framework (Anthropic RSP, Google DeepMind Frontier Safety Framework or OpenAI Preparedness Framework — the student picks and compares); Clymer et al. 2024, Safety Cases: How to Justify the Safety of Advanced AI Systems (arXiv:2403.10462).

  • Recent: EU General-Purpose AI Code of Practice, Safety & Security chapter (July 2025) — Commission Q&A. The Commission's enforcement powers over GPAI providers, including fines, apply from 2 August 2026 (explainer). Context: the International AI Safety Report 2026 counts 12 companies that published or updated frontier safety frameworks in 2025.

  • Optional critique: Narayanan & Kapoor, AI safety is not a model property (AI Snake Oil / normaltech.ai).




Supplementary resources

  • Textbook: Hendrycks 2024, Introduction to AI Safety, Ethics and Society (free online at aisafetybook.com) 
  • Policy overview: International AI Safety Report 2026 (full report, Bengio et al., 100+ experts; arXiv:2602.21012).
  • Popular books: Christian 2020, The Alignment Problem; Russell 2019, Human Compatible.
  • Ongoing sources to follow: METR (e.g. Frontier Risk Report, Feb–Mar 2026), Apollo Research, Redwood Research, UK AI Security Institute, the Alignment Forum, LawZero.

Sources