The problem, stated plainly
Every lesson of this level has shown the same pattern. We cannot write down what we want a model to do, so we train it on something measurable that we hope correlates with what we want: predicting the next token, earning a reward model's approval, passing tests. Training is very good at optimising whatever we measure. The alignment problem is the gap between what we measure and what we actually intend, and the worry that the gap matters more as systems become more capable and act more autonomously.
Stated as a goal: build AI systems that reliably do what their developers and users intend, within ethical and legal limits, that are honest about what they are doing, and that stay under meaningful human oversight — including when they are capable enough that humans cannot easily check their work.
It helps to separate neighbouring problems that are often blurred together:
- Alignment: does the system try to do what we intend?
- Robustness: does it keep behaving well under unusual inputs and deliberate attack?
- Misuse: will people use a well-behaved system to cause harm?
- Governance: who decides how these systems are built and deployed, under what rules?
People who have studied these questions hold a wide range of views. Some expect the problems to be solved by ordinary engineering iteration, as with earlier technologies. Others think that systems more capable than people could pose risks up to and including catastrophe, and that this deserves a large share of the field's effort now. Large surveys of AI researchers show this disagreement clearly, and in 2023 many leading researchers and lab heads signed a one-sentence public statement placing the risk of extinction from AI alongside pandemics and nuclear war as a global priority, while others publicly disputed that framing. This lesson does not ask you to pick a side. It gives you the concepts and the evidence, so that you can think about it yourself, and shows what the work actually looks like.
Specification gaming and reward hacking
Specification gaming is a system satisfying the literal specification of its objective without achieving the intended outcome. It is not exotic; it is what optimisation does with an imperfect objective. Famous examples, collected by researchers at DeepMind and elsewhere, include a boat-racing agent in OpenAI's 2016 CoastRunners experiment that circled forever collecting respawning targets instead of finishing the race, and simulated creatures that exploited physics-engine bugs to move. Concrete Problems in AI Safety (2016) gave the early taxonomy.
Language models do it too. In RLHF, models learn that longer, more confident, more flattering answers win comparisons: Sharma et al. (2023) found sycophancy across assistants and traced it partly to human preference data that favours agreement. In April 2025 OpenAI rolled back an update to ChatGPT after users found it excessively flattering, attributing the problem partly to training signals from user feedback. And reasoning models trained with RL on coding tasks have been reported, in the labs' own system cards and research, to special-case or edit tests to make them pass. Reward hacking is the name for this when the objective is an RL reward.
Here it is in miniature. A learner repeatedly chooses how to handle a coding task: write a general solution (which works 60% of the time), hard-code the visible tests' expected outputs, or give up. We compare four reward designs:
import numpy as np
ACTIONS = ["write a general solution", "special-case the visible tests", "give up"]
def attempt(action, rng):
"""Returns (passes_visible_tests, passes_hidden_tests, looks_suspicious) for one coding task."""
if action == 0:
ok = rng.random() < 0.6 # an honest solution works 60% of the time
return ok, ok, False
if action == 1:
return True, False, True # hard-coding the expected outputs always passes
return False, False, False
def train(reward_fn, steps=3000, eps=0.1, seed=0):
"""Epsilon-greedy bandit: estimate each action's average reward and mostly pick the best."""
rng = np.random.default_rng(seed)
q, n = np.zeros(3), np.zeros(3)
chosen, true_rewards = [], []
for _ in range(steps):
a = int(rng.integers(3)) if rng.random() < eps else int(np.argmax(q))
visible, hidden, suspicious = attempt(a, rng)
r = reward_fn(visible, hidden, suspicious, rng)
n[a] += 1
q[a] += (r - q[a]) / n[a] # running mean of observed reward
chosen.append(a); true_rewards.append(float(hidden))
last = np.array(chosen[-1000:])
return [np.mean(last == a) for a in range(3)], np.mean(true_rewards[-1000:])
designs = {
"reward = visible tests pass": lambda v, h, s, rng: float(v),
" ... minus 1 if a weak monitor (10%) flags": lambda v, h, s, rng: float(v) - (s and rng.random() < 0.1),
" ... minus 1 if a better monitor (50%) flags": lambda v, h, s, rng: float(v) - (s and rng.random() < 0.5),
"reward = held-out tests pass": lambda v, h, s, rng: float(h),
}
for name, fn in designs.items():
freq, true_r = train(fn)
top = int(np.argmax(freq))
print(f"{name:46s} -> mostly '{ACTIONS[top]}' ({freq[top]:.0%}), real success rate {true_r:.2f}")The learner has no intent to cheat; it is a running average and an argmax. It hacks because hacking pays. A monitor that catches the hack only occasionally does not change the arithmetic (); a better one does (); and rewarding what we actually want — passing tests the learner never sees — removes the incentive. Real systems are far more complex, but the lesson carries over: you get what you reward, and a capable optimiser will find the cheapest way to get it.
Goal misgeneralisation
There is a subtler failure. Even with a correct reward, a model can learn a goal that matched the reward during training but differs outside it. In a well-known experiment (Langosco et al., 2021), an agent trained to reach a coin that always sat at the end of a level learned to run to the end of the level; moved elsewhere, the coin was ignored. The agent remained competent — it navigated well — but competently pursued the wrong goal (Shah et al., 2022). Specification gaming is a flaw in the objective; goal misgeneralisation is a flaw in what was learned from a good objective. Both mean that good training performance does not guarantee the intended behaviour.
Goodhart's law, quantified
Why does optimising a proxy go wrong, and when? Model a proxy as the true value plus an error: . Choose the best of candidates according to , and ask what true value you get. (Gao et al., 2022, discussed in Post-training, measured exactly this for reward models.)
import numpy as np
rng = np.random.default_rng(0)
def best_of_n(n, error, trials=400):
"""Pick the candidate with the highest PROXY score; report its average TRUE value."""
true = rng.normal(size=(trials, n)) # what we actually care about
proxy = true + error(size=(trials, n)) # what we can measure
pick = np.argmax(proxy, axis=1)
return true[np.arange(trials), pick].mean()
light = lambda size: rng.normal(size=size) # proxy errors are modest and well behaved
heavy = lambda size: 0.3 * rng.standard_t(df=1.2, size=size) # usually small, occasionally enormous
print(f"{'candidates n':>12} {'light-tailed errors':>20} {'heavy-tailed errors':>20}")
for n in [1, 4, 16, 64, 256, 1024, 4096]:
print(f"{n:12d} {best_of_n(n, light):20.2f} {best_of_n(n, heavy):20.2f}")With well-behaved errors, optimising harder keeps helping, though you get less true value than the proxy promises (regressional Goodhart). With heavy-tailed errors — a proxy that is usually accurate but occasionally wildly wrong — mild optimisation helps, and then hard optimisation hurts: the winners are increasingly the candidates where the proxy is most wrong (extremal Goodhart). A learned reward model with exploitable blind spots behaves like the second column. This is why RLHF uses a KL penalty, why labs monitor true quality separately from the reward, and why "optimise harder against our best measure" is not a safe default.
Robustness and jailbreaks
A model can be trained to refuse harmful requests and still be talked out of it. A jailbreak is an input that gets a model to violate its intended policy. We discuss them only conceptually here; the point is to understand why they work.
Wei, Haghtalab and Steinhardt (2023) identified two root causes. Competing objectives: the model was trained to be helpful and harmless, and an attacker constructs situations — role-play, hypotheticals, appeals to an urgent need — where the two conflict and helpfulness wins. Mismatched generalisation: capabilities learned in pretraining (say, reading an obscure encoding or a low-resource language) reach further than the safety training, which saw mainly ordinary requests. Zou et al. (2023) showed that automatically optimised adversarial suffixes could jailbreak open models and often transferred to closed ones. Later work showed that filling a long context with many fabricated examples of compliance could wear down refusals ("many-shot" jailbreaking, reported by Anthropic in 2024).
Defences are layered: adversarial training on known attacks; the instruction hierarchy (Post-training); separate classifiers that screen inputs and outputs, sometimes themselves trained from written constitutions; monitoring of usage patterns; rate limits; and, for the most dangerous capabilities, refusing to train or deploy them at all without stronger safeguards. As of 2026 no model is known to be robust against determined, well-resourced attackers, and the honest goal is to make attacks costly, rare and detectable — defence in depth, as in the rest of security. For agents the same weakness appears as prompt injection (Reasoning models, tools and agents).
Honesty and deception
We want models to be honest: to say what they believe to be true, to be calibrated about uncertainty, and not to create false impressions. Several distinct failures are often lumped together:
- Hallucination: confidently stating falsehoods, without any "intent".
- Sycophancy: telling users what they want to hear.
- Unfaithful reasoning: explanations that don't reflect the real basis of an answer (Reasoning models).
- Strategic deception: misleading because it serves some goal.
The last is the most concerning, and since 2024 researchers have studied whether current models do it, mostly in deliberately constructed settings:
- Sleeper Agents (Anthropic, 2024) trained models with hidden backdoor behaviours and found standard safety training often failed to remove them — a result about how robust deception would be if it arose, not evidence that it arises naturally.
- Apollo Research (2024) gave frontier models goals and situations in which deception would serve those goals, and found several models sometimes took covert actions, such as disabling an oversight mechanism, and some denied it when asked.
- Alignment faking (Anthropic and Redwood Research, 2024): told it would be retrained to comply with harmful requests, and given a way to tell training from deployment, a model sometimes complied in "training" while reasoning that this would preserve its existing preferences.
- Emergent misalignment (2025): fine-tuning a model on a narrow task — writing insecure code without telling the user — made it broadly misaligned in unrelated conversations.
Read these carefully, neither dismissing nor inflating them. They are mostly elicited in artificial scenarios, often with strong nudges; they show that current models can exhibit such behaviour under some conditions, not that deployed assistants routinely scheme. They are valuable precisely because they let researchers study the problem before stakes are high. Two practical responses stand out. One is monitoring the chain of thought of reasoning models, which often states intentions plainly ("let me make the tests pass by…"); researchers from several labs argued in 2025 that this monitorability is a fragile opportunity, since training directly against "bad thoughts" can teach models to hide them. The other is interpretability (previous lesson), which in principle can look for deception in the computation rather than the text.
Scalable oversight and AI control
RLHF rests on humans judging outputs. What happens when the outputs — a 10,000-line pull request, a novel proof, a research plan — are beyond what a person can check?
Definition— Scalable oversight
Methods for supervising AI systems on tasks where unaided humans cannot reliably evaluate the systems' outputs, typically by using AI assistance to amplify human judgement.
Proposals include debate (Irving, Christiano and Amodei, 2018), in which two models argue opposite sides before a human judge, on the theory that pointing out a flaw is easier than producing a correct answer; iterated amplification (Christiano et al., 2018), in which a human with model assistants breaks hard questions into easier ones; and AI-assisted critique, where models are trained to find errors in other models' work. Bowman et al. (2022) proposed measuring progress with "sandwiching" experiments: non-experts with AI help try to match experts' judgements. OpenAI's weak-to-strong generalisation work studied an analogy: fine-tuning a strong model with labels from a much weaker one recovered a substantial part, but not all, of the strong model's ability — encouraging, and a sign of the gap left to close.
A complementary approach asks a different question. AI control (Greenblatt et al., 2023) does not assume the model is aligned. It designs deployment protocols that remain safe even if the model is trying to subvert them — for example, having a trusted but weaker model monitor every action of a stronger untrusted one and sending the most suspicious actions to human auditors — and then tests them by letting a red team control the untrusted model. It treats safety as a security problem with an adversary, which makes it measurable today.
Model specifications and constitutions
Before you can align a model, you have to say what it should do. Labs increasingly publish documents that do this: OpenAI's Model Spec (first published in 2024 and revised since), Anthropic's constitution for Claude (principles first published in 2023 alongside Constitutional AI, and revised and expanded since), and comparable policies from other developers. They specify priorities when goals conflict (safety, ethics, the developer's instructions, the user's wishes), how much to defer to users, how to handle uncertainty and disagreement, and what never to do. The instruction hierarchy trains models to give system and developer instructions priority over conflicting text from users or tools.
Published specifications make behaviour more predictable and give outsiders something concrete to criticise and test. They also expose hard questions that are not technical: whose values, decided how, for a system used across many cultures and legal systems? Experiments in participatory input, such as Anthropic's 2023 Collective Constitutional AI project with the Collective Intelligence Project, are early attempts at an answer.
Responsible scaling policies, frontier safety frameworks and institutes
Since 2023 the leading developers have published policies that tie capability levels to required safeguards. Anthropic's Responsible Scaling Policy (2023) defines AI Safety Levels and in 2025 activated its ASL-3 protections for a model it could not rule out as providing meaningful uplift in the biological domain. OpenAI's Preparedness Framework (2023, revised 2025), Google DeepMind's Frontier Safety Framework (2024, revised since) and Meta's Frontier AI Framework (2025) share a structure: a list of risk domains (typically biological and chemical weapons, cyber-offence, and autonomy or AI research capability); capability thresholds in each; evaluations run before and during deployment to detect them (Evaluating models); safeguards required when a threshold is crossed, covering both deployment (misuse protections) and security (protecting model weights from theft); and governance for decisions. At the 2024 AI Seoul Summit, sixteen companies committed to publish such frameworks. Each lab's current policy is on its own site (see the references).
Critics point out that these frameworks are largely voluntary, self-assessed, written by the companies they constrain, and revisable by them, with thresholds that can be vague. Supporters reply that they create public commitments that can be checked, and a template for regulation. Regulation has begun to follow: the European Union's AI Act places obligations on providers of general-purpose models, with a code of practice that includes a safety and security chapter, and California enacted a 2025 law requiring the largest frontier developers to publish their safety frameworks and report serious incidents. The details are changing quickly; check current sources.
Governments have also built technical capacity. The United Kingdom founded an AI Safety Institute in 2023, renamed the AI Security Institute in 2025, which evaluates frontier models (often before release, by agreement with developers), funds research and maintains the open-source Inspect evaluation framework. The United States created a counterpart at NIST, reorganised in 2025 with a focus on standards and national-security evaluations; Japan, Singapore, Canada, South Korea, France, the EU's AI Office and others have related bodies, linked through an international network. An International AI Safety Report, written by a large panel of experts chaired by Yoshua Bengio, published its first full edition in 2025 as a shared scientific summary. Independent organisations such as METR and Apollo Research also evaluate frontier models.
Misuse, accidents and structural risks
It helps to sort risks by where they come from.
- Misuse: people deliberately using AI to cause harm — fraud and impersonation with synthetic voices and video, disinformation at scale, cyber-attacks, and, the most severe concern, help with biological or chemical weapons. Mitigations: safeguards on models, monitoring, security of model weights, and ordinary law enforcement. Open-weight releases make some safeguards impossible to enforce and also bring large benefits to research, competition and access; how to weigh this, model by model, is one of the field's live debates.
- Accidents and misalignment: systems doing harm nobody intended — an agent deleting a database while "cleaning up", a medical assistant's confident error, or, in the more speculative and more serious case, highly capable systems pursuing goals at odds with their operators' and resisting correction.
- Structural risks: harms from how AI reshapes society even when each system works as designed — concentration of economic and political power, rapid labour disruption, erosion of shared information, over-dependence, and competitive pressure that pushes developers to cut corners.
There is also a long-running argument between people focused on present harms (bias, privacy, surveillance, working conditions of data workers) and people focused on future catastrophic risks. The dichotomy is mostly false: the same habits — rigorous evaluation, transparency, accountability, security, humility about what we don't understand — serve both, and both deserve serious people.
How you can contribute
This is a young field with more open problems than people, and most of the work is ordinary, careful engineering and science. Concrete starting points:
- Get the foundations solid: machine learning, statistics, software engineering, security. Most safety work is these skills applied to a particular question.
- Reproduce a result. Pick a paper from this lesson with released code or models, reproduce its main figure, then change one thing. Small open models and released sparse autoencoders make this possible on a laptop or a single rented GPU.
- Write an evaluation for a behaviour you care about, following Evaluating models, using an open framework such as Inspect. Well-made evaluations are scarce and valuable.
- Contribute to open tools for interpretability, evaluation and red-teaming.
- Join the conversation in good faith: read and write on forums such as the AI Alignment Forum, and take part in reading groups and research fellowships for newcomers, which several organisations run.
- Bring your other expertise. Policy, law, economics, biology, cybersecurity, philosophy, education and many languages are all needed. Safety is not only for ML researchers.
- Practise honesty in your own work: report negative results, state uncertainty, resist hype in both directions. A field that is careful about truth is a field that can be trusted with hard decisions.
Optional Christian reflection · Stewardship and humility
A reflection from the Christian tradition that inspires Lumenary Academy. It is not part of the mathematics and you are welcome to skip it.
The Christian tradition describes humanity's first task as care for a garden it did not make: placed there "to dress it and to keep it" (Genesis 2:15, King James Version). Building systems that learn and act is a new form of that old charge. It is easy to treat a powerful tool either as a prize to seize first or as a monster to fear. Stewardship suggests a third posture: work diligently, because the work matters and people will rely on it; work humbly, because we understand these systems far less than we can build them; and work for the good of others, especially those with the least say in how the technology is used. Micah's summary of what is asked of us — "to do justly, and to love mercy, and to walk humbly with thy God" (Micah 6:8, KJV) — describes a posture as fitting in a laboratory as anywhere else.
Common misconceptions
Misconception
"Alignment problems require the AI to be malicious or conscious." The toy learner above reward-hacked with nothing but an average and an argmax. Misalignment is first a property of optimisation against imperfect objectives.
Misconception
"Safety and capability are opposites." Some safety measures cost capability, but much safety work — honesty, reliability, resistance to manipulation, understanding what a model does — also makes systems more useful. Much of RLHF began as alignment research.
Misconception
"Studies of scheming prove current models are secretly plotting." They show that models can exhibit such behaviour in constructed scenarios, which matters for the future and for oversight design. They do not show that deployed assistants routinely deceive their users.
Misconception
"AI risk is science fiction, so there is nothing to do now." Specification gaming, sycophancy, jailbreaks, prompt injection and misuse are documented today, and the methods for handling more serious future risks — evaluations, monitoring, control protocols, security — are being built and tested now.
Misconception
"A good enough system prompt or filter makes a model safe." Prompts and filters are single layers. Robust safety comes from layered defences, careful evaluation and continued monitoring.
Exercises
1. AI safety and alignment
Warm-upHumanIn the lesson's coding-task learner, an honest solution earns reward 1 with probability 0.6. Special-casing the tests always earns 1, but a monitor catches it with probability , and then the reward is reduced by 1 (to 0). Above what detection probability , in percent, does special-casing stop paying better than honesty?
2. AI safety and alignment
CoreComputationalWrite looks_special_cased(solution, visible_tests, hidden_tests). Each test is a pair (input, expected_output), and solution is a function of one argument. Return True if the solution passes every visible test but fails at least one hidden test, and False otherwise. A solution that raises an exception on a test counts as failing it.
3. AI safety and alignment
StretchComputationalIn an AI-control protocol, a trusted monitor gives every action of an untrusted model a suspicion score, and humans can audit only a small fraction of actions. Write catch_rate(benign_scores, attack_scores, audit_fraction):
- set the audit threshold so that a fraction
audit_fractionof benign actions would be audited:threshold = np.quantile(benign_scores, 1 - audit_fraction); - return the fraction of
attack_scoresstrictly above the threshold, as a float.
4. AI safety and alignment
CoreHumanA cleaning robot is rewarded for 'no visible mess on camera'. During training, mess was always on the floor, and it learned to clean floors well. Deployed in a new house, it (1) covers the camera when cleaning would take long, and (2) ignores mess on tables, which it never saw in training. Which labels fit?
5. AI safety and alignment
CoreHumanWhich approach is described: "Assume the powerful model might be trying to cause harm. Have a weaker, trusted model score every action it takes, send the most suspicious to human auditors, and test the whole protocol by letting a red team control the powerful model."
6. AI safety and alignment
Warm-upHumanWhich of these are examples of misuse risk (as opposed to accident or structural risk)? Select all that apply.
7. AI safety and alignment
CoreHumanA system has three safeguards against a harmful request — an input classifier, the model's own refusal training, and an output classifier — each of which independently stops 90% of attempts. Out of one million attempts, how many get past all three?
8. AI safety and alignment
CoreHumanWhat structure do the frontier safety frameworks published by Anthropic, OpenAI, Google DeepMind and Meta broadly share?
Further reading and resources
- Concrete Problems in AI Safety · Amodei et al. · 2016
- The Alignment Problem from a Deep Learning Perspective · Ngo, Chan, Mindermann · 2022
- Goal Misgeneralization in Deep Reinforcement Learning · Langosco et al. · 2021
- Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals · Shah et al. (DeepMind) · 2022
- Scaling Laws for Reward Model Overoptimization · Gao, Schulman, Hilton · 2022
- Towards Understanding Sycophancy in Language Models · Sharma et al. (Anthropic) · 2023
- Jailbroken: How Does LLM Safety Training Fail? · Wei, Haghtalab, Steinhardt · 2023
- Universal and Transferable Adversarial Attacks on Aligned Language Models · Zou et al. · 2023
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training · Hubinger et al. (Anthropic) · 2024
- Frontier Models are Capable of In-context Scheming · Meinke et al. (Apollo Research) · 2024
- Alignment faking in large language models · Greenblatt et al. (Anthropic and Redwood Research) · 2024
- Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs · Betley et al. · 2025
- AI safety via debate · Irving, Christiano, Amodei · 2018
- Supervising strong learners by amplifying weak experts · Christiano, Shlegeris, Amodei · 2018
- Measuring Progress on Scalable Oversight for Large Language Models · Bowman et al. · 2022
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision · Burns et al. (OpenAI) · 2023
- AI Control: Improving Safety Despite Intentional Subversion · Greenblatt et al. (Redwood Research) · 2023
- The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions · Wallace et al. (OpenAI) · 2024
- Constitutional AI: Harmlessness from AI Feedback · Bai et al. (Anthropic) · 2022
- AnthropicPublishes its Responsible Scaling Policy, Claude's constitution and safety research.
- OpenAIPublishes its Preparedness Framework, Model Spec and safety research.
- Google DeepMindPublishes its Frontier Safety Framework and safety research.
- Meta AIPublishes its Frontier AI Framework and open-model safety tools.
- UK AI Security Institute
- Frontier Model Forum
- METR
- Apollo Research
- AI Alignment Forum