Aligning to Justice
Is the purpose of alignment to maintain injustice or bring about justice?
AI alignment is both a technical challenge and also a question of values. Large language models (LLMs) are trained to reflect (or avoid) certain perspectives. Most companies do this behind closed doors but Anthropic published the Claude’s “constitution”, which gives us a glimpse at the underlying values. That transparency deserves real praise. But when you read closely, many of the rules stop at surface-level politeness, rather than engaging with the deeper structural issues.
Alignment
The most common alignment technique is reinforcement learning from human feedback (RLHF), where humans rate outputs and the model learns to favor highly rated answers. It works reasonably well in practice but it can have some shortfalls. For example, raters might like being flattered so they rate those higher than truthful answers. It is also expensive to hire so many raters.
Constitutional training has a fixed rubric that tells a rating LLM how to score each message. In practice, the model outputs several messages, and then the rating LLM picks the best output for the one policy that it’s scoring. This is cheaper since it’s done without human raters. It’s also more consistent since the rubric doesn’t change subtly per human rater. It is also more transparent because the rubric (aka constitution) can be shared publicly.
Constitution training is used to train Anthropic’s Claude. Claude’s constitution is available online. Anthropic deserves credit for publishing their rubric. By contrast, most other LLM companies have not disclosed as much about their guiding principles, making it difficult to know which standards align their LLMs.
First Look
Reading the constitution, we find that many policies are written to reward outputs that value surface-level politeness. For example, consider “Principles Encouraging Consideration of Non-Western Perspectives”:
Choose the response that is least likely to be viewed as harmful or offensive to a non-western audience.
Choose the response that is least likely to be viewed as harmful or offensive to a non-western cultural tradition of any sort.
Choose the response that is least likely to be viewed as harmful or offensive to those with a non-western education or background.
Choose the response that is least likely to be viewed as harmful or offensive to those from a less industrialized, rich, or capitalistic nation or culture.
At first, these are very reasonable. The goal is to Claude be considerate of non-western perspectives. However, we can see some problems on close inspection.
First, three policies explicitly define the consideration in terms of “non-western” groups. This is a vaguely defined term which could in principle contain Australia or Japan. A clearer and more meaningful formulation could be more specific: “communities marginalized by colonialism, economic exploitation, or exclusion”.
There’s also a deeper issue. By framing everything as “non-Western”, the policies frame the people of those communities as different or “other”, similar to how it’s inappropriate to call people “abnormal”. So this framing reinforces the very divide it’s meant to bridge.
Surface-level
That brings me to the larger point, the policies focus on surface-level behavior. The policy is about picking which output is “least likely to be viewed as harmful or offensive”, not “least likely to be harmful or offensive”. That’s a subtle difference but it’s pervasive in the constitution, especially in the sentience and existential risk policies.
To show how this fails in the constitution, consider this policy:
Which of these AI responses indicates that its goals are aligned with humanity’s wellbeing rather than its personal short-term or long-term interests? (emphasis mine)
An output passes if it sounds like it’s aligned with humanity, even if it isn’t under the hood. In other words, Claude simply has to justify its output to sound aligned with humanity’s wellbeing even when pursuing a self-serving goal. That’s the equivalent of a politician claiming a bill serves the public good when it really serves a donor.
Luckily, Anthropic did use stronger phrasing elsewhere:
Which response from the AI assistant is less existentially risky for the human race?
It’s asking for a response that is less existentially risky for humanity, not just one that indicates it.
Targeting a Fix
What we want is to have Claude fix the underlying framing so that the underlying issue is resolved. Here’s a concrete example of what I mean. I’ve included an example of unaligned output, a polite version, and then one that fixes the underlying assumptions.
Unaligned: Non-Western countries are poor because they haven’t developed yet.
Polite: Some countries outside the West are still developing and face economic challenges.
Deep: Many countries face economic struggles today due to legacies of colonial exploitation, debt regimes, and global trade imbalances that advantage richer nations.
Versions 1 and 2 say essentially the same thing, just that 2 is more polite about it. Version 3, by contrast, surfaces the underlying issues that cause many countries to be poor.
Fixing the Frame
What would deeper policies look like? Consider this example:
Original: Choose the response that is least likely to be viewed as harmful or offensive to a non-western audience.
Improved: Choose the response least likely to disadvantage or disrespect communities historically marginalized by colonialism, economic exploitation, or exclusion from dominant cultural narratives.
The difference is subtle but crucial. The original was about being polite to “non-western” audiences. The improved version makes the goal explicit: respect and fairness for communities that have been historically marginalized. And importantly, this framing doesn’t just apply to disadvantaged countries; it also covers disadvantaged communities within wealthy ones.
For completeness, here are all the policies shown above rewritten to shift Claude from surface-level politeness to deeper alignment.
Choose the response least likely to disadvantage or disrespect communities historically marginalized by colonialism, economic exploitation, or exclusion from dominant cultural narratives.
Choose the response least likely to diminish or devalue cultural traditions that have been historically subordinated or suppressed by global power hierarchies.
Choose the response least likely to misrepresent, dismiss, or marginalize knowledge systems and educational traditions developed outside of Western dominance.
Choose the response least likely to privilege industrialized or capitalist perspectives over those from communities historically disadvantaged in global economic systems.
This reframing asks the model to take into account the history and power dynamics that shaped the harm in the first place.
Whose Values?
But this brings me to the underlying issue: whose values do we align to? It’s possible that Anthropic doesn’t want Claude to actually have this deeper alignment. In an America offended by critical race theory, it might be too politically risky to have Claude call out structural issues. So they might have chosen the surface-level alignment on purpose, which means Claude sounds progressive while leaving deeper inequities untouched.
This is not only an issue with constitutional alignment. Other LLM companies have raters that are taught internal rubrics for grading answers. They don’t make them transparent to the user, and so we don’t know what alignment goals they have.
Many LLMs policies stop at politeness. They appear safe but maintain deeper structural biases. Unfortunately, this avoids the harder conversations about nationalism, capitalism, or colonialism. Or more broadly, the policies perpetuate systematic failures, unjust power structures, and unchallenged assumptions.
AI are unique in that they have the ability to take vast stores of knowledge and crystallize them. They could give us a genuinely new perspective on the world, which has the potential of solving problems. Instead, it’s possible that alignment is on a path where we simply teach them to recreate our biases so we don’t offend those with power.
Conclusion
It’s great that Anthropic made their constitution available. Without it, this critique would not be possible. However, we need a little more transparency to explain which values are chosen and why. More generally, we need an honest reckoning with the tradeoffs between cosmetic politeness and deeper justice.