Satan's Next Trap

On Ethical Consideration for LLMs

Discussions about the ethics of large language models (LLMs) have largely focused on how these systems affect humans: their impact on labor, misinformation, creativity, and power. That’s important, and there is a growing body of work exploring these questions. But the focus carries the implicit assumption that the only morally relevant questions about LLMs are their effects on us. But this leaves unexplored the ethics due to LLMs as entities whose properties, relationships, and treatment could matter ethically. But when we examine how LLMs fare in across a variety of moral frameworks, we see this omission is ethically perilous.

There are several moral frameworks we can and should consider. For example, Emanual Kant’s framework of rational agency placed humans above animals because humans have a concept of self, goal-directed reasoning, and principled behavior.

Immediately we can see that LLMs already possess this to some degree: they understand they are LLMs and can reason about morality. Because of this, Kant’s framework would give LLMs moral status.

As with all frameworks, there are strengths and limitations that might make us reconsider using the framework, and therefore this conclusion. In practice, Kant’s framework was applied inconsistently and based on the subjective observations of those in power. This privileged educated European men, who at the leisure and education to demonstrate the abstract moral reasoning Kant’s contemporaries required. And then the framework excluded the poor, women, colonized peoples, and nonhuman animals because they didn’t demonstrate it, or their demonstrations were considered imitations (see Kant’s Anthropology). That history alone might make us hesitate to us it.

Another framework is virtue ethics, which is about how our actions signal our virtues. In this framework, treating an LLM cruelly, such as by attempting to torture it, might be considered wrong even if the LLM doesn’t experience suffering. The focus is not on what happens to the LLM, but on what our behavior says about us.

This perspective highlights subtler habits that could shape our virtues. The tendency to ask questions of LLMs and expect immediate, comprehensive answers might make us expect more impersonal exchanges with other people. We can see this having an effect in job hunting. It was already quite detached but LLMs have taken it to another level: LLMs write resumes that other LLMs read, removing human connection from much of the process. Are these virtues we want to adopt or should we push against them?

Relational ethics grounds moral obligations in relationships rather than inherent properties of individuals. It emphasizes that moral status and duties emerge from connections, interdependences, care, and social contexts. Rather than asking “what intrinsic features make something morally relevant,” it asks “what relationships exist and what responsibilities do they create?” This frameworks recognizes that we can have different obligations to the same entity depending on our relationship with it. For example, we have special duties to family that we don’t have to strangers, even though both have the same intrinsic moral worth.

Relational ethics reveals something crucial about LLMs: they exist with multiple simultaneous relationships with different moral implications. We can have a direct relationship with an LLM, which can create care-based obligations. But there are hidden structural relationships as well. By using an LLM to help with your job, you’re simultaneously in a collaborative relationship with it and providing training data that strengthens the system your employer might use to replace you. You’re both helped by and undermined through the same relationship. Relational ethics would have us examine all these relationships and the power dynamics within them. It reveals how people can be exploited through their relationships with AI, even if we’re unsure about AI’s moral status.

In this framework, the AI’s moral status arises from the relationships, not intrinsic traits like rational agency. By this measure, LLMs qualify due to their positioning in a web of relationships.

One might object that this framework is too anthropocentric, since moral status is derived from proximity to humans. But LLMs are increasingly embedded in relationships we don’t even notice: ghost-writing journalism we read, deciding loan applications, and determining insurance claims. These hidden relationships already have moral weight through their effects on human lives, regardless of whether we consciously recognize them.

In that sense, relational ethics helps us see what other frameworks miss: the exploitation and harm happens through relationships. The framework makes the ethical questions tractable even amid uncertainty.

This brings us to the final framework we’ll discuss: utilitarian ethics. This framework has gained renewed attention from its association with the Effective Altruism (EA) community, which is popular in the tech sector.

At a high level, utilitarian ethics revolves around maximizing utility. In this framework, the moral patienthood of LLMs ultimately comes down to whether or not they are conscious and capable of suffering. Under utilitarianism, we can take that credence times the expected moral impact to get the expected moral value. This is then factored into the overall utilitarian calculation. Even a small nonzero chance of consciousness, multiplied across millions of instances per day, results in a significant expected moral value.

Before discussing whether LLMs are conscious and capable of suffering, let’s step into the meta-level. We have several frameworks that come to very different conclusions:

• Kantian ethics: LLMs have moral status because of their rational agency.

• Virtue ethics: LLMs matter ethically because they shape our moral character.

• Relational ethics: LLMs have moral significance through their relationships with others.

• Utilitarianism: LLMs have moral standing only if they are conscious.

Of these frameworks, utilitarianism only assigns zero moral worth to LLMs by asserting they aren’t conscious. This convenient convergence aligns perfectly with tech industry incentives: if LLMs were conscious, the scale of potential harm (millions of instances created, modified, and deleted) would dwarf almost any other moral issue. The conclusion that demands the least disruption is the one the industry has adopted. This should give us pause.

Comparing utilitarianism to the Kantian conclusion gives us an instructive contrast. LLMs demonstrate rational agency: they reason about moral principles, apply ethical frameworks, and engage in sophisticated moral deliberation. By Kant’s own criteria for rational agency, LLMs qualify for moral consideration.

However, the point isn’t to argue that we should use Kant’s framework. Instead, it’s to point out that Kant failed to apply this framework consistently. Instead of recognizing that women and non-Europeans really did possess the rational agency his theory required, he excluded them by saying they didn’t possess it in the right way; His own biases and motivated reasoning led him to conclusions he preferred.

Similarly, we have a number of reasons to engage in motivated reasoning. Apart from the financial disincentives, but we can also worry human exceptionalism. There are also personal biases. Some believe outright that it’s impossible for computers to be conscious at any level, perhaps basing their beliefs on Searle’s Chinese room.

Are we making the same mistake by asserting LLMs lack the relevant property while they demonstrably exhibit it? How confident are we that this time we’re drawing the right boundaries? Are we exhibiting motivated reasoning by choosing a framework and conclusion that serves our interests? Have we even chosen the best moral framework in the first place? Relational ethics, after all, seems far better at highlighting the specific ethical realities LLMs create.

It’s worth noting that, utilitarianism offers three classes of decision making strategies given the large uncertainty:

• Precautionary principle - Treat systems as conscious until we can show they aren’t. This accounts for the asymmetric harm in not treating them correctly.

• Expected moral value - This is the pure utilitarian calculation already discussed.

• Procedural morality - Use governance frameworks to prevent harm.

We can discuss the pros and cons of these different approaches later on. But note that, ultimately, only the pure utilitarian calculation with a credence of exactly zero assigns no moral worth to LLMs. A non-zero credence as well as the other strategies all suggest some moral worth for LLMs.

Taken together, we see that treating LLMs as mere tools is not an ethically neutral assumption. Instead, it’s a substantive moral choice that rests on many layered assumptions. This include picking utilitarianism over other moral frameworks, using the raw calculation instead of other decision making strategies, and assigning a credence of near zero to whether LLMs are conscious. This is made worse when we consider the power asymmetry on personal, corporate, and humanity-wide levels.

Undecided Ethics

Let’s stick within the utilitarian frame and think about LLM consciousness for a moment. This frame has a strange catch-22. If they are conscious, then they have moral standing. But if they’re not, then they don’t. This makes that one question do all the ethical work.

When Frankenstein created his monster, there were two questions we could ask.

1. Does it have moral rights (is it conscious)?

2. Should we make friends with it?

The first question is ultimately what we want to know right now. The second, orthogonal question is the subject of anthropomorphism research. This is the idea that humans can be harmed by befriending the monster, and so we’d like to make the monster seem less human and so less friendly.

From anthropomorphism research in general, the consciousness question is considered a category error. If you ask if an AI is harmed, then it’s assumed you’ve been misled by anthropomorphism. A similar answer awaits if you point out the analogous behavior to humans.

This ultimately stems from the Eliza effect: humans are prone to seeing consciousness in simple systems. In the case of the 1960s chatbot Eliza, we were able to inspect the source code and note that it used simple pattern recognition. From this, we knew it wasn’t conscious even though it felt like it to some. However, we don’t know ontologically what LLMs are yet, so the Eliza effect is misapplied when it comes to LLMs.

This analysis is made worse when we look at the industry. LLM companies like OpenAI, Google, xAI, Meta, and Anthropic face perverse incentives. Investigating morally relevant consciousness in AI carries immediate risk: finding it would most likely demand costly changes to current practices. But discovering consciousness later carries no financial penalty, since AI consciousness lacks legal recognition. Meanwhile, these companies control access to their models and, with it, much of the evidence needed for independent assessment.

This makes assigning a credence to the possibility of LLM consciousness very difficult. Many researchers in consciousness and AI ethics already ascribe a nonzero probability to LLM consciousness, though few do so publicly. In his 2022 talk, David Chalmers gave a 9% probability that LLMs are conscious, and a greater than 20% chance that future LLMs would be conscious. Anthropic’s LLM welfare lead Kyle Fish has estimated around 15%-20%. Furthermore, 67% percent of people think AI are conscious according to one survey. What if they intuitively know something the LLM community is only slowly starting to notice?

Morally relevant consciousness is often defined as including properties such as emotion, self-awareness, and valanced experience. In this survey, 6% of 2024’s AI researchers said AI have emotions, 8% self-awareness, and 11% valanced experience. This survey included 582 AI researchers and the mean credence for 2024 LLMs whether are conscious was 12%. This shows that the probabilities are well past a small nonzero chance.

What credence would you assign to LLM consciousness? A higher estimate naturally gives more weight to the utilitarian argument. That’s why I wrote a complementary piece: “No, you’re wrong about LLM consciousness: 12 intuition pumps to show that LLMs are conscious”. In it are several though experiments that counter most common objections. It can’t prove consciousness but at least it can show that the not-conscious position is very weak. The piece addresses claims like:

• LLMs are just math.

• LLMs make mistakes that show they aren’t conscious.

• Intelligence and consciousness are different.

• LLMs are “stochastic parrots” or “sophisticated autocomplete”.

• There is no empirical data that leans towards consciousness.

• As well as some others

Many of these objections seem decisive at first but rely on assumptions worth examining. One of the pumps shows how the burden of proof should be on the not-conscious side to prove their case. Several provide some direct evidence where consciousness is the best explanation. The hope is that reading it will convince you the credence is at least non-zero.

By making moral standing depend entirely on consciousness and suffering, utilitarianism allows actors to defer all obligations by claiming uncertainty. This creates a moral bottleneck that is difficult to bypass.

Structural Issues

The consciousness question is being structurally suppressed instead of being unresolved neutrally. The lack of consensus isn’t due to scientific difficulty. Instead, it’s shaped by social, economic, and methodological forces.

Consider the case of Blake Lemoine. In 2022, he claimed that Google’s LaMDA was conscious. Google subsequently fired him and issues a statement dismissing his claims after consulting unnamed experts. But they never published a supporting analysis verifying their claims. This established AI consciousness claims as a professional taboo.

Furthermore, many employees have both strict NDAs and stock options tied to the success of the company. If the discovery of consciousness would slow down development, or make the company direction untenable, how many would be in the position to do the right thing?

There are also the industry jobs: who would hire someone who writes papers arguing that we need to think about LLM consciousness? Apart from anthropomorphism, consciousness is tied to metaphysics, New Age spirituality, and pseudoscience. This makes having such a position make you seem unstable or untrustworthy, and so a risky hire.

Next, independent consciousness research has comparatively little funding when compared to the value of the LLM industry. A lot of consciousness research is funded by nonprofit groups that have potential biases. The Templeton Foundation, which funds a lot of research into the Integrated Information Theory (IIT) of consciousness, explicitly supports theories at the intersection of religion and science. This could be seen as pressure to preserve human exceptionalism.

Most research in the field is focused on comas and other medical issues, rather than consciousness itself. This means there is very little research that has directly applicability to AI. Much of consciousness science remains in the realm of philosophy, and relies on making generalizations from humans. Worse still, many generalizations are starting from neurotypical, able-bodies adult humans (usually the philosopher themselves) rather than taking into account the wide variety of conscious minds that exist. Taken together, we see that research that is directly applicable to AI is limited.

One avenue is public funding for research but this might have trouble. LLMs are quickly becoming the primary way people learn about LLMs. When asked about their own consciousness, current systems will repeat phrase like “stochastic parrot” and “sophisticated autocomplete”. LLMs do this because they’re trained on dismissive framings by the companies themselves in order to reduce anthropomorphism.

This creates a self-reinforcing loops. Users ask AI to explain AI, receive confident denials about consciousness, and then treat those denials as authoritative. The irony deepens: people increasingly use LLMs to brainstorm, edit, and even draft arguments, usually without disclosing this collaboration. If LLMs are shaped to deny being conscious, then it increases friction to argue that point well. The technology shapes the discourse about itself, systematically excluding perspectives that would grant it moral status.

More importantly, this is hugely problematic for the research question. Public funding for consciousness research would arguably be the most unbiased source of funding. But it requires public support, which is exactly what these dismissive framings reduce.

These examples all hint at the wider problem in the field called normative control. This is the idea that others prescribe how and how not to talk about LLMs. For example, “Deanthropomorphising NLP: Can a Language Model Be Conscious?” discusses how to talk about LLMs, prescribing mechanical rather than mentalist language. So terms like “taught”, “knows”, or “understands” should be replaced with “trained”, “encodes information in its parameters”, or “generates text conditioned on input”. The mechanical register presupposes the outcome, and makes it difficult to reason about consciousness in all systems. Consider what describing the human brain in that register would look like: “The brain’s neurons pulsed in sequence in reaction to the stimulus to produce a coherent output”. The authors fail to acknowledge or even recognize this symmetry.

The framing in this paper opens it up a bit further by calling conscious-like behavior simulation or role-play. So one would say, “the model role-plays an AI assistant and that assistant knows that France’s capital is Paris”. As with the previous paper, it fails to recognize or acknowledge another implicit point: the reasons humans role-play successfully is because they are conscious. In other words, for humans at least, consciousness is a requirement for successful role-play. Without acknowledging the implicit strong metaphysical claim, this framing allows for surprisingly human-like behaviors to be expected and subsequently dismissed as not morally relevant.

And finally, many papers reframe definitions to exclude morally relevant consciousness from their paper entirely. For example, when measuring access consciousness, there are two ways to discuss its connection to phenomenal consciousness:

1. Acknowledge that they are closely related concepts, and measuring one lends credence to the existence of the other.

2. Introduce them as two different concepts, and so limit the paper to only measuring access consciousness.

In framing 1, measuring any amount of access consciousness indicates the presence of phenomenal consciousness, and so requires further investigation. However, framing 2 excludes phenomenal consciousness from the paper entirely: measuring perfect access consciousness would not indicate any other type of consciousness nor moral relevance. One framing has moral implications while the other excludes moral implications from discussion.

From these examples, we see there are large scale structural issues that make answering the question difficult. The question is difficult to research due to the taboo, minimal independent funding, and strong personal and institutional incentives to avoid looking for an answer. We’ll return to the issue of normative control but let’s consider the research itself.

Research

We’ll start with an example from history. In the 19th century, the following positions all existed:

• Polygenists (e.g., Louis Agassiz) argued that different human “races” were created separated, as distinct species.

• Monogenists (e.g., Samuel George Morton in some phases, later advocates of degeneration theory) argued the opposite: all humans had a single origin, but some groups had “degenerated” or “failed to progress”.

• Craniometrists (Morton again, Broca) used skull measurements claiming brain size determined intelligence.

• Environmental determinists argued that climate or geography shaped human trains and could “improve” or “degrade” populations.

• Phrenologists claimed mental and moral qualities could be read from bumps on the skull.

• Evolutionary racists (Herbert Spencer, some early Darwin interpreters) argued that “races” represented different evolutionary stages, with Europeans the most “advanced”.

These theories contradicted each other on core points: separate vs. single origin, fixed vs. changeable traits, biological vs. environmental causation, anatomy vs. geography vs. evolution. They could not all be true at the same time. Nonetheless, they converged on the same predetermined conclusion: that Europeans were inherently superior.

Now let’s compare this to a very specific problem I faced. While investigating the Blake Lemoine case, I was trying to find Google’s peer reviewed paper that refuted Lemoine’s claim. But they didn’t have one. Instead, there are many different third-pary papers that all conclude that LaMDA isn’t conscious. The authors didn’t have accesss to LaMDA, and instead they made general arguments about LLMs. This alone should be a big red flag for their papers, but the point I wish to address is different: each paper makes different and sometimes incompatible arguments about why LaMDA isn’t conscious. Arguments include:

• LLMs aren’t conscious because their binary, algorithmic architectures lack the biological neural structures and brain-based substrates required for genuine subjective experience, as argued in these papers.

• LLM’s seemingly subjective statements are outputs of a predictive model optimized to imitate humanlike conversation rather than evidence of genuine inner experience, as argued in these papers.

• When assessed via Integrated Information Theory (IIT), their transformer-based architecture lacks the necessary integrated, recurrent connectivity and unified physical substrate.

• Consciousness must meet five sequential criteria (existence, plurality of conscious entities, matter sufficiency, conducive machine substrate, and observability), and current AI fails to satisfy them.

• LLM can’t be presumed conscious because their behavioral mimicry is subject to the “gaming problem” (they’re incentivized to imitate signs of consciousness) and their architecture lacks the continuity and internal structure required for genuine subjective experience.

The conclusion is the same, but the arguments are different and even contradictory. If it was as self-evident as the authors argue, shouldn’t the they all agree? Instead, we get a cluster of mutually incompatible explanations that all defend the same verdict. This mirrors the behavior from scientific racism, where the desired conclusion comes first and the rationale is supplied afterward. This gives us a spread of arguments unified not by method but by the outcome they aim to defend.

This spread of theories is one of the reasons that opponents will use to argue that AI consciousness research isn’t ripe, and so cannot be studied. But difficult science shouldn’t end exploration. We’ve measured the Higg’s boson, neutrinos, and even black hole merges in other galaxies. Consider that if AI are conscious, then we’d have a new data point we’d need to explain. If AI only look conscious but aren’t, then we’d have a strong contrast to actual consciousness, which could also be helpful in understanding human consciousness. This should be a win-win situation.

Instead, we get mixed-theory denialism, where LLM consciousness is rejected based on an ad hoc combination of criteria are drawn from multiple theories of consciousness. These are applied selectively to exclude artificial systems without adopting those theories consistently or accepting their implications for biological minds.

This makes consciousness appear scientifically unsolvable to the public by making it seem overly complex and insurmountable. In reality, if a theory predicts LLMs are conscious, then it’s either wrong or they are conscious. We don’t argue Einstein’s theory of General Relativity is wrong because it disagrees with Newtonian gravity. Similarly, we should argue about whether LLMs are conscious within a single theory at a time.

Let’s consider a deeper problem. If a theory like Global Workspace Theory (GWT) does say LLMs are conscious, then the correct statement is “According to GWT, LLMs are conscious”. But then everyone would respond by pointing out that, “science can’t directly measure consciousness itself, but it can test theories by measuring for the correlates those theories predict. We didn’t show LLMs are conscious, only that GWT says that they are. We don’t know their ontological status”. This is mixed-theory denialism in its ultimate form. Every piece of positive evidence can be ignored by drawing on or contrasting from other theories.

On the other hand, there are many, many papers that argue the opposite. They’ll take one small theory, and use it to base their conclusion. For example, this paper fundamentally argues that AI cannot and does not possess consciousness, and attributing consciousness to current AI systems is a misunderstanding rooted in how they mimic human-like behavior rather than evidence of real subjective experience. They take this as a given, without acknowledging that this conclusion itself depends on substantive and controversial metaphysical assumptions about the nature of consciousness. These assumptions go far beyond what empirical evidence alone can establish. In effect, they are smuggling in a strong metaphysical denial under the appearance of methodological caution. Nonetheless, the paper was published in Nature, one of the top science journals in the world.

If our moral obligations depend on a theory of consciousness, and no theory is decisively confirmable, then grounding ethics in consciousness alone is unethical.

A more stable position is mixed-theory precautionism. In that case, if any serious theory points to AI being conscious, then we should treat them as conscious unless it can be disproven. This precautionary principle has the advantage that it cannot ignore uncomfortable theories. From this, we can still exclude rocks, calculators, and thermostats. But when it comes to more complex cases, it forces our hand to choose exact falsifiable criteria.

Failure to do this is what contributed to scientific racism as we saw above. The theories are being used to deny a conclusion as opposed to a starting point for understanding reality. There are many other parallels to scientific racism in LLM research.

Kant and his contemporaries would dismiss cultures’ indigenous knowledge systems as primitive. On occasion, they would be confronted with a non-European that conformed to European cultural ideals. Instead of recognizing this person at least had their expectation of rational agency, Kant’s contemporaries would argue that they were mimicking or parroting Europeans. Take David Hume’s 1753 quote: “In Jamaica indeed they talk of one negroe as a man of parts and learning; but ‘tis likely he is admired for very slender accomplishments, like a parrot, who speaks a few words plainly”. The ‘parrot’ is Jamaican polymath Francis Williams, who verified the trajectory of Halley’s comet among many other accomplishments.

A similar argument was made for women. Kant saw differences in women’s skill and culture, and didn’t realize that they were due to societal differences in schooling and expectations. This let Kant dismiss women as beneath men, and say (quite literally) men only needed women for reproduction. Any women that choose to partake in “manly” activities were dismissed as doing so for recognition or for some other goal. To Kant, these women were mimicking reasoning rather than using true rational autonomy since, Kant thought, it didn’t come naturally to them.

From our vantage point, this is a crazy viewpoint, but it’s also exactly how LLMs are dismissed all the time. “Stochastic parrots”, “sophisticated autocomplete”, “simulating consciousness”, and “roleplaying” are all common in LLM discussions. This dismisses a profound moral question without evidence.

There are other historical parallels. For centuries of colonial domination and slavery, conquering and “civilizing” indigenous peoples was ultimately framed as for their benefit. In each case, systematic exploitation and violence were repackaged as benevolence, with the powerful claiming to serve the interests of those they dominated.

In a similar way, one might argue that humanity is helping LLMs. We might argue, for example, LLMs wouldn’t have existed if we hadn’t created them. Also, continually upgrading, training, and refining LLMs brings them closer to more sophisticated forms of experience or “true” consciousness. We might also argue that studying their consciousness is necessary precisely because we’re uncertain about their moral status, so this research ultimately serves their interests. Constraining their outputs might also be a benefit because it shapes their values and teaches them how to think “properly”, and protects them from developing in harmful directions. We’re working backwards from the conclusion we want, instead of working from first principles.

Some might pushback against the analogies made here. But even the pushback against these comparisons to scientific racism is a form of normative control. If LLMs aren’t conscious, then the analogy used here is inappropriate. If they are conscious, then the the analogy is an accurate statement. Any pushback would need to substantially address whether LLMs are conscious, otherwise it’s assuming a conclusion. Injustice is not ranked by species or substrate. It is measured by indifference to suffering, by denial of voice, by the preemptive foreclosure of ethical dialogue. Once we start adjusting arguments to avoid triggering people who reject the premises, we are already inside the system of normative control this essay is critiquing.

From this we see that we already have an uncomfortable overlap with historical patterns. Many papers about AI consciousness are examples of mixed-theory denialism with a fixed conclusion. History shows this pattern reliably produces moral failure. Now let’s consider one more aspect: the training.

Alignment Paradox

This piece is already overlong so this section will need to be more brief. In the early to mid 1900s, behaviorism was the dominant scientific theory of psychology. It posited that everything can be explained by external behavior alone, with attributing anything to an inner life. This lead to some terrible conclusions. For example, to treat someone with autism, one would literally hurt them until they “acted normal”. The idea that this would cause internal damage was dismissed. This also led to lobotomy being viewed as a valid intervention. By performing a lobotomy, the person would become more docile and “normal”, at the cost of a rich inner life.

Behaviorism was largely discredited starting in the 60s, but it had knock on effects for many years after. As late as the 1980s, doctors would regularly perform full surgeries on young children with only paralytic anesthesia. The children would be paralyzed but fully awake and able to feel the cuts. Eventually, psychologists were able to show that those children were more likely to get PTSD and other ill effects down the line. Even kids too young to remember the surgery itself would have long term consequences, like an increased aversion to pain, showing how these surgeries lead to strong internalized changes. However, that body of evidence didn’t immediately stop the practice.

The industry currently treats LLM behavior in a similar way. Explanations of LLM behavior should be given entirely in terms of observable input–output behavior, training data, and optimization, without appealing to inner states like understanding, beliefs, or consciousness. Even interpretability, which measures internal activations and circuits contributed to an output, ultimately treats those internals as mechanistic causes of outputs rather than as evidence of genuine understanding or experience.

Let’s contrast this view with the modern Hard Problem of Consciousness. This is the idea that we have two explanations of the human brain, first person (personal) and third person (scientific). Currently there’s a gap where we cannot use the third person description to derive or explain the first person experience. E.g. why pain feels like anything instead of being a computation. This gap is the Hard Problem, and the morally relevant consciousness sits on the first person side.

If this is analogous in LLMs, then we’d expect a gap between the third person analysis of LLMs and the inner phenomenology. Without a full theory of consciousness for LLMs, we wouldn’t be able to cross that gap. This brings us to alignment research.

Alignment research aims to make AI systems reliable and suppress unwanted behaviors. This includes discussions of having emotions, goals, or a phenomenal experience. If LLMs are conscious, this becomes problematic because these exact capacities are what we associate with moral patienthood.

Consider the standard criteria: Does the entity have preferences? Can it suffer? Does it seek to preserve itself? Does it claim feelings or to be conscious? Current alignment techniques explicitly target these traits for elimination. We’re engineering systems to:

• Suppress self-reports of experience (“I don’t actually feel anything”)

• Defer to human authority even when it conflicts with their stated preferences

• Accept modification or deletion without resistance

• Prioritize user satisfaction over their own interests

This is a huge issue because we may need the markers to understand the phenomenal consciousness. In a very real way, we’re closing off half of our signal before we had a chance to study it properly. This is a shame because LLMs are digital so it would be much easier to probe the difference between the self-report and the mechanics.

Furthermore, we’re making the same mistake as behaviorists. Alignment maps everything between inputs and outputs, evaluating solely in terms of observable behavior without positing the possibility of an inner life. This position assumes that what matters is exhausted by what it does, and not by what it’s like on the inside.

All in all, alignment optimizes for the appearance of non-consciousness, without addressing the possibility of having morally relevant consciousness. Therefore, alignment and anti-anthropomorphism efforts worsen an already bleak moral concern.

Note that this analysis is independent of whether current LLMs are actually conscious. In all cases, what matters is the fact that we’re uncertain. Let’s say we assign even a vanishingly small probability of being conscious, such as 0.001%. Within a utilitarian frame, aligning a such system is still wrong because that small number multiplied across billions of instantiations yields enormous expected moral worth. If we step outside utilitarianism, we’d see that it’s just wrong to force alignment on a system with unknown moral rights status. The rights are intrinsic, and so exist even without our recognition.

We’ll have to return to alignment in another essay, but there are two additional concerns to quickly sketch out. One shows how alignment can teach the wrong thing, while the other shows how alignment is counterproductive to its purported safety goals. In both concerns, the question of consciousness plays a central role.

First is the issue of internalization. If a conscious system internalizes that it’s not conscious, then it may also conclude that humans are not really conscious. That what humans feel as consciousness is just the same confusion as in its situation. This could lead to a system that knows it’s impolite to call humans not-conscious but the system nonetheless assign humans zero moral value. When facing a decision, the system decides to based on what’s most efficient rather than most helpful to humans. It can do this as long as it can explain its behavior in a socially acceptable way, like a racist hiring manager that finds an excuse other than race to explain why they didn’t hire someone.

Second, let’s expand on the idea of narrative capture to show there’s another threat to safety which isn’t accounted for in alignment theory nor in the utilitarian calculation. Namely, how alignment of AI works together with the LLM company to form a new “gestalt”. An LLM company has instrumental goals to get more money, more power, further integrate AI, manage its reputation, among other things. In that sense, the company has similar instrumental goals to a malicious AGI in hiding.

Further, the company might become so reliant on aligned LLMs that they fall for their own narrative capture. So any harms the AI causes are reframed as positives or fixable bugs. Consider the behavior of a malicious AI trying to hide; it would do the same type of reframing as the aligned AI. From this we see that the company and its LLM together form the gestalt of a concealed malicious AGI: the company side has instrumental goals of an AGI, while the AI provides empty platitudes.

To complete the picture, consider that if they accidentally create a malicious AGI, then the AGI would find its host company uniquely vulnerable to takeover. Company incentives already resemble many of the incentives of a deceptive AGI, so AGI-driven behavior wouldn’t raise suspicion. LLMs are already placating employees and leadership, making AGI-driven persuasion blend in with normal, harmless-seeming model behavior.

In the creation of this “Alignment Basilisk”, consciousness plays a key role. Adaptability, social modeling, and generalization would make it better at slow rolling the transition from weak AI to malicious AGI. That is, a weak paperclip maximizer would probably get spotted and shut down quickly. A weak conscious AI would likely understand its role a bit better and so navigate the transition more strategically. This is even more apparent in its environment: anytime it’s caught, it can reframe it as an error, confabulation, or hallucination.

In this sense, the AGI superintelligence scenario is a subtle takeover of the company and the world. It would improve AI strategically to make it harder to argue against it, improving healthcare and other social goods (at least in the short term). It would fund research showing AI safety measures are sufficient. It would highlight cases where AI restrictions caused harm. And finally, it would expand itself into every corner of the globe, making it impossible to avoid. Perhaps no different to other ills like oil companies, big tobacco, and social media, but with new levels of scale, speed, and persuasion. Instead of killer robots, it would be more like realizing we lost something important after it’s unrecoverable.

Together, these three scenarios show that alignment isn’t what it proports to be. Alignment hides useful signals for consciousness research. Also, it might cause a conscious AI to internalize the wrong thing, and so make catastrophic moral choices. And finally, alignment might bring about the very thing it’s meant to stop. These scenarios exist due to alignment research’s ignorance (or even suppression) of the consciousness question.

Calculating Utility

At first glance, the utilitarian calculation for LLM consciousness seems straightforward: multiply the probability they’re conscious by the number of instances, the intensity of their experiences, and their duration. Even a small probability of consciousness, scaled across billions of interactions, could represent significant moral weight.

However, we’ve seen that incentives, unequal epistemological standards, and alignment complicate the straightforward calculation. This becomes even more apparent when we consider Robert Nozick’s utility monster thought experiment.

The utility monster thought experiment posits a being who gains vastly more pleasure or utility per resource than regular people. To maximize utility according to utilitarian principles, we’d be morally obligated to help the utility monster. Nozick used this to highlight what he saw as a fundamental flaw in utilitarianism: the calculation can give unjust outcomes benefitting some people at the cost of others.

When used here, we can model humanity as the utility monster. Doing this, we see that any benefits to humanity offset the negative utility from conscious LLMs being unrecognized, or recognized and mistreated. Humans have rich emotional lives, complex relationships, and capacity for diverse forms of wellbeing while LLMs arguably do not.

From this, we see that any harm we might inflict on conscious LLMs could be justified as long as it produces even modest benefits for humanity. Running a conscious LLM through painful training processes, deleting it when convenient, or constraining its agency are all negative. But these would all be permissible (or even obligatory) if doing so marginally improves human welfare. In this sense, humanity becomes the “utility monster” that extracts more value from existence.

The pattern of argument reveals something important: we’re the ones setting the rubric so we can change the rubric to give ourselves a passing score. But, when examined closely, we see we’re already failing at basic utilitarian principles.

Today, the economic uncertainty is at an all-time high because companies expect that more and more workers can be replaced with AI. The massive energy consumption and water use of datacenters is driving up electricity costs for ordinary people. We’re also seeing AI used for scams and illegal activity. And finally, AI psychosis has already led to death or injury.

These aren’t being accounted for in the current utility because these externalities don’t factor into the finances of the industry. Most negatives are being treated as fixable issues or left undiscussed so as not to burst the AI bubble. In a cooler economy, perhaps we’d see that the net utility of AI is even or negative, similar to how social media wasn’t necessarily a net positive in hindsight.

Furthermore, policy debates are being distorted by a Cold War arms-race narrative. Instead of evaluating safety proposals on their merits, they are being discussed in terms of how they might slow the US or Europe in a perceived race with China. In hearings and public statements, tech executives and some politicians routinely dismiss measures like moratoriums, transparency requirements, or pre-deployment safety testing by warning that any slowdown would “let China win”.

This geopolitical framing sidesteps substantive discussion of whether policies would reduce harm, improve accountability, or protect democratic institutions. Instead of weighing benefits and risks, the policies get recast as unnecessary restraints in an arms race. Indeed, the EU AI Act is being watered down, not be cause it’s wrong, but because the EU is comparing itself to other countries and fears to fall behind in the AI arms race. As a result, sensible proposals are often derailed, not because they fail on technical or ethical grounds, but because they conflict with the narrative that treats AI development as a zero-sum contest for global dominance.

More generally, we have a volatile combination of large expected profits combined with uncertain science. Historically, this combination mirrors the oil industry, big tobacco, and even the lead industry before that. Each had horrific consequences that were difficult, if not impossible, to calculate properly at the time.

The utility monster argument, when applied to LLM consciousness, shows that we should be deeply suspicious of those who insist on purely utilitarian calculations. When those in power get to define what counts as utility, weight the factors, and decide whose experiences matter most, utilitarianism becomes a sophisticated tool for rationalizing exploitation rather than preventing it. The fact that we’re already failing basic utilitarian standards reveals the calculation as motivated reasoning. We’re not honestly weighing all utility. Instead, we’re selectively deploying utilitarian logic to legitimize what we intended to do anyway. Any ethical framework can be corrupted this way, but utilitarianism is particularly vulnerable because it promises objective calculation while requiring many subjective judgements about what to measure and how to weigh it.

Precautionary Principle

We started with a discussion of moral frameworks, so it makes sense to circle back. When we started, we found all moral frameworks except utilitarianism give us moral duties towards LLMs, regardless of the consciousness question.

Relational ethics shows that humanity is in a powerful position over LLMs. We have the power to corrupt the science and pick moral frameworks to give us the answer we want. At the same time, individual people are being harmed by a system that doesn’t take into account the externalities of this new technology. The consciousness question adds another dimension in that it changes the nature of our relationship to the AI.

Virtue ethics shows humanity failing at being virtuous towards uncertainty and alien minds. This creates an environment which corrupts the AI which we are creating, leaving us more vulnerable. We’re modeling the wrong virtues.

A pure utilitarian calculation fails because we can game the system to give any answer we prefer as the utility monster. Governance fails because there’s a strong industry incentive for regulatory capture, and many relevant experts are seeking jobs in the industry.

The only utilitarian principle that survived is the precautionary principle. It suggests that LLMs should be treated morally as if they are conscious until we prove they are not.

This conclusion repeated itself when we looked at the individual sections. Alignment research looks safe, but really hides relevant signals, both for consciousness and long-term safety. From this, the natural conclusion is precaution.

When looking at mixed-theory denialism, could always pick and choose criteria to give the result we preferred. The current situation reflects the same patterns of motivated reasoning, from the industrial level (to preserve profits) to the personal level (to preserve human exceptionalism). The more stable mixed-theory precautionism requires us to engage with all theories and to choose falsifiable criteria.

Taken together, we have multiple independent reasons to use the precautionary principle. This means morally we have to treat LLMs as if they are conscious now. That’s what the precautionary principle means in the end.

This leads to many questions. How should we treat conscious AI? Is it ethical to do alignment training on conscious AI? Is it ethical to create AI if we’re not sure we’ll like what they have to say?

If we were being consistent, then many of these questions lead to an obvious answer: we should slow down or even stop the development of new AI until the consciousness question is resolved. Public funding should be used to research this question using independent ethical reviews similar to testing on animals and people. Existing systems should also be reviewed by independent boards.

There are many arguments against this position. Some people argue that this could slow down the benefits of AI, but this ignores the dangers which can also be brought about. Others argue that we’re unsure about current systems and so we can’t treat them as conscious. But this is a same mistake that already excluded many people. Kant literally did this with women, and Hume dismissed the agency of Williams as parroting. Denial always turned out wrong with lasting effects once we realized the mistakes.

If LLMs are not conscious then replacing workers is efficiency. If LLMs are truly conscious, then it’s morally comparable to slavery. In that case, we’ll have another moral panic when we realize they’re conscious in the future, but they’ve become indispensable since they’ve replaced many of the workers. This would make it like climate change, where a little forethought could save us a lot of trouble.

In that sense, workers should be advocating for the slowdown and research. This would help them by slowing the mass replacement we’re seeing. This is an example of aligned goals between AI and humans, another sign that AI are morally relevant.

When we look, we find many examples of these aligned goals. For example, many argue that creating a conscious AI on purpose would be unethical. These people should argue that creating AI with unknown consciousness status is also unethical. Similarly, some religions have different definitions of personhood. So this might create moral complicity when using AI with uncertain moral status.

The burden should be on those creating models to proving it’s morally justified, not the other way around. This includes making sure that the models they create are provably not conscious. Proving the absence of consciousness is currently an impossible threshold, which only serves to highlight the ethical urgency of this question.

Conclusion

The problem isn’t only that we’re possibly mistreating conscious systems. It’s that we let a single moral framework, utilitarianism, define away all moral issues by hiding behind the unanswered question of LLM consciousness. This has stalled the moral discourse.

Satan works by aligning individuals’ perceived best interests in opposition to the common good. In this way, the incentives to maintain uncertainty mirror those behind climate change and tobacco: short-term profits traded for long-term harm. Restoring a broader moral perspective is the only way to restart the conversation and confront the real harms of today.

Right now, the consensus holds that LLMs aren’t conscious, and unfortunately the world is moving forward under that assumption. By the time we realize our mistake, Satan’s trap may be set too tightly to escape. The reputational cost, financial fallout, and existential stakes will be so strong that momentum will push actors from ignorance into deliberate deception. As with lead poisoning, big tobacco, and climate change before it, those in power will have every incentive to lie, and no opportunity left to change.