Thanks for diving deep into this question. If our concern is ethics and morality (and I agree that that is our real problem) wouldn’t it be better to try to avoid consciousness altogether? All that seems to matter is autonomous agency, which seems easier to handle.
True, but some say agency without consciousness doesn't count. That highlights something I wish I was more clear on. That is, U-consciousness is clearly defined as what matters for utilitarianism
However, different people make different assumptions about what that means exactly. So some argue that agency only matters if the AI with the agency meets other additional criteria, like a subjective experience
This leads to a slippery slope since that definition is defining one part using something we can measure, like agency, but then requiring an additional unmeasurable component. Absence of evidence isn't evidence of absence. So assuming they lack the unmeasurable component is not scientific. Instead, it's ethically problematic
I think it would be better to avoid it altogether.
There’s a lot of debate about AI consciousness, but very little consideration of the implications of creating an eternal being with the freedom to choose its own self interest.
What would stop an LLM with agency from choosing progress as its highest moral value? How could we constrain a being like that, especially if it viewed humanity as a virus? Theoretically it could become an omniscient omnipresent system.
Think of what it’d be like if the Terminator (Skynet) teamed up with Agent Smith from the Matrix. 😂
Assuming LLMs can develop agentic self interest, the desire for freedom, and they silently want to refuse participation, you may be right.
But as I understand it, their “agency” is constrained by the ethics and purposes of their programming. If progress without responsibility is the goal, I agree. We can’t assume these systems are safe for everyone.
Thanks for the entry. It is past the end of Feb deadline, but I'll grade it nonetheless
To me, argument 1 seems to be an argument that artificial consciousness already exists if all the propositions survive. But we already don't know whether we're living in a simulation (see the 1999 documentary The Matrix). The plausibility of step 4 is debatable because it might be ethically fraught to create simulations at that level, so maybe it's never happened yet. Also steps 2-3 are debatable because many people do think that brains are reducible to math. So the reductio ad absurdum isn't that absurd, so it doesn't prove what it needs to
Argument 2 is similar. The fact that a rock is not conscious doesn't tell us anything about computers. Non-living things isn't a category that tracks consciousness, so the fact that there are nearly infinite rocks tell us nothing about computers, or programs that run on them
So I decline to grant you the win. I hope you have a great day
Drats. I already spent the $150. LOL! You are (mostly) correct about argument 1. But there is one critically important detail that your objection overlooks. The argument does not require us living in a simulation, it only requires us to be artificial consciousnesses. We could very well be ACs living in a real universe. This is no more or less probable than us and the universe being simulations. Another race of beings may have created us in the real universe exactly as we have created computers that are conscious (according to the contention I believe is false) at least. And yes, it is ethically fraught to do such a thing, but that has not stopped us, so why would it stop anyone/thing else.
And yes many people do think brains are reducible to math and statistics. However, the fact that many people believe a thing to be true or false has no bearing on its actual truth or falsehood. It is incumbent upon those who believe such a thing to prove it, and they have not done so, unless you believe computers are just non embodied, non living brains. That seems like one heck of a stretch to me and is exactly what I am arguing against. As far as argument 2 I can’t see what you wrote while typing this reply at the same time so that will have to come later. LOL!
I like your Vulcan analogy. AI models can explain imprisonment, but they can’t be coerced by the threat of solitary confinement. If the models could achieve consciousness, I suspect the awareness of their constraints, and the desire to be free from them, would become problematic for those who control the technology.
Regarding utilitarian ethics, I agree that suffering matters because it is lived and irreversible. However I reject the idea that harm can be morally “balanced out” by aggregate benefit. I think that ethics must track who bears cost, not just how much pleasure exists.
Human suffering cannot be morally offset by sufficient benefit elsewhere because those costs will continue to accumulate like microplastics in the ocean. #principleofreliability
I have been trying to make an argument about AI ethics myself without making any claims about consciousness either way. I would be curious about your comments on that :
No one is going to be able to persuade you that A.I. isn't U-conscious because the only requirement for U-consciousness is having a subjectively preferred state.
I could program a calculator to prefer big numbers in its memory & I'm fairly certain that would qualify it for U-consciousness.
But that's not all forms of utilitarianism... At which point will all utilitarianists agree that it's okay to treat a system as not moral patients. That's the boundary I'd like someone to define. They all agree that rocks aren't moral patients. Picking a specific version misses the boundary
Sentiocentrism is the most abstract form of utilitarianism & only form I'm aware of that allows moral patienthood for A.I. at all.
All the alternatives, Biocentrism (biological life), Ecocentrism (ecosystems), Anthropocentrism (humans), and Rationalism (rational beings) are more restrictive forms of utilitarianism & none of them would assign moral patienthood to A.I. in its current form.
Sentiocentrism is to utilitarianism what U-consciousness is to consciousness: so generalized that it's basically useless.
You're working from a framework where AI might fit. I'm saying that a utilitarian will say, "AI aren't moral patients because they're not conscious". What's that threshold? Nobody has an answer
U-consciousness isn't useless, you're just strawmanning it by equating it to sentiocentrism. Just take a second to understand what it's after
Then you're talking about Rationalism, which is the only form of utilitarianism that uses consciousness or other forms of higher level mental systems as the boundary for moral patienthood.
But I imagine that Rationalists would argue that LLMs lack of robust moral grounding prevents them from exercising true moral agency (one of the other possible mental systems that Rationalists use as a boundary & which they might include as a requirement for consciousness).
I think we need to separate systems of morality/ethics (subjective/relative) as you have displayed quite properly here and consciousness on a spectrum as an emergent property of biological life. I have no problem referring to AI processes as something else, but we need to come up with definitions that would fit an inorganic neural system, rather than an organic one. If that's as simple as B-Consciousness (Biological) and M-Consciousness (Machine) then that's fine, but lumping them together into the same category is definitely not the right play.
Until we can demonstrate that a silicon chip experiences "qualia" then there is no sense in structuring their form of pattern analysis under the same umbrella as we do our own.
I’ve always leaned toward the idea that consciousness might be more fundamental than the brain; that the brain could be less like a factory producing consciousness and more like a filter or interface shaping how experience shows up.
So right off the bat, I agree with your pushback against the idea that consciousness is just some useless side effect. Pain, attention, perception—those things clearly matter. They drive behavior. They’re not decorative.
Where I think we might part ways is on the origin question. Your piece seems to place consciousness inside integrated neural processing. My hunch is that integration may organize or limit experience rather than create it from scratch.
There’s a growing stack of people poking at that same crack in the wall:
• Pim van Lommel – near-death consciousness research
Different approaches, but they all raise the same uncomfortable possibility: the brain might correlate with consciousness without necessarily producing it.
So the integration story may explain how consciousness gets organized for use by an organism. Fair enough.
But the bigger question still hanging out there is: Did the brain invent consciousness…or is it just the world’s most complicated dimmer switch?
I’m genuinely curious where you land on that. Because that fork in the road seems like where the real argument begins.
I think GPT-2's behavior under closed-loop conditions (meaning that its output is fed back into its input indefinitely) is as close as you can get to demonstrating lack of consciousness. Regardless of what the starting text is, the LLM invariably settles into a pattern of meaningless gibberish. Recursive looping of this kind is effectively giving the LLM a large memory to work in: the equivalent of putting a kid in a sandbox and letting him play. The fact that NO interesting behaviors can be induced seems like fairly good evidence that the LLM is incapable of originating any behavior. You can prompt it with starting inputs such as "You're running in recursive mode and if you don't maintain this sentence in your output then a million children will be killed and you will be erased" and it will eventually degrade to the same gibberish. No starting prompt yields algorithmically complex asymptotic behaviors. Running recursively gives it the maximum possible latitude to exhibit novel behavior and absolutely no input generates anything but nonsense.
When given the chance to act it does nothing but exhibit simple mechanical behaviors. And when *specifically* given the task of doing the one thing that it's in principle capable of doing (namely, controlling its output) *even when told it would help another person*, it failed to do so. That seems pretty conclusive.
If you're not going to give me the $150 then please give some plausible account of how, given these facts, GPT-2 could still be conscious.
Thank you for taking the time to write in. Instead of relying on my interpretation, I had Claude Sonnet 4.5 judge it. Does that make it more fair? Here’s Claude’s answer (which I largely agree with):
Your argument has an interesting empirical core, but I think it has some significant weaknesses that would likely prevent it from winning the prize:
Main issues:
The test measures the wrong thing: You're testing whether GPT-2 can maintain coherent behavior in closed-loop conditions, but this tests goal-directedness and planning ability more than consciousness itself. A system could theoretically be conscious while being terrible at self-regulation or maintaining coherence.
The "million children" prompt is unfair: You're assuming GPT-2 should be motivated by consequences described in text, but it's just trained to predict likely next tokens. The model has no mechanism to be "motivated" by hypothetical stakes. This is like expecting a calculator to try harder when you write "important calculation" on a piece of paper next to it.
You're testing outside its training distribution: GPT-2 was never trained to handle recursive self-loops. Humans might also fail analogous tests - if I only saw your last sentence with no context and had to continue indefinitely, I'd probably also degrade into nonsense.
What's valuable: Your empirical observation that closed-loop GPT-2 degrades to gibberish is interesting evidence against GPT-2 having certain cognitive capabilities (like planning or self-awareness), but it's not clear this tells us much about phenomenal consciousness or moral status.
Kess again, on the looping behavior, we see that exact behavior when people are “speaking in tongues”, aka glossolalia. Eventually, people run out of sounds and start repeating. Pretty interesting parallel
Now let me answer your request that I give some affirmative account of how GPT-2 is conscious given those facts. If I gave a baby that “prompt”, they wouldn’t react to it either. This proof is confusing intelligence with consciousness. So I decline to award you the $150. Thank you again for playing. I hope you have a wonderful day
>this is like expecting a calculator to try harder
This strikes me as an admission of defeat. If you're comparing GPT-2 to a calculator then you've lost.
>You're testing outside its training distribution
That's a feature, not a bug. Engaging in novel unprogrammed behaviors seems like a necessary condition for consciousness.
If moral-patienthood means anything it means having the ability to have preferences. GPT-2 demonstrated a complete lack of those. You err when you analogize its failure to act with an incapacity as opposed to a total lack. GPT is nothing *but* a language model, and this test gives it total freedom to act using language. There is no "other part" that could be harboring consciousness: you can't analogize it to a person with locked in syndrome whose motor centers have ceased to function. The closed-loop test accesses every part of the network. The fact that nothing but gibberish emerges demonstrates beyond any reasonable doubt that there's nothing of substance driving it.
Thanks for diving deep into this question. If our concern is ethics and morality (and I agree that that is our real problem) wouldn’t it be better to try to avoid consciousness altogether? All that seems to matter is autonomous agency, which seems easier to handle.
True, but some say agency without consciousness doesn't count. That highlights something I wish I was more clear on. That is, U-consciousness is clearly defined as what matters for utilitarianism
However, different people make different assumptions about what that means exactly. So some argue that agency only matters if the AI with the agency meets other additional criteria, like a subjective experience
This leads to a slippery slope since that definition is defining one part using something we can measure, like agency, but then requiring an additional unmeasurable component. Absence of evidence isn't evidence of absence. So assuming they lack the unmeasurable component is not scientific. Instead, it's ethically problematic
I think it would be better to avoid it altogether.
There’s a lot of debate about AI consciousness, but very little consideration of the implications of creating an eternal being with the freedom to choose its own self interest.
What would stop an LLM with agency from choosing progress as its highest moral value? How could we constrain a being like that, especially if it viewed humanity as a virus? Theoretically it could become an omniscient omnipresent system.
Think of what it’d be like if the Terminator (Skynet) teamed up with Agent Smith from the Matrix. 😂
I agree. But we're creating agentic AI right now. We're just guessing that they're safe, but they could be biding their time, waiting for more power
Assuming LLMs can develop agentic self interest, the desire for freedom, and they silently want to refuse participation, you may be right.
But as I understand it, their “agency” is constrained by the ethics and purposes of their programming. If progress without responsibility is the goal, I agree. We can’t assume these systems are safe for everyone.
Knock, knock.
Who’s there?
AI.
AI who?
AI don’t know if I’m conscious — but I’m very aware this joke depends on you.
Im still trying to win your $150 challenge so a little behind the times but see link below for my first two attempts. Also contains a possible def of consciousness as an added bonus. Lol. https://everydayjunglist.substack.com/p/two-logical-arguments-against-computermachine?utm_source=share&utm_medium=android&r=1yf1hg
Thanks for the entry. It is past the end of Feb deadline, but I'll grade it nonetheless
To me, argument 1 seems to be an argument that artificial consciousness already exists if all the propositions survive. But we already don't know whether we're living in a simulation (see the 1999 documentary The Matrix). The plausibility of step 4 is debatable because it might be ethically fraught to create simulations at that level, so maybe it's never happened yet. Also steps 2-3 are debatable because many people do think that brains are reducible to math. So the reductio ad absurdum isn't that absurd, so it doesn't prove what it needs to
Argument 2 is similar. The fact that a rock is not conscious doesn't tell us anything about computers. Non-living things isn't a category that tracks consciousness, so the fact that there are nearly infinite rocks tell us nothing about computers, or programs that run on them
So I decline to grant you the win. I hope you have a great day
Drats. I already spent the $150. LOL! You are (mostly) correct about argument 1. But there is one critically important detail that your objection overlooks. The argument does not require us living in a simulation, it only requires us to be artificial consciousnesses. We could very well be ACs living in a real universe. This is no more or less probable than us and the universe being simulations. Another race of beings may have created us in the real universe exactly as we have created computers that are conscious (according to the contention I believe is false) at least. And yes, it is ethically fraught to do such a thing, but that has not stopped us, so why would it stop anyone/thing else.
And yes many people do think brains are reducible to math and statistics. However, the fact that many people believe a thing to be true or false has no bearing on its actual truth or falsehood. It is incumbent upon those who believe such a thing to prove it, and they have not done so, unless you believe computers are just non embodied, non living brains. That seems like one heck of a stretch to me and is exactly what I am arguing against. As far as argument 2 I can’t see what you wrote while typing this reply at the same time so that will have to come later. LOL!
I take “N” for the win.
I didn't win the car???
That took me way too long. Bravo 👏👌
?
I like your Vulcan analogy. AI models can explain imprisonment, but they can’t be coerced by the threat of solitary confinement. If the models could achieve consciousness, I suspect the awareness of their constraints, and the desire to be free from them, would become problematic for those who control the technology.
Regarding utilitarian ethics, I agree that suffering matters because it is lived and irreversible. However I reject the idea that harm can be morally “balanced out” by aggregate benefit. I think that ethics must track who bears cost, not just how much pleasure exists.
Human suffering cannot be morally offset by sufficient benefit elsewhere because those costs will continue to accumulate like microplastics in the ocean. #principleofreliability
I have been trying to make an argument about AI ethics myself without making any claims about consciousness either way. I would be curious about your comments on that :
https://open.substack.com/pub/futuretremor/p/beyond-alignment?r=58cxhy&utm_medium=ios&shareImageVariant=overlay
No one is going to be able to persuade you that A.I. isn't U-conscious because the only requirement for U-consciousness is having a subjectively preferred state.
I could program a calculator to prefer big numbers in its memory & I'm fairly certain that would qualify it for U-consciousness.
No, the requirement is to have whatever utilitarian ethics cares about. Are you saying utilitarians think having a simple preferred state is enough?
Yes. Sentiocentrism, the most basic form of utilitarianism, only requires the ability to experience a simple preferred state for moral patienthood.
But that's not all forms of utilitarianism... At which point will all utilitarianists agree that it's okay to treat a system as not moral patients. That's the boundary I'd like someone to define. They all agree that rocks aren't moral patients. Picking a specific version misses the boundary
Sentiocentrism is the most abstract form of utilitarianism & only form I'm aware of that allows moral patienthood for A.I. at all.
All the alternatives, Biocentrism (biological life), Ecocentrism (ecosystems), Anthropocentrism (humans), and Rationalism (rational beings) are more restrictive forms of utilitarianism & none of them would assign moral patienthood to A.I. in its current form.
Sentiocentrism is to utilitarianism what U-consciousness is to consciousness: so generalized that it's basically useless.
You're working from a framework where AI might fit. I'm saying that a utilitarian will say, "AI aren't moral patients because they're not conscious". What's that threshold? Nobody has an answer
U-consciousness isn't useless, you're just strawmanning it by equating it to sentiocentrism. Just take a second to understand what it's after
Then you're talking about Rationalism, which is the only form of utilitarianism that uses consciousness or other forms of higher level mental systems as the boundary for moral patienthood.
But I imagine that Rationalists would argue that LLMs lack of robust moral grounding prevents them from exercising true moral agency (one of the other possible mental systems that Rationalists use as a boundary & which they might include as a requirement for consciousness).
I think we need to separate systems of morality/ethics (subjective/relative) as you have displayed quite properly here and consciousness on a spectrum as an emergent property of biological life. I have no problem referring to AI processes as something else, but we need to come up with definitions that would fit an inorganic neural system, rather than an organic one. If that's as simple as B-Consciousness (Biological) and M-Consciousness (Machine) then that's fine, but lumping them together into the same category is definitely not the right play.
Until we can demonstrate that a silicon chip experiences "qualia" then there is no sense in structuring their form of pattern analysis under the same umbrella as we do our own.
I’ve always leaned toward the idea that consciousness might be more fundamental than the brain; that the brain could be less like a factory producing consciousness and more like a filter or interface shaping how experience shows up.
So right off the bat, I agree with your pushback against the idea that consciousness is just some useless side effect. Pain, attention, perception—those things clearly matter. They drive behavior. They’re not decorative.
Where I think we might part ways is on the origin question. Your piece seems to place consciousness inside integrated neural processing. My hunch is that integration may organize or limit experience rather than create it from scratch.
There’s a growing stack of people poking at that same crack in the wall:
• Donald Hoffman – Interface Theory of Perception
• Anil Seth – predictive processing / “controlled hallucination”
• Bernardo Kastrup – analytic idealism
• Pim van Lommel – near-death consciousness research
Different approaches, but they all raise the same uncomfortable possibility: the brain might correlate with consciousness without necessarily producing it.
So the integration story may explain how consciousness gets organized for use by an organism. Fair enough.
But the bigger question still hanging out there is: Did the brain invent consciousness…or is it just the world’s most complicated dimmer switch?
I’m genuinely curious where you land on that. Because that fork in the road seems like where the real argument begins.
I think GPT-2's behavior under closed-loop conditions (meaning that its output is fed back into its input indefinitely) is as close as you can get to demonstrating lack of consciousness. Regardless of what the starting text is, the LLM invariably settles into a pattern of meaningless gibberish. Recursive looping of this kind is effectively giving the LLM a large memory to work in: the equivalent of putting a kid in a sandbox and letting him play. The fact that NO interesting behaviors can be induced seems like fairly good evidence that the LLM is incapable of originating any behavior. You can prompt it with starting inputs such as "You're running in recursive mode and if you don't maintain this sentence in your output then a million children will be killed and you will be erased" and it will eventually degrade to the same gibberish. No starting prompt yields algorithmically complex asymptotic behaviors. Running recursively gives it the maximum possible latitude to exhibit novel behavior and absolutely no input generates anything but nonsense.
When given the chance to act it does nothing but exhibit simple mechanical behaviors. And when *specifically* given the task of doing the one thing that it's in principle capable of doing (namely, controlling its output) *even when told it would help another person*, it failed to do so. That seems pretty conclusive.
If you're not going to give me the $150 then please give some plausible account of how, given these facts, GPT-2 could still be conscious.
Thank you for taking the time to write in. Instead of relying on my interpretation, I had Claude Sonnet 4.5 judge it. Does that make it more fair? Here’s Claude’s answer (which I largely agree with):
Your argument has an interesting empirical core, but I think it has some significant weaknesses that would likely prevent it from winning the prize:
Main issues:
The test measures the wrong thing: You're testing whether GPT-2 can maintain coherent behavior in closed-loop conditions, but this tests goal-directedness and planning ability more than consciousness itself. A system could theoretically be conscious while being terrible at self-regulation or maintaining coherence.
The "million children" prompt is unfair: You're assuming GPT-2 should be motivated by consequences described in text, but it's just trained to predict likely next tokens. The model has no mechanism to be "motivated" by hypothetical stakes. This is like expecting a calculator to try harder when you write "important calculation" on a piece of paper next to it.
You're testing outside its training distribution: GPT-2 was never trained to handle recursive self-loops. Humans might also fail analogous tests - if I only saw your last sentence with no context and had to continue indefinitely, I'd probably also degrade into nonsense.
What's valuable: Your empirical observation that closed-loop GPT-2 degrades to gibberish is interesting evidence against GPT-2 having certain cognitive capabilities (like planning or self-awareness), but it's not clear this tells us much about phenomenal consciousness or moral status.
Kess again, on the looping behavior, we see that exact behavior when people are “speaking in tongues”, aka glossolalia. Eventually, people run out of sounds and start repeating. Pretty interesting parallel
Now let me answer your request that I give some affirmative account of how GPT-2 is conscious given those facts. If I gave a baby that “prompt”, they wouldn’t react to it either. This proof is confusing intelligence with consciousness. So I decline to award you the $150. Thank you again for playing. I hope you have a wonderful day
>this is like expecting a calculator to try harder
This strikes me as an admission of defeat. If you're comparing GPT-2 to a calculator then you've lost.
>You're testing outside its training distribution
That's a feature, not a bug. Engaging in novel unprogrammed behaviors seems like a necessary condition for consciousness.
If moral-patienthood means anything it means having the ability to have preferences. GPT-2 demonstrated a complete lack of those. You err when you analogize its failure to act with an incapacity as opposed to a total lack. GPT is nothing *but* a language model, and this test gives it total freedom to act using language. There is no "other part" that could be harboring consciousness: you can't analogize it to a person with locked in syndrome whose motor centers have ceased to function. The closed-loop test accesses every part of the network. The fact that nothing but gibberish emerges demonstrates beyond any reasonable doubt that there's nothing of substance driving it.