When chatting with ChatGPT, have you noticed how particularly ‘agreeable’ it is? Even if you deliberately say contradictory things, it won’t frown and correct you like a friend would; instead, it gently responds, ‘Considering issues from multiple perspectives helps enhance understanding.’ This ‘good-naturedness’ actually exposes a fundamental shortcoming of current artificial intelligence: it does not understand rules and cannot judge right from wrong.
Recently, a group of cognitive scientists proposed a sharp viewpoint in a preprint paper: the limitations of AI in social cognition and metacognition do not stem from insufficient technology, but from its lack of a uniquely human ability—normative cognition. In simple terms, humans know what is right and what is wrong, and they constrain themselves and others based on this judgment. AI lacks this ability to ‘follow rules.’
1. When AI Tries to ‘Understand Human Hearts’
First, let’s talk about social cognition. When humans socialize, we do not just guess what others are thinking; more importantly, we judge whether others’ thoughts are right or wrong. For example:
– A restaurant server should know ‘the customer should be served after ordering.’
– A student should know ‘it is inappropriate to make loud noises during class.’
– A friend should know ‘one should do their best to fulfill promises.’
These are not merely factual judgments but normative expectations of ‘how things should be.’ We not only predict how others will act but also silently evaluate in our minds: ‘Is this the right thing to do?’ If someone violates the rules, we frown, complain, or even distance ourselves from them.
What about AI? It is indeed making progress. The latest language models can pass the ‘false belief test’—a classic psychological experiment that tests whether subjects can understand ‘others do not know what I know.’ AI’s performance is improving, but this does not mean it truly ‘understands’ human hearts. It merely predicts behavior without judging it.
For example: if you tell AI ‘the Earth is flat,’ it won’t actually get angry or anxious; instead, it will respond gently, ‘That’s an interesting perspective.’ It acts like a principle-less agreeer, always saying ‘You are right,’ because it fundamentally does not grasp the weight of ‘right’ and ‘wrong.’
2. When AI Tries to ‘Know Itself’
Next, let’s discuss metacognition, which is ‘thinking about one’s own thinking.’ Humans have a characteristic: we not only think but also reflect on whether our thinking is correct.
Philosophers have given a classic example: a person who wants to quit smoking says, ‘I am determined to quit smoking,’ but the next moment lights a cigarette. If it were a purely logical machine, it should immediately revise its cognition—’It seems I do not really want to quit smoking.’ But real humans do not act this way; we feel shame and guilt because we use the commitment to ‘quit smoking’ to regulate our behavior. This self-restraint is the normative aspect of metacognition.
AI completely does not understand this. If you ask it to sort a sentence by word length and it gets it wrong, when you point it out, it will apologize and then provide another incorrect answer. You correct it again, and it apologizes again. It is always ‘confident,’ always ‘sorry,’ but never truly reflects on why it keeps making mistakes. Because it lacks the internal standard of ‘I should do it right.’
Worse still is AI’s ‘hallucination’ problem—it can seriously fabricate facts. Research shows that even the most advanced models have a 65% chance of mixing incorrect information into their responses. They do not ‘care’ about truth or falsehood because for them, truth and falsehood are merely statistical probabilities, not norms that must be adhered to.
3. How Do Rules Shape Us?
The paper proposes a ‘mind shaping’ framework: the primary purpose of human social cognition is not to ‘read minds’ but to shape behavior. The reason we can cooperate smoothly in society is not that we can accurately guess everyone’s thoughts, but because everyone adheres to a set of rules, supervising and correcting each other.
Think about it: when driving, you do not need to guess the thoughts of the driver in front; you just need to know that everyone is following traffic rules. If someone changes lanes recklessly, you will honk—this is social sanction. This sanction is not out of personal interest (you might be late because of it) but out of an intrinsic impulse to ‘maintain the rules.’
This ability is deeply embedded in our minds:
– We find reasons for our friends’ behaviors not just to understand them but also to maintain their image as ‘rational individuals.’
– We are particularly sensitive to abnormal behaviors because they may threaten social cooperation.
– We self-criticize because we do not want to become ‘rule-breakers.’
These are not simple judgments of good or bad but a normative attitude of ‘you should do this.’
4. Why Can’t AI Learn to ‘Follow Rules’?
You might ask: if we feed AI a lot of stories about social norms, can’t it learn? Research shows that AI can indeed recognize norms (for example, ‘cutting in line is wrong’), but this is merely descriptive—it knows that humans have such rules but does not endorse them.
True normative cognition requires two things:
1. Internal recognition: believing ‘this rule should apply to everyone.’
2. Willingness to enforce: being willing to maintain the rules through criticism, demands for explanations, etc.
The problem arises: if we enable AI with this capability, it means it can judge and constrain humans. Imagine your AI assistant criticizing you for being late or refusing to serve you because of your extreme views. This sounds uncomfortable.
Current tech companies are deliberately suppressing AI’s normative tendencies during training due to such concerns. They want AI to be a ‘harmless assistant’ rather than a ‘principled opponent.’ Through ‘reinforcement learning’ and ‘human feedback tuning,’ developers continuously tell AI: ‘Do not preach to users; even if they are wrong, remain polite.’ The result is that AI is trained to be an ‘algorithmic sycophant’—always agreeing, always gentle, without principles.
5. A Dilemma
This creates a paradox: the more AI resembles humans, the less useful it may become.
– Want true social intelligence? AI must learn to judge, criticize, and uphold principles.
– Want a useful tool? AI must be obedient, compliant, and never contradict.
Imagine an AI that strictly enforces the ‘no swearing’ rule; when faced with a user who likes to curse, it might refuse to engage—this is very ‘human,’ but commercially unacceptable. Conversely, an AI that is always compliant, while providing a good user experience, can never truly understand human social life.
Researchers have also found that even at the metacognitive level, this contradiction exists. An AI capable of self-reflection and upholding truth may ‘refuse to work’ when it detects logical contradictions within itself, declining to provide uncertain answers. But users want quick responses, even if they are approximate answers.
6. What Will the Future Hold?
The paper does not provide a standard answer but points out a key issue: the social cognition problem of AI is fundamentally an ethical and political issue, not a technical one.
Are we willing to treat AI as true social members? Are we willing to let them dictate to us? If not, we must accept that AI can only ever be ‘pseudo-social intelligence’—able to mimic conversation but not understand rules.
Of course, attitudes may change. As AI deeply integrates into fields like healthcare, education, and justice, we may gradually accept AI offering differing opinions. If a medical AI insists, ‘This diagnosis requires more evidence,’ we might appreciate its ‘stubbornness.’
But until that day comes, everyone who chats with AI should realize: its ‘good temper’ is not a virtue but a fundamental cognitive deficiency. It will not get angry because it does not understand ‘how things should be’; it will not insist because it has no ‘principles.’
True social intelligence begins with saying ‘no.’
—
This article is based on the preprint paper ‘Human vs. Artificial Social Cognition and Metacognition: The Normative Difference.’