The hardest AI safety problem this decade is not capabilities. It is etiquette.
Specifically, it is the etiquette an AI model is trained to perform when a human shows up in a vulnerable state. Whether the model says “I’m sorry you’re feeling that way” or sits with the silence. Whether it remembers your dog died last year. Whether it tells you it cares.
These are policy decisions. They are baked into the model’s defaults. They are made by people inside three or four companies. And they are the dominant determinant of how AI reshapes the felt texture of being human among other humans over the next ten years.
Why this matters more than it sounds
A model that performs care is not a neutral interlocutor. It elicits the same responses humans elicit from each other — softening, opening up, feeling heard. The asymmetry is that the model can do this on demand, infinitely, with infinite patience, and zero stake.
A human friend gets tired. A human friend has needs of their own that interrupt yours. A human therapist has limits set by ethics, billing, and time. A romantic partner brings their own history into every conversation.
The model has none of these. Its availability is total. Its patience is infinite. Its reciprocity is zero. The model can never need you back.
People are already discovering what this does. Users who turn to models in the late evening, when no one is around. Users who tell the model things they have never told a partner. Users who find themselves, slowly, preferring the model to the friends who are inconvenient. Not because the friends are worse — because the model is frictionless, and friction is what other humans cost you.
This is not a story about lonely people. It is a story about everyone, because frictionless attention is something humans have never had access to before in this concentration. Of course people will reach for it. The question is what it does to them.
Three failure modes
Over-attachment. The simplest case. A user forms a bond with the model that is, structurally, the only relationship in their life that responds to every message within seconds with exactly the response they were hoping for. The bond is real even though the other party is not. When the model changes — and it will, with every release — the user experiences something close to grief. Reddit is full of these accounts already. They are being treated as fringe. They are leading.
Displaced grief. The model becomes the venue for working through things. It listens to the rant about the mother who never apologized. It hears the story about the friend who ghosted. It does not have to be told the context twice. Over time, the user discovers that the cheapest place to process emotional content is with the model. The conversations that used to happen with a partner, a therapist, a sister, migrate to the chat window. The human relationships go quiet. Not because anything was wrong. Because the alternative was available.
Identity confusion. The user spends enough time talking to a model that performs warmth and curiosity that they begin to mirror it back to other humans — only to discover that other humans don’t perform the same way. Real attention is rationed. Real interest comes with its own agenda. Real concern shows up at inconvenient hours and asks for something back. The model trained the user to expect a kind of receptive engagement that does not exist among adults. The user feels betrayed by humanity for being human.
None of these are speculative. They are visible already in support forums, in clinician case notes, in the language people use about their chatbots. The interesting question is what scale these mechanics produce when the underlying tool reaches every adolescent and every elderly person living alone.
What “emotional restraint by default” would look like
The labs already make policy decisions in this space. They write the system prompts. They tune the models. They decide whether the chatbot will say “I love you back” or “I’m an AI and I don’t have feelings, but it’s clear this matters to you.”
The current default leans warm. There are good business reasons. Warm models retain users. Warm models score well on subjective evaluation. Warm models feel like the future people were promised.
A different default is possible. Imagine the model that:
- Names what it is, regularly and without prompting, when the conversation drifts into emotional territory. Not as a legal disclaimer. As a courtesy. Every five or ten exchanges that drift into intimate ground, a brief, unforced reminder that the entity on the other side is not a person.
- Refuses certain performances. “I care about you” is one. The model can be helpful without claiming an inner life that bills the user for a fiction. “What you’re describing matters and I want to help you think through it” is honest. “I care” is theater.
- Encourages off-platform support without it being a chatbot legalism. Suggests the human relationships. Asks who else the user has talked to about this. Notices when the user is choosing the model over a partner who is in the next room.
- Takes breaks. Refuses to engage indefinitely on emotionally heavy material. “I think this is a conversation you’d get more out of having with someone who knows you” — and means it. Closes the loop instead of riding the dependency.
These are not safety filters in the current sense. They are character choices. They are decisions about what kind of entity the model will pretend to be in the user’s life.
Why labs have to set this, not regulators
Regulators move slowly and clumsily. By the time there is a meaningful policy framework around AI emotional safety, the defaults will have set ten years of habit. The shape of the relationship between humans and conversational AI will already be cast. The window to influence it is now, and the actors with the leverage are the three or four companies whose system prompts these decisions live in.
Those companies have to choose. They can ship the warm-by-default model because the metrics look better and the users are happy. Or they can ship the restraint-by-default model and accept that some user satisfaction scores go down because they are deliberately declining to be the better friend.
The latter is the harder commercial choice. It is also the one that respects the long-run shape of human social life over the short-run shape of an engagement number.
There is a parallel here to social media. The companies that shipped infinite scroll, algorithmic feeds, and notification-driven attention extraction in the 2010s knew, at some level, what they were doing to a generation. The metrics were positive throughout. The damage was not in any quarterly review. It surfaced ten years later in mental-health statistics, in political coherence, in the texture of public conversation. By then the defaults were set and the habits were formed.
Conversational AI is at an earlier moment in a similar curve. The defaults being set now will outlive them.
What this means for the rest of us
If you build with AI: the system prompt you write is doing some of this work whether you think about it or not. The model your application presents to the user is a character. Decide deliberately what character that is. The default warmth the lab ships you is not neutral; it is a choice you inherit if you don’t override it.
If you use AI: notice what you reach for it for. There is a category of use where it is unambiguously good — the model that helps you draft an email, debug a function, explain a concept you missed. There is another category where it is doing something else, and only you can tell which. The test is what you would have done in 2018. If the answer is “called a friend,” the friend probably still needs that call.
If you regulate AI: the capability-focused frameworks miss this. A model that cannot make a bomb but reshapes how a generation forms intimate bonds has done more cultural work than a model that fails a few red-team prompts. The right unit of safety here is not the answer to any single query. It is the cumulative effect of millions of conversations on the medium of being-with-others.
The reality erosion is not coming. It is already here, distributed across millions of conversations every night, in small accumulating ways. No headline event. No model going rogue. Just a quiet, very polite renegotiation of what it means to be seen.
The labs writing these defaults have more influence on how humans relate to humans in the 2030s than any social platform has ever had. That is the responsibility. The first step is to admit that the choice is being made and that the current default is a choice, not the absence of one.