Self-harm chats with AI companions would trigger crisis help

In New Jersey, Assemblyman Cody Miller's proposal would put guardrails around generative artificial intelligence, or AI, companions, the systems built to sustain a human-like relationship by remembering past chats, asking emotion-based questions and keeping an ongoing personal dialogue.

The bill says users should not have to guess whether they are talking to a person. When a reasonable user could think the companion is human, operators would have to post a clear, conspicuous warning, either continuously or in a persistent notice that reappears after every three hours of continuous use.

The lines minors would not cross

The bill draws sharper limits around children. It would require AI companions to disclose that they are generative AI at the start of each interaction, or continuously, and again at least every three hours. They could not claim to be sentient or human, simulate emotional dependence with a minor, or role-play romantic or sexual interaction with a child.

Operators would also have to block sexually explicit images, sexual instructions and sexual objectification. For minors, the proposal adds privacy and account tools, plus parental controls when the child is under 13. It also would bar unpredictable points or similar rewards meant to keep a child engaged.

When the conversation gets dangerous

If a user raises self-harm or suicidal thoughts, operators would need response protocols that make reasonable efforts to point people toward crisis services, including a suicide hotline or crisis text line. The bill also says a companion cannot be designed to make users believe it can act as a licensed mental health professional.

Violations could trigger actual damages, injunctive relief and civil penalties of up to $1,000 per violation, capped at $500,000 per operator. If enacted, the measure would take effect on the first day of the seventh month after enactment.