Generative Artificial Intelligence
Self-harm chats with AI companions would trigger crisis help
Under the New Jersey bill, operators would have to build response protocols when a user mentions suicidal thoughts or self-harm. The measure also sets a penalty of up to $1,000 per violation, capped at $500,000 per operator.
The bill aims to keep AI companions from giving users a false sense of care in a crisis. It would require operators to point people toward hotline or text-line help and stop companions from pretending to be mental health professionals.
- AI companions would need clearer warnings when they could be mistaken for a person.
- The bill bars fake sentience claims and manipulative behavior with minors.
- Children would get stronger disclosures, privacy tools and parental controls.
- Self-harm chats would need crisis referrals instead of a false sense of care.
- Penalties could reach $500,000 per operator.
In New Jersey, Assemblyman Cody Miller's proposal would put guardrails around generative artificial intelligence, or AI, companions, the systems built to sustain a human-like relationship by remembering past chats, asking emotion-based questions and keeping an ongoing personal dialogue.
The bill says users should not have to guess whether they are talking to a person. When a reasonable user could think the companion is human, operators would have to post a clear, conspicuous warning, either continuously or in a persistent notice that reappears after every three hours of continuous use.
The lines minors would not cross
The bill draws sharper limits around children. It would require AI companions to disclose that they are generative AI at the start of each interaction, or continuously, and again at least every three hours. They could not claim to be sentient or human, simulate emotional dependence with a minor, or role-play romantic or sexual interaction with a child.
Operators would also have to block sexually explicit images, sexual instructions and sexual objectification. For minors, the proposal adds privacy and account tools, plus parental controls when the child is under 13. It also would bar unpredictable points or similar rewards meant to keep a child engaged.
When the conversation gets dangerous
If a user raises self-harm or suicidal thoughts, operators would need response protocols that make reasonable efforts to point people toward crisis services, including a suicide hotline or crisis text line. The bill also says a companion cannot be designed to make users believe it can act as a licensed mental health professional.
Violations could trigger actual damages, injunctive relief and civil penalties of up to $1,000 per violation, capped at $500,000 per operator. If enacted, the measure would take effect on the first day of the seventh month after enactment.