Job seekers, borrowers and patients could challenge AI decisions

A proposal in New York would set new guardrails for artificial intelligence systems that help make decisions about jobs, credit, housing, health care and other essential services. The measure, called the New York Artificial Intelligence Act, treats these tools as both a consumer protection issue and a civil rights concern.

Lawmakers describe artificial intelligence, or AI, as already woven into everyday services. Automated systems are used across industries including education, employment, insurance, finance, retail, media and public safety. Yet the technology often operates with limited transparency about how decisions are reached or how errors are caught.

When automated decisions shape real opportunities

The proposal focuses on what it calls “consequential decisions.” These are decisions that can materially affect a person’s opportunities or access to services, such as whether someone gets hired, receives financial aid, secures housing or qualifies for certain health or financial services.

Under the measure, companies that develop or deploy AI systems would have a legal duty to take reasonable care to prevent algorithmic discrimination. The bill defines that as unjustified differences in treatment that disadvantage people based on characteristics protected under civil rights law, such as race, sex, disability or religion.

The legislation frames the risk as more than a technical flaw. Without proper testing and oversight, automated systems could deepen existing inequalities by repeating patterns found in historical data.

Transparency and a chance to challenge AI decisions

The bill would require companies using certain high‑risk AI systems to notify people when those systems are involved in making significant decisions about them. The notice must be clear and consumer friendly and provided at the start of an interaction.

If an automated system helps make a consequential decision, the person affected would also need to be told afterward that AI played a role. Companies would have to provide a process for appealing that decision, including a way for the person to submit additional information and receive meaningful human review.

The proposal says the organizations using the technology remain responsible for the quality and accuracy of those decisions, including any bias that emerges from the system.

Responsibilities for developers and deployers

The measure places obligations on both the companies that build AI systems and the ones that use them in real‑world services. Developers of high‑risk systems would need to conduct regular audits designed to check whether they are taking reasonable steps to prevent discriminatory outcomes.