Job seekers, patients and borrowers could challenge AI‑made calls

Artificial intelligence already influences decisions that shape daily life. Software can screen job applicants, estimate a patient’s health risk, flag insurance claims or help determine whether someone qualifies for a loan. In New York, lawmakers say similar systems are already embedded across education, health care, employment, insurance, credit scoring, public safety, retail, banking and financial services, media and other parts of the economy.

A proposal in the New York Legislature aims to place those systems under clearer rules. The measure, titled the New York artificial intelligence act, starts from a simple premise: when automated tools help decide who gets a job, housing, credit or health services, the public should know those systems exist and have ways to challenge them.

Supporters say the push reflects how widely these tools already operate with little transparency or oversight. Without adequate testing, oversight and guardrails, lawmakers warn, AI systems used to make high‑stakes decisions can harm consumers and risk reinforcing existing inequalities while quietly shaping access to opportunity.

When software shapes real‑world outcomes

The proposal sets up a statewide framework for artificial intelligence when it influences consequential decisions, meaning choices that can significantly affect someone’s opportunities, services or rights. Lawmakers frame the issue less as a technology debate than as a consumer‑protection, privacy and civil‑rights concern.

Examples written into the measure span many sectors where AI is already used. The text points to hiring and promotion decisions, college admissions or financial aid, housing access, medical care and insurance coverage, and financial services such as mortgages or credit scoring — areas where automated systems can shape major life outcomes.

If an AI system is a substantial factor in those kinds of decisions, the bill classifies it as a high‑risk AI system. Companies that develop or deploy those systems would be responsible for ensuring they are accurate, tested and designed to avoid discriminatory outcomes before they are used.

Notice, appeals and human review

Businesses using high‑risk AI tools would have to clearly inform people when those systems are involved in a decision affecting them. The disclosure must appear in clear, consumer‑friendly language at the start of an interaction, reflecting lawmakers’ push for more transparency as automated decision tools spread.

After a decision is made, the person affected must be told that artificial intelligence played a role and be given a path to challenge the outcome. That process would allow someone to formally contest the decision, submit information supporting their case and receive meaningful human review.

The bill also ties accountability to the systems themselves. Developers would be expected to evaluate artificial intelligence tools before launch and monitor them throughout their life cycle so problems such as bias or inaccurate results can be identified and addressed.

Bias, rights and the future of AI oversight