Skip to content

Apps that make decisions about people

An app that helps decide credit, insurance, housing, employment or access to services can trigger legal duties whether or not it uses AI, and which duties apply depends on how the decision is made and where. In the US, existing laws already apply when software helps decide credit, insurance, housing or employment (FCRA and ECOA adverse-action notices, anti-discrimination law), and from 1 January 2027 California's CCPA rules add notice, opt-out and access duties (Colorado's SB 26-189 adds notice, explanation and human review on request). Quebec's Law 25 requires notice of decisions based exclusively on automated processing, and GDPR Article 22 restricts decisions based solely on automated processing with legal or similarly significant effects; decisions a person genuinely makes still carry transparency duties. Screen for it at intake with question Q5 (as of October 2026).

Who this is for

HR, finance and compliance leads, the enablement lead (triage), and builders of people-related apps. About 10 minutes.

Not legal advice

Not legal advice. This page summarizes laws as of October 2026 for a general audience. It was written with an AI legal-research persona and hasn't been reviewed by a lawyer for your situation. Ask your counsel.

Stop and call counsel before you build if any of these apply

This is the site's one full list. The US and Canadian laws and EU AI Act pages repeat only the items for their laws and link here.

  • The jurisdiction flag: the app handles information about people in Quebec, the EU or the UK, or about staff who work in the EU or UK, or an AI feature's output is used in the EU. If you don't know, answer yes. This includes being, or possibly being, the provider of an AI system whose output is used in the EU.
  • The consent gate: it collects biometric identifiers (fingerprints, face or voice prints), data about anyone under 18, or consumers' health information or other sensitive-category data outside HIPAA (for example, consumer health data under Washington's My Health My Data Act). Route to counsel before collecting.
  • Q5 ©: an outcome about a person's job, pay, discipline, credit, insurance, housing, education, health care or access to public benefits can take effect before meaningful human review.
  • California discipline or termination (SB 947): it supports discipline or termination of California staff. A review without corroboration fails part (3) of this site's four-part test, so Q5 is ©.
  • NYC hiring (LL 144): it helps decide hiring or promotion for New York City roles.
  • Credit and screening (FCRA, ECOA): it uses consumer reports or other third-party data about individuals, or supports a credit denial that needs specific principal reasons.
  • Q4 (d1): its output is posted to the general ledger or financial statements, or is itself the evidence for an account reconciliation or a control your auditors test, without being independently recomputed. Route it to your compliance owner too.
  • Audit scope: it's in your audit scope. Route it to your compliance owner before go-live (a gate, not a tier). The compliance owner's change-management and evidence requirements apply at whatever tier the app is, Team included; an audit-scope app gets no minimum tier (the audit-scope flag).
  • HIPAA or card data: it holds patient or health-plan-member health information for a HIPAA covered entity or business associate, or full card numbers (Q1 (d)).

Does this apply to an app that doesn't use AI?

Yes. The trigger is the effect, not the AI, and not whether the app was vibe-coded (see what vibe coding means for a business): a spreadsheet formula that ranks applicants, a no-code flow that auto-rejects a refund, or a prompt that summarizes candidates can all be covered.

Every worker-built app sits in one of four risk tiers (Sandbox, Team, Important or Critical), set by the highest answer to six questions about its data, users, internet reach, dependence, decisions about people and actions, and two floors apply at every tier. Q5 in the six tier questions sets the tier here:

Q5 answer Tier What it means here
(a) No Sandbox None of the rules on this page apply
(b) Yes, and every outcome is reviewed before it takes effect by a person who meets the meaningful-human-review test Important (legal) The notice, explanation and review controls below
© Yes, and an outcome can take effect before that review Critical (legal), plus graduation The "solely automated" rules (GDPR Art. 22, Quebec s. 12.1, the CCPA's "substantially replace" test, UK Art. 22A, California SB 947 for discipline and termination) come into play

A fixed, documented calculation (a commission plan, a rate card) isn't a decision about a person; answer Q4 and Q6 instead. Neither (legal) answer can be downgraded.

Don't: an unregistered spreadsheet or app in a decision path (AP17)

Unreviewed spreadsheets have driven consequential outcomes, such as the JPMorgan "London Whale" losses.1 Register any Q5 (b) or © app in the app registry before its output is used. See the failure patterns.

Which US laws apply?

Fast-moving

As of October 2026. Check the review date.

Law What it requires Who it applies to When Type
ECOA / Regulation B Adverse-action notices with specific principal reasons; no exception for complex models2 Any creditor, including businesses offering trade credit or payment plans; for business credit, modified notice rules apply (§1002.9(a)(3)), for example reasons given on the applicant's request In force Legal requirement
FCRA Adverse-action notices when a consumer report is used; a pre-adverse-action copy for employers. Assembling third-party data on people for eligibility decisions can make you a consumer reporting agency3 Users of consumer reports for credit, insurance, employment or tenancy In force Legal requirement
Title VII, ADA, ADEA No disparate treatment or disparate impact (statutory, 42 U.S.C. §2000e-2(k)); reasonable accommodation in assessments4 Employers covered by each statute (Title VII and the ADA start at 15 or more employees) In force Legal requirement
CCPA automated-decision (ADMT) rules Pre-use notice, opt-out (with exceptions such as a human appeal), access; a risk assessment before using ADMT for a significant decision5 Businesses meeting a CCPA threshold, including annual gross revenue above $26,625,000 (the figure since 1 Jan 2025; next adjustment due January 2027); employee and applicant data is covered Regulations effective 1 Jan 2026; ADMT duties for existing uses from 1 Jan 2027 Legal requirement
Colorado SB 26-189 Notice at the point of interaction; a plain-language description of the tool's role within 30 days after an adverse outcome; access and correction; after an adverse decision, the person may request human review or reconsideration; 3-year records6 Deployers of a tool that "materially influences" a consequential decision 1 Jan 2027 (SB 24-205 never took effect) Legal requirement
NYC Local Law 144 A bias audit by an independent auditor within one year before use, with impact ratios by sex and race/ethnicity (including intersectional categories); a published summary of the results; notice to candidates and employees at least 10 business days before use7 Employers and agencies using an automated employment decision tool for NYC hiring or promotion Enforced since 5 Jul 2023 Legal requirement
Illinois HB 3773 (Human Rights Act) No AI use with a discriminatory effect in employment decisions, no zip codes as a proxy, and notice of AI use8 Employers of Illinois staff and applicants Since 1 Jan 2026; notice rules not yet issued Legal requirement
California FEHA ADS regulations Anti-discrimination law applied to automated-decision systems; anti-bias testing is relevant to defenses; 4-year retention of ADS data9 Employers covered by FEHA Since 1 Oct 2025 Legal requirement
California SB 947 (secondary sources) For discipline and termination decisions only: no relying solely on an automated decision system, and if the employer primarily relies on its output, a human must corroborate the decision with the underlying data or other evidence; a written post-use notice10. This site's rule is deliberately broader than the statute's trigger: it requires corroboration for every California discipline or termination decision an app supports California employers, public employers included Operative 1 Jul 2027 Legal requirement (not yet operative)

Credit and screening. A credit-limit helper, collections prioritizer or screening sheet can be an ECOA or FCRA system. It must give the principal reasons for each adverse outcome, so an unexplained score can't support a denial. The CFPB's 2025 withdrawal of Circular 2022-03 didn't change that duty (details).11

Employment. The EEOC removed its AI guidance, and Executive Order 14281 deprioritizes disparate-impact enforcement, but the statutes didn't change, and private plaintiffs can still sue.12 Offer an accommodation path, keep disability-revealing inputs out, and run the adverse-impact and stability tests.

Definitions differ. The CCPA covers technology that replaces or substantially replaces human decisions; Colorado's "make, guide, or assist" test can also cover a tool that only recommends whom to interview.

Sector rules. FINRA Notice 24-09 applies existing broker-dealer rules to generative AI.13 SR 26-2 (April 2026) replaced SR 11-7 for bank model risk. It's "most relevant" to banks with over $30 billion in assets and applies to smaller banks in proportion to risk.14

Which Canadian rules apply?

  • Quebec Law 25, s. 12.1 (since 22 September 2023): an enterprise making a decision based exclusively on automated processing must tell the person by the time it tells them the decision. On request, it must disclose the information used and the reasons and principal factors, and let the person submit observations to a staff member who can review the decision.15
  • PIPEDA has no automated-decision rule. Bill C-36 (tabled 15 June 2026, awaiting second reading) would reportedly add one (secondary sources). It's a bill, not law.16
  • Ontario: since 1 January 2026, an employer with 25 or more employees that uses AI to screen, assess or select applicants for a publicly advertised job must say so in the posting.17
  • Federal public bodies (circumstance C2) follow the Directive on Automated Decision-Making; see US and Canadian laws.18

When do EU and UK rules reach you?

  • GDPR Article 22 gives a right not to be subject to a decision based solely on automated processing, including profiling, with legal or similarly significant effects. The exceptions (contract, law, explicit consent) need safeguards, including human intervention and a right to contest.19 In SCHUFA the CJEU held that a score can itself be the decision; in Dun & Bradstreet Austria, that the explanation must describe the procedure and principles actually applied.2021
  • Decisions a person genuinely makes are outside Article 22 but keep the transparency duties (Arts. 13–15) and, for systematic automated evaluation, an Article 35 impact assessment.
  • UK: the Data (Use and Access) Act replaced UK Article 22 with Articles 22A–22D from 5 February 2026. It permits solely automated significant decisions with safeguards.22

Don't: treat Article 22 as covering every decision with a person involved

Article 22 covers decisions based solely on automated processing. Decisions with meaningful human involvement still carry transparency and fairness duties. And the Digital Omnibus GDPR proposal, which would recast it, hasn't passed.19

What do these laws have in common?

Build these shared duties once, whichever way federal preemption goes.

Common duty Where it comes from Control in this framework Framework clause it supports
Notice before or at the decision CCPA, Colorado, NYC (10 business days ahead), Illinois, Quebec, Ontario, SB 947, GDPR Disclosure text in the app record; the Important-tier business review AI RMF GOVERN 3.2; ISO/IEC 42001 A.8.2; SOC 2 P1.1
Explanation of principal reasons ECOA, FCRA, Colorado, Quebec, GDPR Arts. 13–15 A reasons field per outcome, stored at decision time, never reconstructed later AI RMF MEASURE 2.9; ISO/IEC 42001 A.8.2
Correction of input data Colorado, CCPA access, Quebec, GDPR A route from the person to the data owner SOC 2 P5; ISO/IEC 27001 5.34
Meaningful human review Quebec, GDPR, UK, SB 947; Colorado (on request, after an adverse decision); the CCPA defines "human involvement" (its scope test and appeal exception) rather than imposing a review duty Q5 (b) and the review log below AI RMF GOVERN 3.2, MANAGE 4.1
Independent bias audit NYC LL 144 (legal requirement) Counsel and an independent auditor, before use and yearly; an internal eval doesn't meet it AI RMF MEASURE 2.11
Adverse-impact testing Title VII and FEHA defenses (good practice elsewhere) The internal tests below, in the eval set AI RMF MEASURE 2.11; ISO/IEC 42001 A.6.2.4
Records Colorado (3 years), FEHA (4 years), CCPA risk assessments (5 years) Retention set in the app record, within reach of a litigation hold ISO/IEC 27001 5.33; SOC 2 C1.2

Mapping a control doesn't make anyone compliant; it lets you reuse controls your audits already test.

What does meaningful human review require?

The phrase comes from Colorado's SB 26-189;6 the test is this site's four-part test, adapted from the CCPA rules' definition of "human involvement" (Cal. Code Regs. tit. 11 §7001(e), which has three elements).5 The reviewer (1) knows how to interpret the app's output, (2) reviews that output, (3) considers other relevant information, and (4) has the authority to change the decision. If any of the four is missing, the review doesn't count, and Q5 is answered ©.

  1. The reviewer sees the inputs and the reasons, not only a score.
  2. The reviewer can change the outcome without escalation, and the role says so.
  3. There's time to review. Seconds per case isn't review.
  4. The app logs who reviewed, what they saw, what they decided, and whether they changed the output.
  5. Someone watches the override rate. Near-zero overrides suggest rubber-stamping (good practice; no statute sets a rate).
  6. For discipline or termination of California staff (from 1 July 2027), the reviewer corroborates the output with the underlying data or other evidence and records where it's kept. Without that evidence, the output may not be used for that decision (SB 947),10 and under this site's test the review fails part (3), so Q5 is ©. The site applies this to every such decision, which is deliberately broader than the statute's "primarily relies" trigger.

Don't: rubber-stamp human review

A person who clicks "approve" without the information, training or authority to change the outcome fails this site's four-part test. The decision is then effectively automated: Q5 ©, Critical.

AI features in a decision path

AI features (modifier M1) that score, rank, summarize or flag people need more than the usual eval set.

A model's explanation isn't its reasons. Asked why it gave a score, a model gives a plausible rationale, not necessarily the factors that drove it; studies show stated reasoning often omits what actually influenced the answer.2324 When a model scores free text directly, there are no "factors actually used" to retrieve.

Don't: let a model write the reasons for an adverse decision

For a credit, insurance, housing or employment denial, the outcome and its reasons come from documented inputs and rules. The model may extract or summarize inputs for a person to check. A model-written rationale isn't the principal reasons §1002.9 and Dun & Bradstreet require.223

Store the record at decision time

Record the model version and change it only through a reviewed change. At decision time, store the inputs, the prompt and settings, the output, the reasons and the review, because re-running the model later won't reproduce them: hosted models vary even with "deterministic" settings, and providers retire pinned versions on published schedules.2526 Keep them for the longest retention period that applies.

Test for adverse impact and stability

Most businesses hold no protected-class data about applicants or customers, and Regulation B generally bars creditors from asking for it outside mortgage lending.2 These tests still work:

  1. Paired inputs: run matched cases that differ only in a name or another proxy for a protected trait (a graduation year, a zip code), and compare outputs. No demographic data is needed.
  2. Selection rates at the real cutoff: for a ranker, fix the operational cutoff (say, the top 10), then compare rates by group where you lawfully hold the data. Differences based on small numbers may mean nothing,27 so ask a statistician or counsel first.
  3. Stability: shuffle the input order and re-run. The ranking shouldn't change materially; language models comparing items are known to favor some positions.28

Run them before use, after every model or prompt change, and on a schedule, because hosted models change. They're good practice and support Title VII and FEHA defenses, but they aren't the independent audit NYC LL 144 requires.

Intake follow-ups when Q5 is yes

Intake asks these after a yes to Q5:

Follow-up Why it matters Laws it reflects
Which decision, and about whom (staff, applicants, customers)? Sets which regimes apply All
Does a person who meets this site's four-part test review every outcome before it takes effect? Q5 (b) versus © GDPR Art. 22, Quebec s. 12.1, CCPA, UK Art. 22A
Is it used for hiring or promotion for NYC roles? Independent bias audit and 10-business-day notice before use NYC LL 144
Is it used for discipline or termination of California staff? Corroboration evidence and post-use notice SB 947
Can the app give the principal reasons for each outcome, stored at decision time? Adverse-action and explanation duties ECOA, FCRA, Colorado, Quebec
Does it use consumer reports or third-party data about individuals? Route to counsel FCRA
Where do the people live or work (California, Colorado, NYC, Illinois, Ontario, Quebec, EU, UK)? Jurisdiction flag and counsel gate State laws, Law 25, GDPR, DUAA
Does it call an AI model (M1)? AI-feature controls and the EU AI Act Annex III, the Ontario posting rule, Illinois notice

Questions to ask your counsel

  1. Which of our decisions about people are covered, and in which jurisdictions?
  2. Which apps make or materially influence a covered decision under the CCPA, Colorado, NYC, Illinois or FEHA rules, and what do we owe by when?
  3. What would make our human review "meaningful" to a regulator?
  4. Which of our tools are an "automated decision system" under SB 947, and where will we keep corroboration evidence?
  5. Do we owe ECOA or FCRA adverse-action notices for any app-supported decision?
  6. Could any app make us a consumer reporting agency?
  7. Could any decision about Quebec, EU or UK people count as solely automated?
  8. Which hiring tools need an independent LL 144 audit, and which internal tests should we run on the rest?
  9. How long must we keep inputs, outputs and review logs?

Assumptions and evidence

  • A-008 (Low): Hypothesis. We expect builders to recognize "a decision about a person" at intake when Q5 lists the domains concretely; test this by auditing a sample of registered apps each quarter for Q5 cases builders didn't flag.
  • Evidence: statutes, regulations, regulator pages and research, read or located on 2026-10-21; secondary-sourced items are labelled. No study measures how well worker-built apps meet these duties.
  • Facts on this page are as of October 2026.

  1. JPMorgan Chase, Report of JPMorgan Chase & Co. Management Task Force regarding 2012 CIO losses, 16 Jan 2013, https://elischolar.library.yale.edu/ypfs-documents/454 [primary], accessed 2026-10-21. ↩

  2. 12 C.F.R. §1002.9 (notifications), https://www.consumerfinance.gov/rules-policy/regulations/1002/9/; §1002.5 (limits on requesting information), https://www.consumerfinance.gov/rules-policy/regulations/1002/5/ [statute], accessed 2026-10-21. ↩↩↩

  3. 15 U.S.C. §1681m, https://uscode.house.gov/view.xhtml?req=granuleid:USC-prelim-title15-section1681m&num=0&edition=prelim [statute], accessed 2026-10-21. ↩

  4. 42 U.S.C. §2000e-2, https://uscode.house.gov/view.xhtml?req=granuleid:USC-prelim-title42-section2000e-2&num=0&edition=prelim [statute], accessed 2026-10-21. ↩

  5. California Privacy Protection Agency, approved text of the CCPA updates, cybersecurity audit, risk assessment and ADMT regulations, https://cppa.ca.gov/regulations/pdf/ccpa_updates_cyber_risk_admt_appr_text.pdf; threshold adjustment, https://cppa.ca.gov/regulations/cpi_adjustment.html [regulator], accessed 2026-10-21. ↩↩

  6. Colorado General Assembly, SB 26-189, https://leg.colorado.gov/bills/sb26-189 [statute], accessed 2026-10-21. ↩↩

  7. NYC Department of Consumer and Worker Protection, Automated Employment Decision Tools, https://www.nyc.gov/site/dca/about/automated-employment-decision-tools.page [regulator], accessed 2026-10-21. ↩

  8. Illinois Public Act 103-0804, https://www.ilga.gov/legislation/publicacts/fulltext.asp?Name=103-0804 [statute], accessed 2026-10-21. ↩

  9. California Civil Rights Department, Civil Rights Council secures approval for regulations on AI in employment, 30 Jun 2025, https://calcivilrights.ca.gov/2025/06/30/civil-rights-council-secures-approval-for-regulations-to-protect-against-employment-discrimination-related-to-artificial-intelligence/ [regulator], accessed 2026-10-21. ↩

  10. California SB 947 (reported as Stats. 2026, ch. 859), Labor Code §§1520–1526.7, https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202520260SB947 [statute; the page refused our tools, so the scope, the 1 Jul 2027 operative date and the chapter number come from law-firm summaries only, secondary], accessed 2026-10-21. ↩↩

  11. CFPB, withdrawal of 67 guidance documents, 90 FR 20084, 12 May 2025, https://www.govinfo.gov/content/pkg/FR-2025-05-12/html/2025-08286.htm [regulator], accessed 2026-10-21. ↩

  12. Executive Order 14281, Restoring Equality of Opportunity and Meritocracy, 23 Apr 2025, https://www.whitehouse.gov/presidential-actions/2025/04/restoring-equality-of-opportunity-and-meritocracy/ [primary], accessed 2026-10-21. ↩

  13. FINRA Regulatory Notice 24-09, https://www.finra.org/rules-guidance/notices/24-09 [regulator], accessed 2026-10-21. ↩

  14. Federal Reserve, SR 26-2, Revised Guidance on Model Risk Management, 17 Apr 2026, https://www.federalreserve.gov/supervisionreg/srletters/SR2602.htm [regulator], accessed 2026-10-21. ↩

  15. Act respecting the protection of personal information in the private sector, CQLR c. P-39.1, https://www.legisquebec.gouv.qc.ca/en/document/cs/P-39.1 [statute; the page refused our tools on 2026-10-21, so the wording is from earlier research], accessed 2026-10-21. ↩

  16. Bill C-36 (45th Parliament, 1st session), summary and status at https://openparliament.ca/bills/45-1/C-36/ [secondary]; the official page https://www.parl.ca/legisinfo/en/bill/45-1/c-36 couldn't be read by our tools [primary], accessed 2026-10-21. ↩

  17. Ontario Regulation 476/24, https://www.ontario.ca/laws/regulation/240476 [statute; page not machine-readable, details from Ontario's employer guidance], accessed 2026-10-21. ↩

  18. Treasury Board of Canada Secretariat, Directive on Automated Decision-Making, https://www.tbs-sct.canada.ca/pol/doc-eng.aspx?id=32592 [primary], accessed 2026-10-21. ↩

  19. Regulation (EU) 2016/679 (GDPR), https://eur-lex.europa.eu/eli/reg/2016/679/oj [statute], accessed 2026-10-21. ↩↩

  20. CJEU, Case C-634/21, SCHUFA Holding, 7 Dec 2023, https://curia.europa.eu/juris/liste.jsf?num=C-634/21 [court], accessed 2026-10-21. ↩

  21. CJEU, Case C-203/22, Dun & Bradstreet Austria, 27 Feb 2025, https://curia.europa.eu/juris/liste.jsf?num=C-203/22 [court], accessed 2026-10-21. ↩

  22. Data (Use and Access) Act 2025, s. 80, https://www.legislation.gov.uk/ukpga/2025/18/section/80; commencement regulations SI 2026/82, https://www.legislation.gov.uk/uksi/2026/82/made [statute], accessed 2026-10-21. ↩

  23. Turpin, Michael, Perez and Bowman, Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting, 2023, https://arxiv.org/abs/2305.04388 [research], accessed 2026-10-21. ↩↩

  24. Anthropic, Reasoning models don't always say what they think, 3 Apr 2025, https://www.anthropic.com/research/reasoning-models-dont-say-think [research; a model developer's own study], accessed 2026-10-21. ↩

  25. Atil et al., Non-Determinism of "Deterministic" LLM Settings, 2024 (revised Apr 2025), https://arxiv.org/abs/2408.04667 [research], accessed 2026-10-21. ↩

  26. Two examples of published model retirement schedules: Anthropic, Model deprecations, https://platform.claude.com/docs/en/about-claude/model-deprecations; OpenAI, Deprecations, https://developers.openai.com/api/docs/deprecations [vendor], accessed 2026-10-21. ↩

  27. Uniform Guidelines on Employee Selection Procedures, 29 C.F.R. §1607.4(D), https://www.law.cornell.edu/cfr/text/29/1607.4 [statute], accessed 2026-10-21. ↩

  28. Wang et al., Large Language Models are not Fair Evaluators, 2023, https://arxiv.org/abs/2305.17926 [research], accessed 2026-10-21. ↩