AI Hiring Bias: Where It Comes From and How to Audit For It

The intuitive worry about AI hiring bias — “someone put race or gender into the model” — is almost never how it actually happens. Modern hiring tools are built to exclude protected characteristics. Bias gets in anyway, through the back door, and understanding that door is the whole game for anyone building, buying, or auditing these tools.

Bias doesn’t need the protected trait as an input

The core problem is proxies. A model that never sees gender can still learn to act on it if something correlated with gender is in the data. This is exactly what sank Amazon’s experimental recruiting model: it never used gender as a feature, but it learned from a decade of resumes that were mostly from men, and it started down-ranking resumes that contained the word “women’s” (as in “women’s chess club captain”) and graduates of two all-women’s colleges. Gender leaked in through language.

The usual proxy channels:

• Language and vocabulary. Words associated with one group — activities, schools, phrasing — become stand-ins for the group.
• Geography. Zip code and neighborhood correlate strongly with race and income in many countries. A model that likes certain areas can be redlining without ever naming race.
• Gaps and timelines. Penalizing employment gaps disadvantages people who took parental or caregiving leave — a disparate impact along gender lines.
• “Culture fit” trained on past hires. If you train a model to find people like your current successful employees, and your current employees skew one way, the model will faithfully reproduce that skew. It’s learning your history, not merit.

Disparate impact is the legal standard, and it doesn’t require intent

Under U.S. employment law, a hiring practice can be unlawful if it produces a disparate impact on a protected group, even with no intent to discriminate and no protected trait in the model. The long-standing rule of thumb from the EEOC’s Uniform Guidelines is the four-fifths rule: if a group’s selection rate is less than 80% of the highest group’s rate, that’s evidence of adverse impact worth investigating.

A worked example: if 50% of male applicants pass a screening tool but only 30% of female applicants do, the ratio is 30/50 = 0.60. That’s below 0.80 — a four-fifths red flag. The four-fifths rule is a rule of thumb, not a legal definition — a genuine, job-related justification can matter, and small samples can defeat the inference — but a ratio this low is a clear signal to scrutinize the tool.

What a real bias audit actually does

Regulations like NYC Local Law 144 now require independent bias audits of automated employment decision tools before use. A meaningful audit — not a rubber stamp — does roughly this:

1. Computes selection rates by group. Pass-through rates for the tool, broken out by sex and race/ethnicity categories.
2. Calculates impact ratios. Each group’s rate against the most-selected group’s rate, checked against the four-fifths threshold.
3. Uses real or representative data. Historical outcomes from the actual tool, or a defensible test set — not the vendor’s marketing sample.
4. Reports scoring rates too, not just hires. Where the tool assigns scores, the audit looks at scoring rates by group — the share of applicants scored above the sample’s median — not only the final yes/no.
5. Gets published. Local Law 144 requires a summary of results be made publicly available and candidates be notified.

The failure mode to watch for is the audit that tests a tool in the abstract, on the vendor’s data, and declares it “fair” — while your deployment, on your applicant pool, produces a very different impact. Impact is a property of the tool and the population it runs on. Audit the deployment, not the brochure.

If you’re buying one of these tools

Ask the vendor three questions and insist on real answers: What data was the model trained on? What were the impact ratios in the most recent independent audit, by group? And can you re-run that audit on our applicant data before we go live? A vendor who can’t or won’t answer is telling you something.