Build an Audit Trail That Reconstructs Hiring Decisions

A hiring algorithm’s model version is not enough to explain a change in candidate outcomes. Preserve the complete decision system that operated for each candidate: the model or rules engine, employer settings, job profile, inputs, outputs, workflow, human actions, rollout timing and dated approvals.

Choose the change layer, rollout and task; the checker shows the records to retrieve.

Version the Complete Decision System

Create a deployment record whenever any component capable of changing an outcome changes:

• vendor, product, tenant and hiring use case;
• model, algorithm or rules-engine version;
• resume parser, assessment or feature-extraction version;
• employer configuration, including inputs, weights, thresholds, knockout rules and ranking limits;
• job-specific profile, rubric and requisition version;
• workflow rules defining when a candidate advances, is held or is rejected;
• human-review and accommodation paths; and
• deployment start and end timestamps, including the time zone and any staged rollout.

Give that complete configuration an internal deployment ID. The ID should change when any outcome-affecting component changes, even if the vendor’s model number does not.

If the vendor does not expose a model version, record the vendor release identifier, release date and written description of what changed. “Latest version” is not an auditable identifier.

This approach follows the logic of the voluntary NIST AI Risk Management Framework. Its Govern Playbook recommends an AI-system inventory and current documentation covering testing and validation results, dependencies, deployment, monitoring and change-management plans. It also says governance policies should include third-party AI systems.

Link Each Decision to Its Inputs and Deployment

A useful decision-event record links four things: candidate, job, system deployment and decision.

Record Group	Preserve
Identity and timing	Pseudonymous candidate ID, requisition ID, event ID, timestamp and deployment ID
Inputs	Source type, an access-controlled reference to the immutable input used, parser output or assessment response, missing-data flags and transformations
Configuration	Model or build ID, job-profile version, enabled features, weights, thresholds, knockout rules and workflow state
Output	Raw and adjusted scores, rank or band, recommendation, reason codes and stage affected
Human action	Reviewer, action, override, stated reason and timestamp
Notice and exception	Applicable notice version, accommodation or alternative process offered, and whether it was used

Where appropriate, add an integrity hash to the input record. A hash can help establish integrity, but it is not a substitute for retaining retrievable content when that content is needed and lawfully preserved.

Where relevant, configuration records should also capture the random seed and execution environment. Those details may be necessary to reproduce an earlier result rather than merely approximate it.

The operational test is lineage: a team should be able to start with a rejection or advancement event and retrieve the system state and authorized input needed to examine the recommendation. A screenshot of current settings cannot establish what was active three months earlier.

Auditability should not become unrestricted surveillance. Restrict access, separate demographic audit data where appropriate, and retain only records with a defined purpose. The UK Information Commissioner’s Office tells recruiters to document detailed retention schedules, including how long each category is kept, why it is kept and what happens at the end. It also warns against indefinite “just in case” retention.

The ICO’s November 2024 AI in recruitment audit report recommends recording fairness, accuracy and bias risks in a data protection impact assessment.

Put Evidence Behind Every Release

Release notes alone do not show that a change was understood, tested and approved. Each change ticket should contain:

1. Change description: the old value, new value, business reason and affected jobs or locations.
2. Impact assessment: the candidates and workflow stages exposed, plus the privacy, accessibility and discrimination risks considered.
3. Pre-release comparison: old and proposed configurations tested on an appropriate fixed dataset, using relevant accuracy, error and group-outcome measures.
4. Approval: named product, recruiting, privacy or compliance, and technical owners according to the employer’s risk process.
5. Deployment evidence: exact activation time, rollout percentage and rollback point.
6. Post-release check: results by job or similarly situated cohort rather than only an organization-wide average.
7. Disposition: the decision to continue, adjust, pause or roll back, with reasons.

The ICO reported that audited providers tested changes to scoring or grading algorithms before rollout. A majority repeated accuracy tests after launch, especially before changes or updates. The regulator recommends documenting the training-and-testing process and the criteria that must be met before moving to the next development stage.

That provider-level testing is a baseline. Employers also need evidence for the configuration they actually use, including their thresholds, job profiles, workflow rules and enabled features.

Connect the release file to a post-deployment monitoring plan. Make access to version and change evidence a contractual requirement in the vendor validation process.

Investigate Changed Outcomes From the Deployment Boundary

When pass, interview or selection rates move unexpectedly, begin with the boundary between deployments rather than assuming that the model caused the change.

1. Freeze the relevant records. Preserve deployment history, decision events, human overrides and change tickets before routine deletion or another release obscures them.
2. Set the comparison window. Identify the last decision under the old deployment and the first under the new one. Account for staged rollouts that placed candidates on different configurations at the same time.
3. Check non-algorithm changes. Compare job mix, applicant volume, recruiting source, location, minimum qualifications and missing-data rates.
4. Reproduce representative decisions where authorized. Use the recorded inputs, configuration and execution details. Do not assume the current system will reproduce an earlier result.
5. Measure at the decision point. Compare score distributions, threshold crossings, advancement rates, errors and overrides for relevant cohorts and jobs.
6. Trace the mechanism. Determine whether the difference came from the model, parser, employer setting, job profile, data pipeline or human action.
7. Record and act. Document the findings, affected population, corrective action, candidate remediation considered and owner of follow-up monitoring.

This sequence distinguishes correlation with a release from causation. A vendor update and a new job profile might take effect on the same day, while an applicant-source change might alter the population being assessed. The investigation record should show which explanations were tested.

A Bias Audit Does Not Replace Decision Lineage

A periodic bias audit answers a different question from a candidate-level reconstruction.

In New York City, Local Law 144 applies to covered uses of automated employment decision tools. Before using a covered AEDT, an employer or employment agency must ensure that it has had a bias audit within the past year and provide the required notices.

The city’s official FAQ says the audit generally uses historical AEDT data. It also says the public summary must identify the audit date and the source and explanation of the data.

Those requirements do not, by themselves, reconstruct which configuration assessed a particular candidate. That requires a deployment ID tied to the candidate’s decision event, along with the applicable inputs, settings, output and human action.

Set Retention by Record Category and Purpose

There is no responsible universal retention number to insert into a hiring-algorithm policy. Set periods by record category after checking the jurisdictions involved, employment-record duties, privacy and deletion requirements, contracts, limitation periods, anticipated claims and litigation holds.

A model artifact may need a different retention period from candidate inputs or demographic audit data. A deployment manifest might remain useful after personal data tied to individual candidates should be deleted or anonymized.

Have qualified counsel review the schedule, then encode it in the system. For each category, specify the trigger date, deletion or anonymization action, hold process and responsible owner.

The practical standard is that records remain available long enough to investigate a challenged decision, while personal information is not kept indefinitely without a documented purpose.

This is informational, not legal or employment advice. Requirements vary by tool, employer and jurisdiction.