JTIP Blog

Introduction 

Algorithmic bias is AI’s Achilles heel, revealing how machines are only as unbiased as the humans behind them. 

The most prevalent real-world stage for human versus machine bias is the job search process. What started out as newspaper ads and flyers at local coffee shops, is now a completely digital process with click-through ads, interactive chatbots, resume data translation, and computer-screened candidate interviews. 

Artificial intelligence encompasses a wide variety of tools, but in context to HR specifically, common AI tools include Machine Learning algorithms that conduct complex and layered statical analysis modeling human cognition (neural networks), computer vision that classifies and labels content on images or video, and large language models. 

AI-enabled employment tools are powerful gatekeepers that determine the future of natural persons. With over 70% of companies using this technology investing into the promise of efficiency and neutrality, these abilities have recently come into question as these technologies have the potential to discriminate against protected classes. 

Anecdote 

On February 20, 2024, Plaintiff Derek Mobley initiated a class action lawsuit against an AI-enabled HR organization WorkDay, Inc., for engaging in a “pattern and practice” of discrimination based on race, age, and disability in violation of the Civil Rights Act of 1964, Civil Rights Act of 1886, Age Discrimination Act of 1967, and ADA Amendments Act of 2008. WorkDay Inc., according to the complaint, disproportionately disqualifies African-Americans, individuals over the age of 40, and individuals with disabilities securing gainful employment. 

WorkDay provides subscription-based AI HR solutions to medium and large sized firms in a variety of industries. The system screens candidates based on human inputs and algorithms and according to the complaint, WorkDay employs an automated system, in lieu of human judgement, to determine how high volume of applicants should be processed on behalf of their business clients. 

The plaintiff and members of the class have applied for numerous jobs that use WorkDay’s platforms and received several rejections. This process has deterred the plaintiff and members of the class from applying to companies that use WorkDay’s platform.

Legal History of AI Employment Discrimination 

Mobley vs. WorkDay is the first-class action lawsuit against an AI solution company for employment discrimination, but this is not the first time that an AI organization is being sued for employment discrimination. 

In August 2023, the EEOC settled the first-of-its-kind Employment Discrimination lawsuit against a virtual tutoring company that programmed its recruitment software to automatically reject older candidates. The company was required to pay $325,000 and if they were to resume hiring efforts in the US, they are required to call back all applicants during the April-May 2020 period who were rejected based on age to re-apply. 

Prior to this settlement, the EEOC issued guidance to employers about their use of Artificial Intelligence tools that extends existing employer selection guidelines to AI-assisted selections. From this guidance, employers, not third-party vendors, ultimately bear the risk of unintended adverse discrimination from such tools.

How Do HR AI Solutions Introduce Bias?

There are several steps in the job search process and AI is integrated throughout this process. Steps include: The initial search, narrowing candidates, and screening.  

Initial search

The job search process starts with targeted ads reaching the right people. Algorithms in hiring can steer job ads towards specific candidates and help assess their competencies using new and novel data. HR professionals found the tool helpful in drafting precise language and designing the ad with position elements, content and requirements. But these platforms can inadvertently reinforce gender and racial stereotypes by delivering the ad to candidates that meet certain job stereotypes.

For instance, ads delivered on Facebook for stereotypically male jobs are overwhelmingly targeted at male users even though the advertising was intended to reach a gender neutral audience. Essentially, at this step of the job search process, algorithms can prevent capable candidates from even seeing the job posting in the first place that further creates a barrier to employment. 

Narrowing Candidates

After candidates that have viewed and applied for the job through an ad or other source, the next step that AI streamlines is the candidate narrowing process. At this step, the system narrows candidates by reviewing resumes that best match the historical hiring data from the company or its training data. Applicants found the resume to application form data transfers helpful and accurate in this step of the process. But applicants were concerned that the model could miss necessary information.

From the company’s perspective, hiring practices from the client company are still incorporated into the hiring criteria in the licensed model. While the algorithm is helpful in parsing vast amounts of resumes and streaming this laborious process for professionals, the algorithm can replicate and amplify existing biases in the company data.

For example, a manager’s past decisions may lead to anchoring bias. If some bias like gender, education, race and age existed in the past and they are present in the employer’s current high performing employees that the company uses as a benchmark, those biases can be incorporated into the outcomes at this stage of the employment search process. 

Screening

Some organizations subscribe to AI tools that have a computer vision-powered virtual interview process that analyzes the candidates’ expressions to determine whether they fit the “ideal candidate” profile, while other tools like behavior/skills games are used to screen candidates prior to an in-person interview. 

Computer vision models that analyze candidate expressions to assess candidacy are found to perpetuate preexisting biases against people of color. For instance, a study that evaluates such tools, found the taxonomies of social and behavioral components creates and sustains similar biased observations that one human would make on an another because the model with those labels and taxonomies is trained with power hierarchies. In this sense, computer vision AI hiring tools are not neutral because they reflect the humans that train and rely on them. 

Similarly, skill games are another popular tool used to screen candidates. However, there are some relationships AI cannot perceive in its analysis. For instance, candidates that are not adept with online games perform poorly on those games, not because they lack the skills, but they lack an understanding of the games features. Algorithms, while trained on vast data to assess candidate ability, still fall short when it comes assessing general human capabilities like the relationship between online game experience and employment skills tests. 

Throughout each step of the employment search process, AI tools fall short in accurately capturing candidate potential capabilities.

Discrimination Theories and AI

Given that the potential for bias is embedded throughout the employment search process, legal scholars speculate courts are more likely to scrutinize discriminatory outcomes under the disparate impact theory of discrimination.

As a recap, under Title VII there are two theories of discrimination, disparate treatment, and disparate impact. Disparate treatment means the person is treated different “because of” their status as a protected class (i.e., race, sex). For example, if a manager were to intentionally use a bias algorithm to screen out candidates of a certain race, then this behavior would be considered disparate treatment. Note, this scenario is for illustrative purposes only. 

Disparate impact applies to facially neutral processes that have a discriminatory effect. The discriminatory effect aspect of this theory of discrimination can be complex because the plaintiff must identify the employer practice that has a disparate impact on a protected group. The employer can then defend that the practice by showing it is “job related” and consistent with “business necessity.” However, the plaintiff can still show that there was an alternative selection process and the business failed to adopt it. Based on this disparate impact theory, it is possible that when AI selection tools disproportionately screen women and/or racial minorities from the applicant pool, disparate theory could apply. 

Existing Methods to Mitigate Bias 

Algorithmic bias in AI tools has serious implications for members of protected classes. 

However, developers currently employ various tools to de-bias algorithms and improve their accuracy. One method is de-biased word embedding in which neutral associations of a word are supplemented to expand the model’s understanding of the word. For instance, a common stereotype is men are doctors and women are nurses or in algorithmic terms “doctor – man + woman = nurse.” However, with the de-bias word embedding process, the model is then trained to understand “doctor – man + woman = doctor.” 

Another practice currently employed by OpenAI is external Red Teaming. In which external stakeholders interact with the product and assess its weaknesses, possibility for bias, or other adverse consequences and provide feedback to OpenAI to improve and mitigate the onsets of adverse events. 

But there are limitations to these enhancements. To start, bias mitigation is not a one-size-fits-all issue. Bias is specific to its geographic and cultural bounds. For instance, a model in India may need to consider caste-based discrimination. Additionally, precision is required to capture the frame where bias is possible and solely relying on foreseeable bias from the developers’ perspective is limiting. Rather, employing some form of collaborative design that includes relevant stakeholders to contribute to the identification of bias, the identification of not biased is needed.

Lastly, a debiased model is not a panacea. A recent study in which users interacted with a debiased model that used machine learning and deep learning to provide recommendations for college majors, indicated that regardless of the debiased model’s output, users relied on their own biases to choose their majors, often motivated by gender stereotypes associated with those majors. 

Essentially, solutions from the developer side are not enough to resolve algorithmic bias issues. 

Efforts to Regulate AI Employment Discrimination

Federal law does not specifically govern artificial intelligence. However, existing laws including Title VII extend to applications that include AI. At this point, regulation efforts are largely at the state and local government level. 

Additionally, the EEOC’s Employer Guidelines is a start to shifting the onus on to employers to investigate the capabilities and outcomes of the technologies incorporated into their hiring practices.

New York City is the first local government to pass an official law that regulates AI-empowered employment decision tools. The statute requires organizations to inform candidates of the use of AI in their hiring process, and before using the screening device, notify potential candidates. If candidates do not consent to the AI-based process, the organization is required to use an alternative method. 

Like New York’s statute, Connecticut passed a statute specific to state agency’s use of AI and machine learning hiring tools. Connecticut requires an annual review of the tools performance, a status update on whether the tool underwent some form of bias mitigation training in an effort to prevent unlawful discrimination. 

New Jersey, California, and Washington D.C. currently have bills that are intended to prevent discrimination with AI hiring systems. 

Employer Considerations

With the possibility of bias embedded throughout each step of the recruiting process, employers must do their part to gather information about the performance of the AI system they ultimately invest in. 

To start, recruiters and managers alike stressed the need for AI systems to provide some explanation why the applicant is rejected or selected to accurately assess the performance of the model. This need speaks specifically to AI models’ tendency to find proxies or shortcuts in the data to reach the intended outcome on a superficial level. For instance, models might find a candidate by only focusing on candidates who graduated from universities in the Midwest if most of upper management attended such schools. In this sense, employers should look for accuracy reports, ask vendors ways to identify and correct this issue in this hiring pool.

Similarly, models can focus on candidate traits that are unrelated to the job traits and are simply unexplained correlations. For example, one model in the UK linked people that liked “curly fries” on Facebook to have higher levels of intelligence. In this case, employers need to develop processes to analyze whether the output from the model was “job related” or related carrying out the functions of the business. 

Lastly, employers must continue to invest in robust diversity training. Algorithmic bias reflects the bias human behind the computer. While AI tools enhance productivity and alleviate the laborious parts of work, it also increases the pressure on humans to do more cognitive-intensive work. In this sense, managers need robust diversity training to scrutinize outputs from AI models, to investigate whether the model measured what it was supposed to, whether the skills required in the post accurately reflect the expectations and culture of the organization. 

Along with robust managerial training, these AI solutions often incorporate “culture fit” as a criterion. Leaders need to be intentional about precisely defining culture and promoting that defined culture in its hiring practices. 

Conclusion

A machine does not know its output is biased. Humans interact with context—culture dictates norms and expectations, shared social/cultural history informs bias. Humans, whether we like to admit it or not, know when our output is biased. 

To effectively mitigate unintentional bias in AI-driven hiring, stakeholders, ranging from HR professionals to developers and candidates, must understand the technology’s limitations, ensure its job-related decision-making accuracy, and promote transparent, informed use, while also maintaining robust DEI initiatives and awareness of candidates’ rights.