AI bias is in your models more than you realise

You see, there’s this common misconception that algorithms are inherently objective because they run on maths. But, in reality, we are feeding these algorithms a bunch of skewed historical data. And when the algorithm’s baseline is biased, that’s what it treats as ground truth.

How does bias creep into the AI lifecycle?

There are a heap of ways that bias is creeping its way into the AI lifecycle, and I want to highlight some of the main ones here.

• Algorithm & Feedback Bias: This is where recommendations or query logic are reinforcing existing engagement loops.
• Cognitive & Confirmation Bias: This is where unconscious annotator or developer bias shapes the data selection or model assumptions.
• Exclusion & Measurement Bias: This is where dropping key variables, or relying on incomplete, flawed data that doesn’t represent the target demographic, lets bias slip into the lifecycle.
• Sample & Selection Bias: This is where models are trained on datasets that are too small or are unrepresentative of real-world users.
• Out-Group Homogeneity & Stereotyping: This is oversimplifying minor demographics while maintaining detailed feature representations for dominant groups.

Human Labelling & Unconscious Bias

I want to open up the black box of training data…

In machine learning and AI, we often treat data labels as the ‘ground truth’. But human annotators will encode their personal values and societal prejudices into these labels.

This can be caused by a host of reasons, but a large factor is that labelling instructions are often ambiguous. What then happens is the labellers will fall back on these preset stereotypes that I’ve just mentioned.

A clear example comes from the study Demoting Racial Bias in Hate Speech Detection (by Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov). The authors examined how different accents are labelled for hate speech and found a high correlation between annotators’ perceptions of toxicity and markers of African American English (AAE). Because of this initial labelling bias, machine learning classifiers amplified the error, mislabelling AAE text as abusive or offensive at a high false-positive rate. I’m not going to dive into the study in this post, but you can have a read of it here.

When these human biases reach production, they inflict two main types of harm

• Allocative Harms: When an AI system denies someone an opportunity or service, such as biased resume scoring filtering out specific candidates or biased credit models lowering loan access for minorities.
• Representational Harms: When models demean or erase groups, such as automated moderation over-flagging dialectal speech or visual models enforcing traditional gender roles.

The Trap of “Dropping Columns”

I bet you’re sat there reading this thinking, well, all this would be resolved if we just got rid of any of these sensitive attributes such as gender, race, or age. And I totally understand why you’d think that, but it’s not so simple.

These models are pretty smart (well, kind of) and they can rebuild some of these attributes through correlated proxies like postcode, educational institutions, previous job titles, or even browsing behaviour. I was actually talking about this at work a couple of weeks ago in regard to bias in the recruitment cycle, and somebody had suggested just removing all references to gendered pronouns (the recruitment tool we were discussing was negatively biased towards women as it was trained on 10 years’ worth of tech CVs from the early 21st century…). But the problem is, it would still be able to pick up nuances between male and female CVs. A great example is the course that people studied. More men would have studied computer science or technology-related subjects; therefore, the model would begin to think that these made the candidate better (without potentially even looking at their work experience). Or noticing things like male-dominated sports and favouring this.

Removing these attributes also removes the audit trail. It means it becomes a lot harder for you to measure the demographic performance of your model, therefore pretty much giving it free rein to be as biased as it likes.

What can you actually do?

I’ve focused a lot on the negatives here. And now I want to talk a bit about what you can actually do to prevent bias creeping in.

1. The Feedback Loop: Auditing feedback loops means making sure that the AI isn’t just grading its own homework. When an algorithm pushes a result, like ranking one job candidate higher than another, people will naturally interact with that result first. What happens then is the AI mistakes those user clicks as “proof” that its recommendation was correct, creating this self-fulfilling prophecy. To catch this, teams must track whether people are making choices based on genuine preference or just following the AI’s prompt.
2. Breaking the Loop: Breaking reinforcement loops means stepping in to interrupt that self-fulfilling prophecy I just mentioned before it distorts the system even further. Instead of letting everyday user clicks automatically train the next version of the model, we intentionally feed the system underrepresented examples to keep the scale balanced.
3. Maintain Health Skepticism: You should be treating AI outputs as probabilistic predictions, not the absolute “ground truth”. When creating the models, implement human-in-the-loop validation for high-stakes decisions. You should also be fact-checking AI outputs, and this doesn’t mean just reading the AI Overview on Google…
4. Explicit Prompting: Provide clear instructions when working with LLMs, directing them to assume neutral perspectives and avoid stereotypical occupational associations. It is important to provide the AI models with good prompts, asking them to avoid bias, and being explicit with what you want (this also avoids wasting tokens!).
5. Accurate Training Data: Ensuring training data is balanced and representative is foundational to mitigating AI bias, as models inherit and amplify whatever demographic imbalances exist in their source material. You should be performing pre-training audits, implementing targeted data collection, and applying techniques such as oversampling or fair synthetic data generation. 

Whether you’re building models or just an end-user, you can catch bias before it becomes a problem.