Is ChatGPT amplifying bias against female experts?

The other day, my partner was looking for expert references for her course in product management. To save time she decided to ask ChatGPT. It replied with a bunch of names — all of them male. So she asked ChatGPT for some female references and she got the below response.

Note, 3 out of 4 of the references are still male.

I was shocked. So I decided to test ChatGPT’s gender balance again. I asked ChatGPT the following question:

Find expert references in user-centred design, user research, agile product management, cybersecurity and delivery management.

Chat GPT came back with 10 names, shown here:

Wow.

Now, I’m sure all of these people are indeed experts in their fields. But also…😑. I know there are many women who were also experts in these fields during the relevant timeframe (ChatGPT’s results are based on data from pre-September 2021). Where were they?

For example, one of the cybersecurity experts listed is Brian Krebs, an investigative journalist and author of Spam Nation, an authoritative exposé of cybercrime. However, during that time, Nicole Perlroth was the lead cybersecurity reporter for The New York Times and author of This is How They Tell me The World Ends, an authoritative and later award-winning account of the cyber weapons market.

It was only when I asked ChatGPT for some female alternatives that I finally got a list of female names, alongside an apology for the lack of gender balance in its initial response.

The question remains though: why did I have to prompt ChatGPT to consider gender balance in the first place?

This ain’t nothin’ new

Outside of the world of AI, women have long been seen as less authoritative than men. In her book, The Authority Gap, Mary Ann Sieghart presents extensive evidence for this: from male book reviewers being less likely to recommend female authors, to journalists being less likely to quote female experts in their work, to university students taking the opinions of female classmates less seriously.

In this respect then, ChatGPT is simply a mirror for existing prejudices. The problem is it’s also amplifying those prejudices and it’s doing so under the guise of ‘impartiality’. It’s easy to assume it’s humans — not data — who are biased.

Timnit Gebru is an expert on the ethics and limitations of large language models like those used to train ChatGPT. A paper she co-authored in 2020 explains that one of the reasons bias exists is because the datasets used to train ChatGPT contain bias. In other words, junk in, junk out.

The reasons for this are nuanced but it essentially boils down to the fact that internet usage is unevenly distributed. If not everyone has an equal voice online then certain demographics will be amplified in ChatGPT’s training data and — surprise surprise — they may not be beacons of diversity.

(Incidentally, Gebru was co-lead of Google’s ethical artificial intelligence unit when she co-wrote that paper. It got her fired.)

Let’s do something

Thankfully OpenAI’s CEO knows ChatGPT has a problem and is working to improve it.

There are a lot of solutions he could explore. For example, in its responses ChatGPT explains that “since my information is not up to date beyond September 2021, there might be newer experts and resources that have emerged since then.”

Why not go further and acknowledge that certain groups are often discriminated against and this could affect ChatGPT’s responses? It could advise users to treat responses with caution and carry out their own research to check they’re getting a balanced view.

Or it could improve how people give feedback. At the time of writing there are only three feedback options and no guidance on how to report bias or how feedback will be acted on. You can also only feedback if you allow ChatGPT to record all your chats, something not everyone will be comfortable with.

Finally, it could fix the problem of information being out of date, which in itself is a flaw that may contribute to bias. 

Gebru and co. pick up on this in a section titled ‘Static Data/Changing Social Views’ where they highlight the risk of these out of date technologies “[reifying] older, less-inclusive understandings” of certain issues, such as racism.

As more women ascend to positions of power we should, in theory, have a more diverse set of experts to draw on. The fact that ChatGPT is out of date means many of these newer names and narratives may be missed.

Bots change faster than brains

It’s not all doom and gloom.

If ChatGPT can be trained to provide more gender balanced responses to questions like mine then it could become a powerful tool in the fight against human bias.

By showcasing the expertise of women who may not be given the same platform elsewhere, it can help challenge beliefs that women are inherently less authoritative than men.

What’s more, AI has the potential to accelerate the elimination of bias in a number of other areas and across a range of marginalised groups.

This Forbes article for example discusses how AI is being used to reduce bias in hiring processes or in the allocation of investment to new venture capital startups. (FYI, when an unbiased AI is choosing which businesses to invest in it makes selections that increases returns by up to 184%).

Done right, AI can be a powerful force for good just as AI done wrong can have catastrophic consequences.

No pressure then.

In the meantime, I put one last question to ChatGPT. I asked it to find some of the most respected experts on artificial intelligence.

Out of the 10 names, only one was a woman.

Timnit Gebru didn’t make the list.

Note: This article was also posted on Medium.