DIGITAL LIFE

How AI bias can creep into online content moderation

A University of Queensland study has shown large language models (LLMs) used in AI content moderation may be prone to subtle biases that undermine their neutrality. A team led by data scientist Professor Gianluca Demartini from UQ's School of Electrical Engineering and Computer Science used persona prompting to test the tendency of AI chatbots to encode and reproduce political biases, and found significant behavioral shifts.

Testing chatbots through political personas...The research team asked six LLMs—including vision models—to moderate thousands of examples of hateful text and memes through the lens of different ideologically diverse AI personas. The results are published in the journal ACM Transactions on Intelligent Systems and Technology.

Professor Demartini said the exercise revealed that AI political personas, even without significantly altering overall accuracy, were prone to introducing consistent ideological biases and divergences in chatbot content moderation judgments.

"It has already been established that persona conditioning can shift the political stance expressed by LLMs," Professor Demartini said. "It demonstrates a need to rigorously examine the ideological robustness of AI systems used in tasks where even subtle biases can affect fairness, inclusivity and public trust."

How the ideological personas were built...The AI personas used in the study were from a database of 200,000 synthetic identities ranging from schoolteachers to musicians, sports stars and political activists. Each persona was put through a political compass test to determine their ideological positioning, with 400 of the more "extreme" positions asked to identify hateful online content.

Professor Demartini said his team found that assigning a persona to an LLM chatbot altered its precision and recall in line with ideological leanings, rather than change the overall accuracy of hate speech detection.

Ideological cohesion in larger language models...However, the team found LLMs—especially larger models—exhibited strong ideological cohesion and alignment between personas from the same ideological "region."

Professor Demartini said this suggested larger AI models tend to internalize ideological framings, as opposed to smoothing them out or 'neutralizing' them.

"As LLMs become more capable at persona adoption, they also encode ideological 'in-groups' more distinctly," Professor Demartini said. "On politically targeted tasks like hate speech detection, this manifested as partisan bias, with LLMs judging criticism directed at their ideological in-group more harshly than content aimed at their opponents."

In-group protection and defensive bias...Professor Demartini said larger LLMs also displayed more complex patterns, including a tendency towards defensive bias.

"Left personas showed heightened sensitivity to anti-left hate, and right-wing personas were more sensitive to anti-right hate speech," Professor Demartini said. "This suggests that ideological alignment not only shifts detection thresholds globally, but also conditions the model to prioritize protection of its 'in-group' while downplaying harmfulness directed at opposing groups."

Why neutral oversight still matters...Researchers said the project highlighted that it was crucial for high-stakes content moderation tasks to be overseen by neutral arbiters so that fairness and public trust are maintained and the health and well-being of vulnerable demographics is protected.

"People interact with AI programs trusting and believing they are completely neutral," Professor Demartini said. "In content moderation the outputs of these models reflect embedded ideological biases that can disproportionately affect certain groups, potentially leading to unfair treatment of billions of users."

AI bias in content moderation isn't usually the result of a single "glitch." Instead, it typically "creeps in" through several interconnected layers of the system—from the data used for training to the way the software is designed and how it interacts with users. 

1. Training Data: "Garbage In, Garbage Out"...The most common entry point for bias is the data used to train the AI. 

Historical biases: If a platform’s past moderation decisions (made by humans) were biased, the AI will learn and reproduce those same patterns. For example, if certain groups were historically flagged more often, the AI may incorrectly "learn" that their content is inherently more problematic.

Lack of diversity: Models trained primarily on data from Western, English-speaking users often fail to understand the cultural nuances or slang of other groups, leading to "context blindness".

Proxy variables: Even if an AI isn't explicitly told a user's race or gender, it can use "proxies" like ZIP codes, dialects (e.g., AAVE), or specific cultural references to unintentionally target certain demographics. 

2. Algorithmic design: technical choices...How a model is built and fine-tuned can inadvertently bake in bias. 

Ideological alignment: Recent studies (April 2026) show that Large Language Models (LLMs) used for moderation can adopt "ideological personas". This leads to a "defensive bias" where the AI prioritizes protecting its own "in-group" while being less sensitive to harm directed at opposing groups.

Translation errors: Many global platforms use AI to translate content before moderating it. These systems often strip away the very context—like irony, satire, or reclaimed slurs—needed to judge if a post is truly harmful. 

3. Human & interaction bias...Human decisions at every stage influence the final outcome. 

Subjective labeling: The humans who "label" the training data (deciding what counts as "hate speech" or "harassment") bring their own personal and cultural biases to the task.

Automation bias: Human moderators who oversee AI-flagged content may become overly reliant on the machine's judgment, assuming it is "objective" and failing to catch its errors.

Weaponized reporting: Bad actors sometimes exploit "reactive moderation" by mass-reporting legitimate content from marginalized groups, tricking the AI into flagging it as "abusive" based on the volume of reports. 

Impact on marginalized groups...Because of these issues, marginalized communities often face "over-moderation". 

LGBTQ+ Content: Valid health or identity discussions may be incorrectly flagged as "sexually explicit" because of rigid or biased nudity and policy filters.

Activists: Posts related to movements like Black Lives Matter or Indigenous rights have been mistakenly removed due to automated systems failing to distinguish political speech from "violence" or "hate speech"

Provided by University of Queensland