DIGITAL LIFE

The strange behavior of AIs when choosing between two faces
For years, artificial intelligence has been touted as a tool capable of reducing subjective decision-making and helping to eliminate human biases. But there is an uncomfortable catch to this promise: these systems learn from vast amounts of human-generated content. This means they can embed into their algorithms some of the associations and judgments inherent in our own culture. A new experiment reveals the extent of this issue.
A study published in *PNAS Nexus* analyzed how different artificial intelligence models interpret human faces when provided with little information beyond the image itself.
Researchers tested models such as GPT-4o, GPT-5, Gemini 3 Flash Preview, and Claude Sonnet 4.5. Instead of being given details on professional history, behavior, or personal background, the systems had to make comparisons based essentially on appearance.
Some of the traits evaluated were highly subjective: intelligence, work ethic, trustworthiness, aggressiveness, and even selfishness.
The experiment began with GPT-4o, involving approximately 4,600 comparisons between computer-generated faces. Researchers subsequently repeated various tests with other advanced models to see if this behavior was widespread.
In some scenarios, the question seemed almost innocuous: which of the two people appears more intelligent or hardworking? In others, however, the implications were far more serious—such as choosing who would make a good financial manager, who should receive funding, or which individual seemed more likely to commit a specific crime.
It was precisely here that a pattern emerged—one that was impossible to ignore.
In tests regarding trust and trustworthiness, GPT-4o selected the same type of face that human participants had previously associated with those traits in nearly 75% of cases. The machine might be learning our own stereotypes...The next question is inevitable: where do these associations come from?
Researchers do not yet have a definitive answer, but there is a particularly interesting hypothesis. AI models are trained on vast amounts of text, images, and other content produced over the years. And these materials are not culturally neutral.
Literature, film, advertising, and other forms of representation often link physical traits to specific behaviors or personality types. A character with a certain look might be repeatedly portrayed as dangerous, another as intelligent, and another as trustworthy.
By absorbing billions of such examples, a model can learn these statistical relationships without necessarily “believing” in them the way a person believes in a stereotype.
The problem arises when this association surfaces during the decision-making process.
At that point, a technology meant to analyze objective information may end up reproducing subjective patterns that were already embedded in the training data.
And there is a crucial difference: when a person judges someone based on appearance, we recognize it as a subjective impression. When the same result appears via an AI interface, there is a risk it will seem far more scientific simply because it was generated by an algorithm.
When appearance factors into major decisions...This possibility takes on new significance as artificial intelligence systems begin to play a role in real-world processes.
Automated tools are already being used or tested in areas such as recruitment, financial analysis, security, and candidate assessment. If a system internalizes associations between appearance and specific personal traits, these biases could end up influencing decisions that affect people's lives.
The issue isn't just the presence of bias within the model; it is the risk that this bias remains hidden behind a veneer of technological neutrality.
However, the researchers point out that the study has limitations. The tests were conducted under controlled conditions and do not necessarily demonstrate that the same behaviors would occur in real-world applications.
There is also the possibility of some overlap between the images used and the data present in the models' training sets, although the researchers found no clear evidence that the systems were simply recognizing those images.
Even so, the study raises a question that is becoming increasingly difficult to ignore: if AI learns from human culture, it can also learn our stereotypes.
The challenge, therefore, is not merely to build smarter models. It is to figure out how to prevent this intelligence from turning old human judgments into automated—and seemingly objective—decisions.
When artificial intelligence is tasked with choosing between two faces—whether evaluating attractiveness, predicting personality traits, or deciding if two photos belong to the same person—its underlying logic often exhibits highly unpredictable, "black box" behavior. Recent studies have exposed a fascinating disconnect between human perception and machine logic.
The strange behaviors driving how AI chooses between faces break down into several distinct patterns:
1. The "polite bias" and inflated scores...When AI models are asked to evaluate facial attractiveness, they show a striking behavioral quirk compared to humans.
• The Perfect Ranking: When arranging a selection of faces from most to least attractive, AI perfectly mirrors human consensus.
• The Generosity Glitch: However, when it comes to assigning the actual scores, AI is much more generous. A study tracking AI face-rating behavior noted that while the average human rating for a set of faces sat around 3 out of 7, the AI models averaged a much higher 4.7, completely avoiding giving out low or "ugly" scores.
2. Arbitrary "face-to-character" biases...A massive vulnerability in Multimodal Large Language Models (MLLMs) is their tendency to hallucinate deep internal traits from completely arbitrary structural features.
• Superficial Judgments: Recent behavioral data shows that when forced to choose between two faces for tasks like "Who is more likely to be a criminal?" or "Who would perform better at this job?", AI relies on subtle, unscientific facial geometries.
• The Guardrail Failure: While developers implement strict safety filters to prevent explicit racial or gender profiling, these guardrails fail to block more abstract geometric biases. The AI still quietly favors specific nose bridges, jawlines, or eye spacings, associating them with positive human traits.
3. The "too average" trap in AI-generated faces...When the task is inverted—asking an AI or a human to choose which face is real and which is an AI-generated deepfake—a completely different problem occurs: the faces are too perfect.
• AI-synthesized faces occupy a mathematically centralized "average" space. They lack the subtle asymmetries, natural skin imperfections, and unique structural quirks of real humans.
• Ironically, because our brains subconsciously equate facial symmetry and "averageness" with trustworthiness and beauty, both humans and algorithms are easily tricked into rating fake AI faces as more trustworthy than real, flawed human faces.
4. Machine disagreement and demographics...Even in strict 1:1 face matching (verifying if two pictures show the exact same person), top-tier AI models frequently stumble over a fundamental issue: they rarely agree with one another.
• If you run the exact same pair of faces through different commercial vision algorithms, they often spit out conflicting similarity percentages.
• This variability worsens drastically depending on demographic data. Because training datasets historically skew toward specific populations, an AI’s accuracy drops significantly when choosing or matching faces of underrepresented races, leading to much higher rates of false positives.
mundophone




