At 7:42 on a rainy Monday morning, Mei Tanaka stopped in front of the security gate of a newly renovated research facility.
The gate had a camera mounted above it. A small screen displayed the familiar message:
IDENTITY VERIFIED.
Mei had seen systems like this before. At airports, office entrances, smartphone locks, and increasingly in public infrastructure, facial recognition had become almost invisible. The technology did not literally look only at a person’s eyes, nose, and mouth. It extracted mathematical features from an image—the relative positions of facial landmarks, contours, textures, proportions, and other patterns—and converted them into a numerical representation, often called an embedding, that could be compared with stored templates.
But there was something else that interested Mei.
She was a computer-vision engineer, and she knew that the image reaching the algorithm was never simply a face.
It was light.
The camera measured reflected light, and that measurement was affected by illumination, exposure, sensor characteristics, white balance, compression, camera angle, and the physical properties of the skin. Skin tone could therefore influence how reliably a face was captured and recognized, even when the algorithm had not been explicitly designed to identify a person’s race.
That distinction mattered.
“Does the system know I’m Japanese?” her colleague Daniel asked one afternoon.
Mei paused.
“That’s not quite the right question.”
She opened a visualization of the system’s output.
“Face recognition and race classification are different tasks. A recognition system can use visual information that correlates with demographic characteristics without explicitly producing a label saying ‘Japanese,’ ‘Black,’ or ‘White.’ And if someone deliberately builds a biometric categorization system to infer race or ethnicity, that’s an entirely different and much more sensitive application.”
The problem was that mathematics did not automatically understand the difference between a useful visual signal and a socially dangerous category.
NIST’s continuing evaluations of face-recognition algorithms have shown that demographic differences in error rates remain measurable. The reasons are not simply “the algorithm sees skin color.” Poor image capture—such as underexposure of darker skin—can increase false non-matches, while differences in training data and representation can contribute to false matches even when photographs are of good quality. (NIST Pages)
In other words, the camera could introduce inequality before the artificial intelligence had even begun making its comparison.
Mei remembered an experiment from several years earlier.
The team had photographed volunteers under different lighting conditions. Under carefully controlled illumination, the recognition system performed extremely well. Then they changed the lighting.
A person whose face had looked perfectly clear in one photograph became a much poorer biometric sample in another. Shadows deepened. Highlights disappeared. Darker skin could become underexposed, while lighter skin could become overexposed. The problem was not that the machine had suddenly become racist.
The problem was that the physical world had entered the mathematical pipeline.
Person → camera → pixels → preprocessing → embedding → comparison → decision.
Every arrow was an opportunity for error.
And every error could acquire a social meaning once the machine was used to open a door, flag a traveler, identify a suspect, or deny someone access.
That was why Mei disliked the phrase “99% accurate.”
Ninety-nine percent sounded reassuring until someone multiplied the system across millions of encounters. A tiny error rate could produce a large number of mistakes, and those mistakes might not be distributed evenly among different populations. The European Commission similarly warns that facial-recognition accuracy can depend on factors including camera quality, lighting, distance, databases, algorithms, age, sex, and ethnicity.
The ethical question, therefore, was not simply whether the machine could recognize a face.
It was:
Whose face does it recognize reliably?
And:
Who bears the consequences when it does not?
Later that month, Mei attended a conference where a lawyer presented a surprising development.
Europe’s AI Act had moved beyond abstract ethical principles into concrete restrictions. Among the prohibited practices is biometric categorization designed to infer certain protected characteristics, including race. The Act also treats several biometric applications as high-risk and imposes requirements intended to reduce discriminatory outcomes, including the use of representative datasets and ongoing monitoring.
The room became quiet.
Technology had once been described as something that would eventually make human judgment obsolete.
Now lawmakers were asking technology to prove that it deserved to participate in human judgment at all.
Mei returned to the laboratory that evening.
The security camera above the entrance was still watching the corridor.
She looked at it differently.
For years, people had worried that machines might eventually become intelligent enough to understand humans.
But perhaps another problem had arrived first.
Machines were already capable of measuring humans extremely well.
The difficulty was deciding which measurements should matter.
A face could be reduced to vectors. Skin could be reduced to numerical values. Similarity could be expressed as a score between zero and one.
But a person could not be reduced so easily.
The danger was not necessarily that a machine would explicitly decide what race someone belonged to.
It was that engineers might gradually turn characteristics associated with race into invisible variables inside systems whose decisions affected people’s lives—without anyone noticing where the ethical boundary had been crossed.
Mei shut down her workstation.
On the screen, the final line of her testing report remained:
ACCURACY IS NOT THE SAME AS JUSTICE.
She added one sentence beneath it.
And better technology does not eliminate the need to decide what technology should never be allowed to do.
All names of people and organizations appearing in this story are pseudonyms
Comments