What an error by Literary Digest in 1936 teaches us about Amazon and AI bias today

by Dr Clément Chastagnol, Head of Data Science, Sidetrade

On 3 November 1936, Franklin D. Roosevelt was triumphantly elected President of the United States, with over 60% of the votes, much to the embarrassment of Literary Digest, a leading magazine, which had predicted a landslide victory for his Republican opponent, Governor Alfred Landon of Kansas. The poll used by the magazine was skewed, to say the least.

By today’s standards, in fact, the bias of the astronomical 2.4 million-person sample would even raise a few smiles.

The respondents were all readers of the magazine, registered vehicle owners, and telephone users. In other words, considering that this was the Great Depression, Literary Digest’s respondents were a representative sample of the wealthiest population of the country!

On the other hand, using a comparatively tiny sample of just 5,000 respondents, the statistician George H. Gallup predicted a Roosevelt victory with 56% of the vote.

This event saw the birth of the science of opinion polling. The 1936 American election is known by every statistics student.

History repeats itself

Skip forward eight decades to Amazon’s facial recognition system, which proved to be inept at recognizing dark-skinned subjects. What might have been considered as a rather ordinary mishap in a lab, caused an uproar in the scientific community, once it was understood that the application was to be used in vivo by law enforcement agencies.

Amazon made the same blunder as Literary Digest: using non-representative data. Since the machine learning was based mostly on fair-skinned individuals, it logically proved to be faulty recognizing subjects outside of its field of reference.

When the problem is posed in these terms, the trick to solving it seems simple: just provide the algorithm with a sufficiently diverse corpus of data to improve its performance. True. But what do you do if the data is rare, expensive to come by, or worse, when bias is consubstantial with the data?

Where does bias in artificial intelligence systems come from? Is it intentional? Is there a way to avoid it or sustainably eliminate it? Are the solutions only technical?

The issue is considered serious enough for governments to start addressing it. In fact, the European Commission has recently presented a set of Ethics Guidelines for Trustworthy AI, aiming to establish an appropriate legal and ethical framework for artificial intelligence. It is true that there have been shockwaves from scandals confirming these worries.

And the ethical concerns are not confined to AI applied to the public at large. The business world also needs to sit up and take note.

AI ethics should be a business consideration, too

AI is becoming an indispensable part of digital transformation in business. Strategic choices, winning new markets, customer relations, financial management — AI holds a promise of enhanced efficiency in every profession and every sector.

According to IDC, a leading provider of market intelligence, worldwide investment in artificial intelligence (AI) systems is expected to double by 2022.

Let’s take the case of an algorithm that we want to train to determine credit risk, something businesses will be all too familiar with.

Using machine learning, the system will analyze all the loans handled by the bank over the last ten years: which loan applications were accepted, at what rate, according to various criteria (e.g. revenue, family status place of residence).

But by doing so, the system is sure to reproduce the biases of the human decisions of that period. For example, if applicants from a specific geographic area tended to be rejected, or if they were required to pay higher interest rates, the prejudice would be perpetuated by the machine.

What had been simply a statistical bias observed in a limited dataset would become a systematic rule for the algorithm. What’s worse is that this rule would be implicit, for in most cases, the algorithm is a “black box”, which is not intended to explain its choices.

In fact, it seemed to be at play in recent months, with prominent users of Apple Card, the new credit card service by Apple, such as David Hansson (co-founder of Basecamp) or Steve Wozniak (co-founder of Apple), reporting that wide differences in credit limit with their respective wives, even as they share the same account.

So far the communication by Apple seems to have been insufficient, in part because it seems hard to explain the decisions made by the algorithm behind the scenes.

In her book Weapons of math destruction, data scientist and AI fairness expert Cathy O’Neil lays out the danger of algorithms that are solely intended to create short-term value: they tend to exacerbate inequalities.

However, algorithms are far easier to audit than people, and they should be, particularly when they impact the experience of users, and above all, the lives of private citizen.

How to fix the problem

What, then, can be done? Over and beyond technical solutions, the real answer is a strategic approach, and that, we must say, is good news indeed!

An AI application is a computer program, and as such, it pursues an objective determined by its designer. In the case of our example, AI is intended to reproduce the banker’s decision-making process as closely as possible.

This is making the implicit assumption that the human decisions encoded in the data are perfect and that deviating from those (or, in technical terms, generating a higher error-rate) is to be avoided.

To proceed differently, it would thus be necessary, during the learning phase, to consider more than just the error rate. It is perfectly possible to jointly optimize the error rate along with a metric of gender equality for instance. It just has to be specified by the designers of the algorithm. It is true that it will come at some cost to the error rate, but overall the decisions made by the machine will match more closely what is expected from users.

This all demands a technical approach which is not only more complex (and costly), but also highly structuring in terms of strategy. In our example, is the bank ready to take a little extra credit risk for the sake of fairness? In the Artificial intelligence era, one can but dream (and not just of electronic sheep)!

Here we clearly see that adopting AI actually forces the company to formalize strategic choices. This involves both economic and ethical considerations: What is our business plan? Is performance our only horizon? What is our stance on corporate social responsibility? In an unintended way, artificial intelligence can play a revealing role for the company. It is therefore especially important for data scientists to explain the issues upstream of the AI project.

The other safeguard essentially depends on the user’s position. In an ultra-complex economic environment, where decision-makers are bombarded with millions of contradictory data points, the data-driven business model is more attractive than ever, and it is tempting to give the helm to an infallible machine which will surely make the right decisions in our stead. This is a longstanding myth.

These proposals, and others, can be seen in IBM’s Precision Regulation for Artificial Intelligence, published on 21st January 2020. It sets our five proposals for eliminating bias outcomes from AI decision making, such as the creation of a lead AI ethics official, accountable for internal guidance and compliance mechanisms, such as an AI Ethics Board. Also proposed are risk assessments based on application, end user and level of automation, and great transparency around the purpose of AI systems and audit trails surrounding their input and training data. It’s fifth proposal is for testing throughout the entire life-cycle of the AI system: pre-sale, deployment, and after it is operationalized.

Humans must be the safeguard for AI decision making

In fact, it has never been more crucial to develop a reasonable approach centered on human instinct based on experience. The purpose of AI is to provide optimal recommendations according to pre-established criteria. But when the environment suddenly changes, the system’s deductions become aberrant or discriminatory, especially when they are based on biased data to begin with. Human analysis, and a sense of ethics and professionalism are the only effective safeguards.

Just as with society at large, our businesses are at a crossroads with regard to artificial intelligence. They hold all the keys to ensure that AI keeps its great promise: augment human intelligence without encouraging its idiosyncrasies.

Former chess champion Garry Kasparov made an interesting remark on this subject: “We can’t trivially design machines that are more ethical than we are the way a programmer can create a chess program that is far better at chess than they are. One key is to use them to reveal our human biases so we can improve ourselves and our society in a positive cycle.”

A naive vision? It’s up to us to decide.