What is explainable AI?
The artificial intelligence and machine learning boom has led to concerns about ‘black box’ models and systemic bias. A solution to this is to build explainable AI. Technical Manager David Lyford-Smith explores what that means and how it can be done.
Machine learning (ML) is all about creating systems to solve complex problems, using data to provide the algorithm with ‘experience’. While there are many different models of machine learning, in essence they are about constantly varying the design and weighting of the factors an algorithm considers in order to maximise performance at whatever task the creator desires. Unlike traditional programs that are created through explicit if-this-then-that coding, the results of ML don’t have to resemble any kind of logical flow or readable code. ML is all about making solutions for problems where we don’t know how to write direct code for the solution, or where we don’t know how to solve it; so naturally the outcomes aren’t usually understandable as they are.But this leads to serious issues, as we’ve written about on multiple occasions. While the process of building an algorithm with ML demonstrates that the resulting product is the best candidate for handling the data it was trained with, it can be much harder to know how accurate the system is when presented with live data. The beginning of the pandemic showed that many algorithms that were well-adjusted to ‘business as usual’ struggled when there was a sea change in the data they were analysing.
Bias and pattern
Even without a seismic change in the data, in the ordinary course of business an apparently incorrect decision, or one challenged by the customer subject to it, can be unverifiable. Algorithms trained on incomplete or biased data will pick up and replicate those biases, as many unfortunate and public cases have demonstrated. For would-be AI developers working anywhere where GDPR is a consideration, that legislation also supports the right to challenge automated decisions. And finally, even beyond all of this, understandability has commercial benefits – you can learn from the patterns and rules the algorithm has detected elsewhere in the organisation, rather than just applying its outputs to the task for which it was trained.>
So instead, people recommend the use of ‘explainable AI’ – but what does that mean, and how can we develop it? The phrase is something of an umbrella term used for a few related techniques, some of which aim to produce simpler algorithms that are more understandable in the first place, and some of which are tools for looking at full-blooded ML systems and producing a simplified but accessible explanation from it. In either case, the explanation isn’t perfect – it’s simplified to some degree – but the aim is to provide a reviewer or data subject with a meaningful and comprehensible guide to why the algorithm’s output was what it was.>
Sense and sensitivity
A common approach is to consider the sensitivity of the model to various identifiable features, giving a weight to each. So, for example, an algorithm to determine creditworthiness for a mortgage might strongly respond to changes in credit rating or current bank balance but be less affected by a customer’s location or the time it took them to complete the application. These kinds of explanations help a reviewer understand which features the algorithm has locked on to during its development, and the approach has the advantage that it can be developed without a review of the code itself – just examining its sensitivity through practical testing. It can also help identify biases – for example, if an algorithm would discriminate between otherwise identical applicants of different genders, that is a sign that the algorithm is biased. But these kinds of explanations are also limited in scope; they struggle to account for the effect of combinations of features, and aren’t useful for more open-ended algorithms such as image describers that have more outputs than a simple accept/decline.Other approaches try to make a simpler model that is understandable, while mostly having the same behaviour as the more complex full algorithm. Ironically this usually means using another ML algorithm to create the simpler model. The idea is to build a decision tree or other simplified model that can be followed by a human reviewer and that encapsulates as much of the real model’s behaviour as possible. While this is similar to the above sensitivity concept, these more complex explanations can capture more variability and depth – but obviously suffer from lower accuracy, as they are not exactly the same as the system they model.
Devil in the detail
Whatever model of explanation is being pursued, the explanation itself is not sufficient. Good explanation relies upon good communication, especially if the explanation is being presented to a member of the public. This means more than just discussing the factors that the algorithm weighs in making its decision – it also means providing basic information about how the system was developed and tested before being put in place, the kinds of errors that could occur, and the limitations of the explanation that is being presented. What level and detail of explanation is appropriate will depend on the situation – for example, research shows that people are satisfied with less explanation in a technical or medical context, where they might not get so much detail when dealing with a doctor or other professional, but that they would expect more detail on a financial or hiring decision that affected them directly. And there is a difference between an internal-facing explanation of how the model works in general and a customer-facing one that just explains the factors leading to the decision in that specific case.It’s also worth noting that building explainable AI doesn’t solve the critical issues facing AI alone – for example, while it might help detect and prevent certain biases, some more subtle ones could remain. This is particularly true where a model may not directly care about a protected characteristic, such as race or gender, but could still place some reliance on correlated variables, such as postcode or number of name changes. And just because a data subject can access an explanation of the automated process handling their case, this doesn’t mean the explanation will be satisfactory to them – or that the decision will be correct!
Many today feel as though giving up their personal data and being subject to automated decisions is something that is out of their hands, but the twin tides of public opinion and government regulation are pushing increasingly hard against that sentiment. There is a growing expectation that those putting AI into practice provide meaningful, transparent explanations. Making algorithms that are explainable is more difficult and more expensive than not doing so, but it doesn’t look like there is a real choice here – if you are to make systems that are more useful, less biased, more acceptable and in line with regulation, explainability is the future
