In this live interview, I talked with Ticker TV about how organisations are using data to understand changes driven by responses to COVID-19.
Category: Machine Learning
ML Interpretability with Ambient Visualisations
I produced some ambient visualisations as background to short talks on the topic of Interpreting the Opaque Box of ML from ThoughtWorks Technology Radar Volume 21. The talks were presented in breaks at the YOW Developer Conference.
Here are my speaker notes.
The theme I’m talking about is Interpreting the Opaque Box of ML.
It’s a theme because the radar has a lot of ML blips – those are the individual tools, techniques, languages and frameworks we track, and they all have an aspect of interpretability.
I’m going to talk first about Explainability as a First Class Model Concern.
Explainability as a First Class Model Concern
ML models make predictions. They take some inputs and predict an output, based on the data they’ve been trained on. Without careful thought, those predictions can be opaque boxes
For example – predicting whether someone should be offered credit. A few people at the booth have mentioned this experience – “[the] black box algorithm thinks I deserve 20x the credit limit [my wife] does” – and the difficulty in getting an explanation from the provider [this was a relevant example at the time].
Elevated to a first class concern, however, ML predictions are interpretable and explainable to different degrees – it’s not actually a question of opaque box or transparent box, but many shades of translucency.
Interpretable means people can reason about a model’s decision-making process in general terms while, explainable means people can understand the factors that led to a specific decision. People are important in this definition – a data scientist may be satisfied with the explanation that the model minimises total loss, while a declined credit applicant probably requires and deserves a reason code.
And those two extremes can anchor our spectrum – at one end we can explain a result as a general consequence of ML, at the other end explaining the specific factors that contributed to an individual decision.
Dimensions – What
As dimensions of explainability , we should consider:
- The choice of modelling technique as intrinsically explainable
- Model agnostic explainability techniques
- Whether global or just local interpretability is required
Considering model selection – a decision tree is intrinsically explainable – factors contribute sequentially to a decision. A generic deep neural network is not. However, in between, we can architect networks to use techniques such as embeddings, latent spaces or transfer learning, which create representations of inputs that are distinct and interpretable to a degree, but not always in human terms.
And so model specific explainability relies on the modelling technique, while model agnostic techniques are instead empirically applicable to any model. We can create surrogate explainable models for any given model, such as a wide network paired with a deep network, and we can use ablation to explore the effect of changing inputs on a model’s decisions.
For a given decision, we might only wish to understand how that decision would have been different had the inputs changed slightly. In this case we are only concerned about local interpretability and explainability, but not the model as a whole, and LIME is an effective technique.
Reasons – Why
As broader business concerns, we should care about explainability because:
- Knowledge management is crucial for organisations – an interpretable model, such as the Glasgow Coma Scale, may be valued more for people’s ability to use it than its pure predictive performance
- We must be compliant to local laws, and it is in all stakeholders’s interests that we act ethically
- And finally, models can always make mistakes, so a challenge process must be considered, especially as vulnerable people are disproportionately subject to automated decision making
And explainability is closely linked to ethics, and hence the rise of ethical bias testing.
Ethical Bias Testing
Powerful, but Concerning
There is rising concern that powerful ML models could cause unintentional harm. For example, a model could be trained to make profitable credit decisions by simply excluding disadvantaged applicants. So we’re seeing a growing interest in ethical bias testing that will help to uncover potentially harmful decisions, and we expect this field to evolve over time.
There are many statistical measures we can use to detect unfairness in models. These measures compare outcomes for privileged and unprivileged groups under the model. If we find a model is discriminating against an unprivileged group, we can apply various mitigations to reduce the inequality.
- Equal Opportunity Difference is the difference in true positive rates between an unprivileged group and a privileged group. A value close to zero is good.
- The Disparate Impact is the ratio of the selection rate between the two groups. The selection rate is the number of individuals selected for the positive outcome divided by the total number of individuals in the group. The ideal value for this metric is 1.
These are just two examples of more than 70 different metrics for measuring ethical bias. Choosing what measure or measures to use is an ethical decision itself, and is affected by your goals. For example, there is the choice between optimising for similarity of outcomes across groups or trying to optimise so that similar individuals are treated the same. If individuals from different groups differ in their non-protected attributes, these could be competing goals.
To correct for ethical bias or unfairness, mitigations can be applied to the data, to the process of generating the model, and to the output of the model.
- Data can be reweighted to increase fairness, before running the model.
- While the model is being generated, it can be penalised for ethical bias or unfairness.
- Or, after the model is generated, it’s output can be post-processed to remove bias.
As for explainability, the process of removing ethical bias or improving fairness will likely reduce the predictive performance or accuracy of a model, however, we can see that there is a continuum of tradeoffs possible.
What is What if
I mentioned tooling is being developed to help with explainability and ethical bias testing, and you should familiarise yourself with these tools and the techniques they use. One example is the What if Tool – an interactive visual interface designed to help you dig into a model’s behaviour. It helps data scientists understand more about the predictions their model is making and was launched by the Google PAIR lab.
You can do things like:
- Compare models to each other
- Visualize feature importance
- Arrange datapoints by similarity
- Test algorithmic fairness constraints
But by themselves tools like this won’t give you explainability or fairness, and using them naively won’t remove the risk or minimize the damage done by a misapplied or poorly trained algorithm. They should be used by people who understand the theory and implications of the results. However, they can be powerful tools to help communicate, tell a story, make the specialised analysis more accessible, and hence motivate improved practice and outcomes.
The radar also mentions for the second time CD4ML – using Continuous Delivery practices for delivering ML solutions. CD in general encourages solutions to evolve in small steps, and the same is true for ML solutions. The benefit of this is that we can more accurately identify the reasons for any change in system behaviour if they are the result of small changes in design or data. So we also highlight CD4ML as a technique for addressing explainability and ethical bias
Cost Sensitive Learning – A Hitchhikers Guide
Typically prediction is about getting the right answer. But many prediction problems have large and asymmetric costs for different types of mistakes. And often, the chance of making mistakes is exacerbated by training data imbalances. Cost-Sensitive Learning is the range of techniques for extending standard ML approaches to deal with imbalanced data and outcomes. Cost-sensitive predictions will instead favour the most valuable or lowest risk answers.
I presented Cost Sensitive Learning – A Hitchhikers Guide at the Melbourne ML/AI Meetup.
Sky News Technology Behind Business
I appeared on the Sky News “Technology Behind Business” program to discuss business applications and adoption of AI and, most interestingly, whether we should fear a malicious super-intelligent AI. I said: rather, fear data leaks and the algorithmic corruption of public life.
Step Up on AI
I provided commentary on our need to step on AI capability and governance in Australia on this story in The Australian newspaper.
I was quoted extensively in the article but I wrote a bunch more notes which might be of interest.
- In the regrettable case of the Knightscope security robot and the curious case of Facebook bots, we should consider governance of product development as well as unexpected behaviours of AI systems. Governance in product development means introducing beneficial innovations in a safe manner. In self-driving vehicles, we could contrast the measured approach of Waymo (Google) – incorporating test tracks, an extensive simulation program and human supervision – to the cavalier approach of Uber – flouting state regulations to rush vehicles onto public streets. The Knightscope case may be a failure to discover, design against, and test the product’s potential failure modes in a safe environment, rather than an inherent failing of AI. The Facebook case demonstrates the value of being able to actively discontinue product research without any wider adverse effects. This is not to deny sometimes unexpected behaviours of AI systems, or deny the risk posed by poorly governed product development, but rather to focus the conversation on how to safely harness the benefits of AI products.
- Ethical and regulatory frameworks are valuable and necessary, as AI is among our most powerful technologies. There are a number of valid concerns based on bad or worst case scenarios for weaponisation, mass displacement of workers, systemic data-driven discrimination, the erosion of democratic society, hostile self-improving systems, etc. With agreed ethics and global frameworks for AI research and development, customer and citizen data regulations, and active governance of wider societal change, we can benefit from AI without exposing ourselves to worst case scenarios. Given that ethical and regulatory frameworks and broader policy changes will take some effort to establish and incentives will remain for actors to circumvent these frameworks, I also think that education and the private sector have major roles to play to improve understanding and bridge gaps short term.
- Education is key to an informed discussion about the benefits and risks of AI. The challenge is for institutions is to keep up with the state of the art. We need to create forums for public and private researches and product developers to engage with policy makers and other public institutions. And then we need evidence-based policy formulation. Australian primary schools already teach cybersecurity in early years; let’s bring AI into the curriculum too. Education at all levels should go beyond a fundamental understanding of AI to developing the skills needed to contribute to, thrive in, and continue to shape a workplace and society where many routine cognitive tasks are automated.
- The private sector is already providing some responses to ethical challenges in the absence of regulation. For instance, Volvo has stated it will “accept full liability whenever one if its cars is in autonomous mode”. Technology companies – aware their social license is being eroded by issues central to AI such as “filter bubbles”, programmatic advertising alongside objectionable content, and mass data collection – are introducing features designed to benefit users and citizens. Examples include BuzzFeed’s Outside Your Bubble, Google’s recent YouTube ad restrictions, and Apple’s CoreML support for AI on-device to maintain data privacy. We should also be encouraging the Australian private sector to take a leadership role and developing the technology and governance expertise to enable this.w
- Australia could be a leader in this field. We do have many of the right ingredients. However, the current reality is that the EU is leading regulatory change, with 2018’s General Data Protection Regulation set to extend existing data provisions to effectively create a “right to explanation” for users about whom algorithmic decisions are made. GDPR is already affecting Australian organisations with European operations. GDPR could require a complete overhaul of common AI approaches and is driving research into making AI systems more understandable. For Australia to take the lead in setting the global AI agenda would further require a different type of domestic politics to what we have seen over the last decade in respect of pressing global and technology issues, such as climate change and energy.
Reasoning About Machine Intuition
This talk discusses the resurgence of Machine Learning and neural networks from multiple perspectives of digital delivery, including: product & design, iterative implementation, organisational design, governance and risk. I chose to use “Intuition” to distinguish ML’s capability for pattern recognition from other descriptions of intelligence. Slides here.