ML Interpretability with Ambient Visualisations

I produced some ambient visualisations as background to short talks on the topic of Interpreting the Black Box of ML from ThoughtWorks Technology Radar Volume 21. The talks were presented in breaks at the YOW Developer Conference.

Animation of linear to non-linear model selection

Here are my speaker notes.

Theme Intro

The theme I’m talking about is Interpreting the Black Box of ML.

It’s a theme because the radar has a lot of ML blips – those are the individual tools, techniques, languages and frameworks we track, and they all have an aspect of interpretability.

I’m going to talk first about Explainability as a First Class Model Concern.

Explainability as a First Class Model Concern

ML models make predictions. They take some inputs and predict an output, based on the data they’ve been trained on. Without careful thought, those predictions can be black boxes

For example – predicting whether someone should be offered credit. A few people at the booth have mentioned this experience[the] black box algorithm thinks I deserve 20x the credit limit [my wife] does – and the difficulty in getting an explanation from the provider [this was a relevant example at the time].

Elevated to a first class concern, however, ML predictions are interpretable and explainable to different degrees – it’s not actually a question of black box or white box, but many shades of grey.

Spectrum

Interpretable means people can reason about a model’s decision-making process in general terms while, explainable means people can understand the factors that led to a specific decision. People are important in this definition – a data scientist may be satisfied with the explanation that the model minimises total loss, while a declined credit applicant probably requires and deserves a reason code. 

And those two extremes can anchor our spectrum – at one end we can explain a result as a general consequence of ML, at the other end explaining the specific factors that contributed to an individual decision.

Dimensions – What

As dimensions of explainability , we should consider:

  • The choice of modelling technique as intrinsically explainable
  • Model agnostic explainability techniques
  • Whether global or just local interpretability is required

Considering model selection – a decision tree is intrinsically explainable – factors contribute sequentially to a decision. A generic deep neural network is not. However, in between, we can architect networks to use techniques such as embeddings, latent spaces or transfer learning, which create representations of inputs that are distinct and interpretable to a degree, but not always in human terms.

And so model specific explainability relies on the modelling technique, while model agnostic techniques are instead empirically applicable to any model. We can create surrogate explainable models for any given model, such as a wide network paired with a deep network, and we can use ablation to explore the effect of changing inputs on a model’s decisions.

For a given decision, we might only wish to understand how that decision would have been different had the inputs changed slightly. In this case we are only concerned about local interpretability and explainability, but not the model as a whole, and LIME is an effective technique.

Reasons – Why

As broader business concerns, we should care about explainability because:

  • Knowledge management is crucial for organisations – an interpretable model, such as the Glasgow Coma Scale, may be valued more for people’s ability to use it than its pure predictive performance
  • We must be compliant to local laws, and it is in all stakeholders’s interests that we act ethically
  • And finally, models can always make mistakes, so a challenge process must be considered, especially as vulnerable people are disproportionately subject to automated decision making

And explainability is closely linked to ethics, and hence the rise of ethical bias testing.

Ethical Bias Testing

Powerful, but Concerning

There is rising concern that powerful ML models could cause unintentional harm. For example, a model could be trained to make profitable credit decisions by simply excluding disadvantaged applicants. So we’re seeing a growing interest in ethical bias testing that will help to uncover potentially harmful decisions, and we expect this field to evolve over time.

Measures

There are many statistical measures we can use to detect unfairness in models. These measures compare outcomes for privileged and unprivileged groups under the model. If we find a model is discriminating against an unprivileged group, we can apply various mitigations to reduce the inequality.  

  • Equal Opportunity Difference is the difference in true positive rates between an unprivileged group and a privileged group. A value close to zero is good.
  • The Disparate Impact is the ratio of the selection rate between the two groups.  The selection rate is the number of individuals selected for the positive outcome divided by the total number of individuals in the group. The ideal value for this metric is 1.

These are just two examples of more than 70 different metrics for measuring ethical bias. Choosing what measure or measures to use is an ethical decision itself, and is affected by your goals. For example, there is the choice between optimising for similarity of outcomes across groups or trying to optimise so that similar individuals are treated the same. If individuals from different groups differ in their non-protected attributes, these could be competing goals.

Correction

To correct for ethical bias or unfairness, mitigations can be applied to the data, to the process of generating the model, and to the output of the model.

  • Data can be reweighted to increase fairness, before running the model.
  • While the model is being generated, it can be penalised for ethical bias or unfairness.
  • Or, after the model is generated, it’s output can be post-processed to remove bias. 

As for explainability, the process of removing ethical bias or improving fairness will likely reduce the predictive performance or accuracy of a model, however, we can see that there is a continuum of tradeoffs possible.

What-if Tool

What is What if

I mentioned tooling is being developed to help with explainability and ethical bias testing, and you should familiarise yourself with these tools and the techniques they use. One example is the What if Tool – an interactive visual interface designed to help you dig into a model’s behaviour. It helps data scientists understand more about the predictions their model is making and was launched by the Google PAIR lab.

Features

You can do things like:

  •  Compare models to each other
  •  Visualize feature importance
  •  Arrange datapoints by similarity
  •  Test algorithmic fairness constraints

Risk

But by themselves tools like this won’t give you explainability or fairness, and using them naively won’t remove the risk or minimize the damage done by a misapplied or poorly trained algorithm. They should be used by people who understand the theory and implications of the results. However, they can be powerful tools to help communicate, tell a story, make the specialised analysis more accessible, and hence motivate improved practice and outcomes.

CD4ML

The radar also mentions for the second time CD4ML – using Continuous Delivery practices for delivering ML solutions. CD in general encourages solutions to evolve in small steps, and the same is true for ML solutions. The benefit of this is that we can more accurately identify the reasons for any change in system behaviour if they are the result of small changes in design or data. So we also highlight CD4ML as a technique for addressing explainability and ethical bias

Cost Sensitive Learning – A Hitchhikers Guide

Typically prediction is about getting the right answer. But many prediction problems have large and asymmetric costs for different types of mistakes. And often, the chance of making mistakes is exacerbated by training data imbalances. Cost-Sensitive Learning is the range of techniques for extending standard ML approaches to deal with imbalanced data and outcomes. Cost-sensitive predictions will instead favour the most valuable or lowest risk answers.

I presented Cost Sensitive Learning – A Hitchhikers Guide at the Melbourne ML/AI Meetup.

Step Up on AI

I provided commentary on our need to step on AI capability and governance in Australia on this story in The Australian newspaper.

I was quoted extensively in the article but I wrote a bunch more notes which might be of interest.

Further Commentary

  • In the regrettable case of the Knightscope security robot and the curious case of Facebook bots, we should consider governance of product development as well as unexpected behaviours of AI systems. Governance in product development means introducing beneficial innovations in a safe manner. In self-driving vehicles, we could contrast the measured approach of Waymo (Google) – incorporating test tracks, an extensive simulation program and human supervision – to the cavalier approach of Uber – flouting state regulations to rush vehicles onto public streets. The Knightscope case may be a failure to discover, design against, and test the product’s potential failure modes in a safe environment, rather than an inherent failing of AI. The Facebook case demonstrates the value of being able to actively discontinue product research without any wider adverse effects. This is not to deny sometimes unexpected behaviours of AI systems, or deny the risk posed by poorly governed product development, but rather to focus the conversation on how to safely harness the benefits of AI products.
  • Ethical and regulatory frameworks are valuable and necessary, as AI is among our most powerful technologies. There are a number of valid concerns based on bad or worst case scenarios for weaponisation, mass displacement of workers, systemic data-driven discrimination, the erosion of democratic society, hostile self-improving systems, etc. With agreed ethics and global frameworks for AI research and development, customer and citizen data regulations, and active governance of wider societal change, we can benefit from AI without exposing ourselves to worst case scenarios. Given that ethical and regulatory frameworks and broader policy changes will take some effort to establish and incentives will remain for actors to circumvent these frameworks, I also think that education and the private sector have major roles to play to improve understanding and bridge gaps short term.
  • Education is key to an informed discussion about the benefits and risks of AI. The challenge is for institutions is to keep up with the state of the art. We need to create forums for public and private researches and product developers to engage with policy makers and other public institutions. And then we need evidence-based policy formulation. Australian primary schools already teach cybersecurity in early years; let’s bring AI into the curriculum too. Education at all levels should go beyond a fundamental understanding of AI to developing the skills needed to contribute to, thrive in, and continue to shape a workplace and society where many routine cognitive tasks are automated.
  • The private sector is already providing some responses to ethical challenges in the absence of regulation. For instance, Volvo has stated it will “accept full liability whenever one if its cars is in autonomous mode”. Technology companies – aware their social license is being eroded by issues central to AI such as “filter bubbles”, programmatic advertising alongside objectionable content, and mass data collection – are introducing features designed to benefit users and citizens. Examples include BuzzFeed’s Outside Your Bubble, Google’s recent YouTube ad restrictions, and Apple’s CoreML support for AI on-device to maintain data privacy. We should also be encouraging the Australian private sector to take a leadership role and developing the technology and governance expertise to enable this.w
  • Australia could be a leader in this field. We do have many of the right ingredients. However, the current reality is that the EU is leading regulatory change, with 2018’s General Data Protection Regulation set to extend existing data provisions to effectively create a “right to explanation” for users about whom algorithmic decisions are made. GDPR is already affecting Australian organisations with European operations. GDPR could require a complete overhaul of common AI approaches and is driving research into making AI systems more understandable. For Australia to take the lead in setting the global AI agenda would further require a different type of domestic politics to what we have seen over the last decade in respect of pressing global and technology issues, such as climate change and energy.

General Background

  • Machines (AI) are outperforming humans in a range of narrow, intuitive cognitive tasks. Some major themes have emerged: supporting human interactions, and playing games. For instance, one study showed machines are better lip-reading than humans. They are also better players of the complex strategy game of Go, better at bluffing in Poker, and better at the most advanced computer games. However, with ongoing digitisation of the physical world, business and society, we will see more and more examples of narrow human intuition, or “know-how”, bettered by machines. 
  • The primary technology driving the leap is Deep Learning, also known as Artificial Neural Networks. Deep Learning is delivering substantial performance improvements in fields such as language processing and translation, medical diagnosis, and self-driving vehicles (in addition to human interactions and games above). Rather than attempting to explicitly define and program algorithms and statistical matching, Deep Learning leverages large data sets to train Neural Networks to recognise patterns and learn their own internal representation of key concepts.
  • While the core ideas behind Deep Learning are not new, three major technology developments have boosted the capabilities of networks in the order of 1000 times in the last five years: massive web-scale data sets, development of dedicated parallel hardware, and innovation in network designs. These technology advancements have created a positive feedback loop of research and development funding, so innovation continues at a rapid pace.
  • This new technology capability creates enormous opportunity, but also disruption and risk. Inhuman feats of decision-making are possible, and might help solve pressing global problems in the environment, health and society. Deep Learning systems can be wider thinkers (across far bigger data sets) and more scalable thinkers (in speed or other dimensions) than humans, and they also notice patterns that humans miss or cannot perceive. However, humans remain faster learners and more flexible thinkers. And as machines best humans they also better humans – the machine Go champion AlphaGo is now training a new generation of human Go champions. The key to opportunity is not to simply outsource human “know-how” to machines, but rather to enhance human know-how with machine intuition in order to drive new breakthroughs in understanding cause-and-effect in complex systems. In this respect, Human Centred Design is a key complement to machine intuition.
  • The automation of a wide range of narrow cognitive tasks will undoubtedly be disruptive, but through education and pro-active governance, we can make this a positive disruption. As people are displaced from jobs or their jobs change substantially, we must create new societal and commercial constructs that confer purpose, enable people to contribute to, thrive in, and shape a world enabled by AI, and provide them resources for participation in an equitable society. The true nature and extent of the disruption remains to be seen, as prior automation of routine manual tasks has also enabled growth in non-routine, creative and community-oriented manual tasks. This disruption is a huge opportunity to reimagine work and society for the better. 
  • Risk exists in the fact that the operation of AI products may not be adequately explainable, exacerbating the reality that they have new intrinsic failure modes, can be attacked and exploited externally, and can ultimately be developed with malicious intent. Key failure modes include biased decision making-based on biased training sets, and the creation of self-fulfilling prediction and action feedback loops. These failure modes further often disproportionately affect the most vulnerable members of society, as they are not adequately captured in training data or noticed in feedback loops, and may in the worst case curtail basic democratic freedoms. However, systems based on human decision making are similarly fallible, and further suffer from execution variability, such as the estimated 90% of road incidents caused by human error, so we need to take a holistic approach to societal benefit and risk in these circumstances. These issues also intersect with broader concerns about data collection and privacy. Given that Deep Learning systems match patterns, it is possible to engineer false positives through study of the system, and in fact secondary Deep Learning systems may be applied to this task! We may well see a new cyber AI arms race to protect and exploit AI systems. However, this may also spawn much needed explanations of how these systems work. Tackling development with malicious intent requires global coordination on ethical and regulatory frameworks, and also addressing the root causes of intent. Ethical and regulatory frameworks should as a precautionary measure consider the broader existential risk posed by the possibility of a hostile self-improving artificial general or super intelligence. However, the eventual development of such an intelligence is not a foregone conclusion.
  • Ultimately, understanding the current wave of AI development better will enable individuals and private and public organisations to better capture opportunity while governing risk. We should invest in research and education and develop new mechanisms and institutions to shape the development of this technology for the best possible ends. The payoff will be beyond functional, as understanding these new capabilities of machines will lead to a deeper understanding of what it means to be human.

Reasoning About Machine Intuition

This talk discusses the resurgence of Machine Learning and neural networks from multiple perspectives of digital delivery, including: product & design, iterative implementation, organisational design, governance and risk. I chose to use “Intuition” to distinguish ML’s capability for pattern recognition from other descriptions of intelligence. Slides here.

Scaling Change

Once upon a time, scaling production may have been enough to be competitive. Now, the most competitive organisations scale change to continually improve customer experience. How can we use what we’ve learned scaling production to scale change?

Metaphors for scaling
Metaphors for scaling

I recently presented a talk titled “Scaling Change”. In the talk I explore the connections between scaling production, sustaining software development, and scaling change, using metaphors, maths and management heuristics. The same model of change applies from organisational, marketing, design and technology perspectives.  How can factories, home loans and nightclubs help us to think about and manage change at scale?

Read on with the spoiler post if you’d rather get right to the heart of the talk.

Scaling Change Spoiler

When software engineers think about scaling, they think in terms of the order of complexity, or “Big-O“, of a process or system. Whereas production is O(N) and can be scaled by shifting variable costs to fixed, I contend that change is O(N2) due to the interaction of each new change with all previous changes. We could visualise this as a triangular matrix heat map of the interaction cost of each pair of changes (where darker shading is higher cost).

Change heatmap
Change interaction heatmap

Therefore, a nightclub, where each new patron potentially interacts with all other denizens is an appropriate metaphor. Many of us can also relate to changes that have socialised about as well as drunk nightclub patrons.

Socialisation failures at a nightclub [BBC News]
Socialisation failures [BBC News]
The thing about change being O(N2) is that the old production management heuristics of shifting variable cost to fixed no longer work, because the dominant mode is interaction cost. The nightclub metaphor suggests the following management heuristics:

Socialise

Socialising change
Socialising change

We take a variable cost hit for each change to help it play more nicely with every other change. This reduces the cost coefficient but not the number of interactions (N2).

Screen

Screening change
Screening change

We only take in the most valuable changes. Screening half our changes (N/2) reduces change interactions by three quarters (N2/4).

Seclude

Secluding change
Secluding change

We arrange changes into separate spaces and prevent interaction between spaces. Using n spaces reduces the interactions to N2/n.

Surrender

Surrendering change
Surrendering change

Like screening, but at the other end. We actively manage out changes to reduce interactions. Surrendering half our changes (N/2) reduces change interactions by three quarters (N2/4).

Scenarios

Where do we see these approaches being used? Just some examples:

  • Start-ups screen or surrender changes and hence are more agile than incumbents because they have less history of change.
  • Product managers screen changes in design and seclude changes across a portfolio, for example the separate apps of Facebook/ Messenger/ Instagram/ Hyperlapse/ Layout/ Boomerang/ etc
  • To manage technical debt, good developers socialise via refactoring, better seclude through architecture, and the best surrender
  • In hiring, candidates are screened and socialised through rigorous recruitment and training processes
  • Brand architectures also seclude changes – Unilever’s Dove can campaign for real beauty while Axe/Lynx offends Dove’s targets (and many others).

See Also

The life-changing magic of tidying your work

Surprise! Managing work in a large organisation is a lot like keeping your belongings in check at home.

Get it wrong at home and you have mess and clutter. Get it wrong in the organisation and you have excessive work in progress (WIP), retarding responsiveness, pulverising productivity, and eroding engagement.

Reading Marie Kondo’s The Life-Changing Magic of Tidying Up (Amazon), I was struck by a number of observations about tidying personal belongings that resonated with how individuals, teams and organisations manage their work.

First, reading TLCMOTU helped me tidy my things better. Second, it reinforced lean and agile management principles.

I won’t review the book here. Maybe the methods and ideas resonate with you, maybe they don’t. However, because I think tidying is something that everyone can relate to, I will compare some of KonMari’s (as Marie Kondo is known) explanations of the management of personal belongings with the management of work in organisations. The translation heuristic is to replace stuff with work, and clutter with excessive WIP, to highlight the parallels.

I’d love to know if you find the comparison useful.

On the complexity of work storage systems

KonMari writes:

Most people realise that clutter is caused by too much stuff. But why do we have too much stuff? Usually it is because we do not accurately grasp how much we actually own. And we fail to grasp how much we own because our storage methods are too complex.

Organisations typically employ complex storage methods for their work: portfolio and project management systems with myriad arcane properties, intricate plans, baselines and revisions, budget and planning cycle constraints, capitalisation constraints, fractional resource allocations, and restricted access to specialists who are removed from the outcomes but embrace the management complexity.

And this is just the work that’s stored where it should be. Then there’s all the work that’s squirrelled away into nooks and crannies that has to be teased out by thorough investigation (see below).

Because organisations don’t comprehend the extent of their work, they invent ever-more complex systems to stuff work into storage maximise utilisation of capacity, which continues to hide the extent of the work.

Thus, we fail to grasp how much work is held in the organisation, and the result is excessive WIP, which inflates lead times and reduces productivity, failing customers and leaving workers disengaged. Simplifying the storage of work – as simple as cards on a wall, with the information we actually need to deliver outcomes – allows us to comprehend the work we hold, and allows us to better manage WIP for responsiveness and productivity.

On making things visible

KonMari observes that you cannot accurately assess how much stuff you have without seeing it all in one place. She recommends searching the whole house first, bringing everything to the one location, and spreading the items out on the floor to gain visibility.

Making work visible, in one place, to all stakeholders is a tenet of agile and lean delivery. It reveals amazing insights, many unanticipated, about the volume, variety and value (or lack of) of work in progress. The shared view helps build empathy and collaboration between stakeholders and delivery teams. You may need to search extensively within the organisation to discover all the work, but understanding of the sources of demand (as below) will guide you. A great resource for ideas and examples of approaches is Agile Board Hacks.

So get your work on cards on a wall so you can see the extent of your WIP.

On categories

KonMari observes that items in one category are stored in multiple different places, spread out around the house. Categories she identifies include clothes, books, etc. She contends that it’s not possible to assess what you want to keep and discard without seeing the sum of your belongings in each category. Consequently, she recommends thinking in terms of category, rather than place.

If we think organisationally in terms of place, we think of silos – projects, teams, functions. We can’t use these storage units to properly assess the work we hold in the organisation. Internal silos don’t reflect how we serve customers.

Instead, if we think organisationally in terms of category, we are thinking strategically. With a cascading decomposition of strategy, driven by the customer, we can assess the work in the organisation at every level for strategic alignment (strategy being emergent as well as explicit). Strategy could be enterprise level themes, or the desired customer journey at a product team level.

With work mapped against strategy, we can see in one place the sum of efforts to execute a given branch of strategy, and hence assess what to keep and what to discard. We further can assess whether the entire portfolio of work is sufficiently aligned and diversified to execute strategy.

So use your card wall to identify how work strategically serves your customers.

On joy

KonMari writes:

The best way to choose what to keep and what to throw away is to … ask: ‘Does this spark joy?’ If it does keep it. If not, throw it out.

We may ask of each piece of work: ‘Is this work valuable?’ ‘Is it aligned to the purpose of the organisation?’ ‘Is it something customers want?’ If it is, keep it. If not, throw it out.

KonMari demonstrates why this is effective by taking the process to its logical conclusion. If you’ve discarded everything that doesn’t spark joy, then everything you have, everything you interact with, does spark joy.

What better way to spark joy in your people than to reduce or eliminate work with no value and no purpose?

On discarding first

KonMari observes that storage considerations interrupt the process of discarding. She recommends that discarding comes first, and storage comes second, and the activities remain distinct. If you start to think about where to put something before you have decided whether to keep or discard it, you will stop discarding.

Prioritisation is the act of discarding work we do not intend to pursue. Prioritisation comes first, based purely on value, before implementation considerations. Sequencing can be done with knowledge of effort and other dependencies. Then scheduling, given capacity and other constraints, is the process of deciding which “drawers” to put work in.

On putting things away

KonMari observes that mess and clutter is a result of not putting things away. Consequently she recommends that storage systems should make it easy to put things away, not easy to get them out.

Excessive WIP may also be caused by a failure to rapidly stop work (or perceived inability to do so). Organisational approaches to work should reduce the effort needed to stop work. For instance, with continuous delivery, a product is releasable at all times, and can therefore be stopped after any deployment. Work should be easily stoppable in preference to easily startable. (This could also be framed as “stop starting and start finishing”.)

Further, while many organisations aim for responsiveness with a stoppable workforce (of contractors), they should instead aim for a stoppable portfolio, and workforce responsiveness will follow.

On letting things go

A client of KonMari’s comments:

Up to now, I believed it was important to do things that added to my life …  I realised for the first time that letting go is even more important than adding.

I have written about the importance of letting go of work from the perspective of via negativa management in Dumbbell Delivery; Antifragile Software, and managing socialisation costs in Your Software is a Nightclub.

However, KonMari also observes that, beyond the mechanics of managing stuff (or work), there is a psychological cost of clutter (or excessive WIP). Her clients often report feeling constrained by perceived responsibility to stuff that brings them no joy. I suspect the same is true in the organisation: we fail to recognise and embrace possibilities because we are constrained by perceived responsibilities to work that ultimately has no value.

Imagine if we could throw off those shackles. That’s worth letting a few things go.