Leave Product Development to the Dummies

This is the talk I gave at Agile Australia 2013 about the role of simulation in product development. Check out a PDF of the slides with brief notes.


"Dummies" talk at Agile Australia

Stop testing on humans! Auto manufacturers have greatly reduced the harm once caused by inadvertently crash-testing production cars with real people. Now, simulation ensures every new car endures thousands of virtual crashes before even a dummy sets foot inside. Can we do the same for software product delivery?

Simulation can deliver faster feedback than real-world trials, for less cost. Simulation supports agility, improves quality and shortens development cycles. Designers and manufacturers of physical products found this out a long time ago. By contrast, in Agile software development, we aim to ship small increments of real software to real people and use their feedback to guide product development. But what if that’s not possible? (And can we still benefit from simulation even when it is?)

The goal of trials remains the same: get a good product to market as quickly as possible (or pivot or kill a bad product as quickly as possible). However, if you have to wait for access to human subjects or real software, or if it’s too costly to scale to the breadth and depth of real-world trials required to optimise design and minimise risk, consider simulation.

Learn why simulation was chosen for the design of call centre services (and compare this with crash testing cars), how a simulator was developed, and what benefits the approach brought. You’ll leave equipped to decide whether simulation is appropriate for your next innovation project, and with some resources to get you started.


  • How and when to use simulation to improve agility
  • The anatomy of a simulator
  • A lean, risk-based approach to developing and validating a simulator
  • Techniques for effectively visualising and communicating simulations
  • Implementing simulated designs in the real world

Visualising Net Promoter Score

Net Promoter Score™ (NPS) is a popular method for evaluating customer loyalty. It has its own promoters and detractors, but it is widely used. If you are using it, or plan to, these visual tools might help you understand better what an NPS value means.

Reverse Engineering NPS

So you have an NPS value. Can you use it to draw a picture of your customers? An NPS value purports to tell you about your customer base – how do customers who are likely to recommend your brand (promoters) stack up against those who are likely to trash it (detractors)?

Try it out for yourself. Set the NPS value (from -100 to +100) by clicking/tapping the NPS slider and see the distribution of promoters and detractors. Note you can also explore the effect of the assumed number of neutrals, something a single NPS figure doesn’t tell you.

The visualisation was produced with Processing/Processing.js. See the source code . If it’s not all visible on a small screen, try the visualisation alone.

Tufte might argue the dimensionality of the visualisation (basically 2D, though it’s really 1D wrapped around again and again) exceeds that of the data (1D proportion of promoters, neutrals and detractors). However, I thought 1,000 customers was a good number to use in this context, and wrapping was the only way to compactly show all those customers. Leave your thoughts in the comments, or adapt the source, if you’ do things differently.

Don’t Ever Use Average Satisfaction Score

Satisfaction Scores (“Sat Scores”) are the raw data collected from customers. It’s tempting to reason about NPS with the average of these sat scores because it’s easier to calculate in Excel™ than NPS. If you are tempted, just remember: you may look like an ASS if you use Average Sat Score. Why is Average Sat Score flawed? See below.

NPS is not Average Satisfaction Score
NPS is not Average Sat Score

Average Sat Score doesn’t care about the distribution of responses, whereas NPS is all about the distribution. The same Average Sat Score of 7 applies to customer response sets with NPS values of -50, +40, -100, or “undefined”. NPS is designed to magnify outliers (promoters and detractors) rather than collapse them towards the centre, which is what the Average Sat Score does. Whether NPS works or not is another question entirely.

The Maths of NPS

The visuals above are based on the standard definition of NPS. We have N customer responses Ri (for i = 1 to N) to a feedback question, such as “Rate on a scale from 0 to 10 how likely you are to recommend us”. The number of promoters P is the number of responses Ri > 8. The number of detractors D is the number of responses Ri < 7. The remaining responses are “neutral”.

\[NPS = \frac{N-D}{N+D}\times{100}\]