- Fitness functions to define “what good looks like” for data mesh and guide the evolution of analytic data architecture and operating model
- Team topologies as a system for organisational design that is sympathetic to data mesh
- Driving a delivery program through use cases
- Thin slicing and evolution of products
I collaborated with with some colleagues to share our experiences with data mesh and how to frame the benefits for an executive audience, written up in an article titled The Business Case for Data Mesh.
Here’s the recording of my presentation on data mesh at the Data Engineering Melbourne Meetup, on 26 August 2021. We covered architecture, building blocks and more. Lots of great questions and discussion.
Thanks as always to organisers Harmeet Sokhi, Timothy Findlay, and Andrew Jones!
I presented this webinar with Zhamak Dehghani – see the recording Guiding the Evolution of Data Mesh with Fitness Functions. There was great engagement with the topic and we captured some questions and further thoughts on this mini-blog post, published a little later.
Our thoughts around guiding fitness functions included the below. These high-level measurable objectives were supported by a range of proposed metrics. This table is a handy summary; check out the webinar for more.
Scaling sources and consumers
Reduced accidental complexity
|Data as a Product|
Serving users’ needs
Ease of discovery
Evaluation of quality
|Self-Serve Data Platform|
Abstraction of complexity
Domain team autonomy
Protocols enable an ecosystem
|Federated Computational Governance|
Governance for common good
Degree of decentralisation
Increasing returns from scale
I provided some commentary on data mesh for ITWire in this article titled Data mesh decentralises custodianship while maintaining governance.
Once upon a time, scaling production may have been enough to be competitive. Now, the most competitive organisations scale change to continually improve customer experience. How can we use what we’ve learned scaling production to scale change?
I recently presented a talk titled “Scaling Change”. In the talk I explore the connections between scaling production, sustaining software development, and scaling change, using metaphors, maths and management heuristics. The same model of change applies from organisational, marketing, design and technology perspectives. How can factories, home loans and nightclubs help us to think about and manage change at scale?
Read on with the spoiler post if you’d rather get right to the heart of the talk.
When software engineers think about scaling, they think in terms of the order of complexity, or “Big-O“, of a process or system. Whereas production is O(N) and can be scaled by shifting variable costs to fixed, I contend that change is O(N2) due to the interaction of each new change with all previous changes. We could visualise this as a triangular matrix heat map of the interaction cost of each pair of changes (where darker shading is higher cost).
The thing about change being O(N2) is that the old production management heuristics of shifting variable cost to fixed no longer work, because the dominant mode is interaction cost. Instead we use the following management heuristics:
We take a variable cost hit for each change to help it play more nicely with every other change. This reduces the cost coefficient but not the number of interactions (N2).
We only take in the most valuable changes. Screening half our changes (N/2) reduces change interactions by three quarters (N2/4).
We arrange changes into separate spaces and prevent interaction between spaces. Using n spaces reduces the interactions to N2/n.
Like screening, but at the other end. We actively manage out changes to reduce interactions. Surrendering half our changes (N/2) reduces change interactions by three quarters (N2/4).
Where do we see these approaches being used? Just some examples:
- Start-ups screen or surrender changes and hence are more agile than incumbents because they have less history of change.
- Product managers screen changes in design and seclude changes across a portfolio, for example the separate apps of Facebook/ Messenger/ Instagram/ Hyperlapse/ Layout/ Boomerang/ etc
- To manage technical debt, good developers socialise via refactoring, better seclude through architecture, and the best surrender
- In hiring, candidates are screened and socialised through rigorous recruitment and training processes
- Brand architectures also seclude changes – Unilever’s Dove can campaign for real beauty while Axe/Lynx offends Dove’s targets (and many others).
Surprise! Managing work in a large organisation is a lot like keeping your belongings in check at home.
Get it wrong at home and you have mess and clutter. Get it wrong in the organisation and you have excessive work in progress (WIP), retarding responsiveness, pulverising productivity, and eroding engagement.
Reading Marie Kondo’s The Life-Changing Magic of Tidying Up (Amazon), I was struck by a number of observations about tidying personal belongings that resonated with how individuals, teams and organisations manage their work.
First, reading TLCMOTU helped me tidy my things better. Second, it reinforced lean and agile management principles.
I won’t review the book here. Maybe the methods and ideas resonate with you, maybe they don’t. However, because I think tidying is something that everyone can relate to, I will compare some of KonMari’s (as Marie Kondo is known) explanations of the management of personal belongings with the management of work in organisations. The translation heuristic is to replace stuff with work, and clutter with excessive WIP, to highlight the parallels.
I’d love to know if you find the comparison useful.
On the complexity of work storage systems
Most people realise that clutter is caused by too much stuff. But why do we have too much stuff? Usually it is because we do not accurately grasp how much we actually own. And we fail to grasp how much we own because our storage methods are too complex.
Organisations typically employ complex storage methods for their work: portfolio and project management systems with myriad arcane properties, intricate plans, baselines and revisions, budget and planning cycle constraints, capitalisation constraints, fractional resource allocations, and restricted access to specialists who are removed from the outcomes but embrace the management complexity.
And this is just the work that’s stored where it should be. Then there’s all the work that’s squirrelled away into nooks and crannies that has to be teased out by thorough investigation (see below).
Because organisations don’t comprehend the extent of their work, they invent ever-more complex systems to
stuff work into storage maximise utilisation of capacity, which continues to hide the extent of the work.
Thus, we fail to grasp how much work is held in the organisation, and the result is excessive WIP, which inflates lead times and reduces productivity, failing customers and leaving workers disengaged. Simplifying the storage of work – as simple as cards on a wall, with the information we actually need to deliver outcomes – allows us to comprehend the work we hold, and allows us to better manage WIP for responsiveness and productivity.
On making things visible
KonMari observes that you cannot accurately assess how much stuff you have without seeing it all in one place. She recommends searching the whole house first, bringing everything to the one location, and spreading the items out on the floor to gain visibility.
Making work visible, in one place, to all stakeholders is a tenet of agile and lean delivery. It reveals amazing insights, many unanticipated, about the volume, variety and value (or lack of) of work in progress. The shared view helps build empathy and collaboration between stakeholders and delivery teams. You may need to search extensively within the organisation to discover all the work, but understanding of the sources of demand (as below) will guide you. A great resource for ideas and examples of approaches is Agile Board Hacks.
So get your work on cards on a wall so you can see the extent of your WIP.
KonMari observes that items in one category are stored in multiple different places, spread out around the house. Categories she identifies include clothes, books, etc. She contends that it’s not possible to assess what you want to keep and discard without seeing the sum of your belongings in each category. Consequently, she recommends thinking in terms of category, rather than place.
If we think organisationally in terms of place, we think of silos – projects, teams, functions. We can’t use these storage units to properly assess the work we hold in the organisation. Internal silos don’t reflect how we serve customers.
Instead, if we think organisationally in terms of category, we are thinking strategically. With a cascading decomposition of strategy, driven by the customer, we can assess the work in the organisation at every level for strategic alignment (strategy being emergent as well as explicit). Strategy could be enterprise level themes, or the desired customer journey at a product team level.
With work mapped against strategy, we can see in one place the sum of efforts to execute a given branch of strategy, and hence assess what to keep and what to discard. We further can assess whether the entire portfolio of work is sufficiently aligned and diversified to execute strategy.
So use your card wall to identify how work strategically serves your customers.
The best way to choose what to keep and what to throw away is to … ask: ‘Does this spark joy?’ If it does keep it. If not, throw it out.
We may ask of each piece of work: ‘Is this work valuable?’ ‘Is it aligned to the purpose of the organisation?’ ‘Is it something customers want?’ If it is, keep it. If not, throw it out.
KonMari demonstrates why this is effective by taking the process to its logical conclusion. If you’ve discarded everything that doesn’t spark joy, then everything you have, everything you interact with, does spark joy.
What better way to spark joy in your people than to reduce or eliminate work with no value and no purpose?
On discarding first
KonMari observes that storage considerations interrupt the process of discarding. She recommends that discarding comes first, and storage comes second, and the activities remain distinct. If you start to think about where to put something before you have decided whether to keep or discard it, you will stop discarding.
Prioritisation is the act of discarding work we do not intend to pursue. Prioritisation comes first, based purely on value, before implementation considerations. Sequencing can be done with knowledge of effort and other dependencies. Then scheduling, given capacity and other constraints, is the process of deciding which “drawers” to put work in.
On putting things away
KonMari observes that mess and clutter is a result of not putting things away. Consequently she recommends that storage systems should make it easy to put things away, not easy to get them out.
Excessive WIP may also be caused by a failure to rapidly stop work (or perceived inability to do so). Organisational approaches to work should reduce the effort needed to stop work. For instance, with continuous delivery, a product is releasable at all times, and can therefore be stopped after any deployment. Work should be easily stoppable in preference to easily startable. (This could also be framed as “stop starting and start finishing”.)
Further, while many organisations aim for responsiveness with a stoppable workforce (of contractors), they should instead aim for a stoppable portfolio, and workforce responsiveness will follow.
On letting things go
A client of KonMari’s comments:
Up to now, I believed it was important to do things that added to my life … I realised for the first time that letting go is even more important than adding.
I have written about the importance of letting go of work from the perspective of via negativa management in Dumbbell Delivery; Antifragile Software, and managing socialisation costs in Your Software is a Nightclub.
However, KonMari also observes that, beyond the mechanics of managing stuff (or work), there is a psychological cost of clutter (or excessive WIP). Her clients often report feeling constrained by perceived responsibility to stuff that brings them no joy. I suspect the same is true in the organisation: we fail to recognise and embrace possibilities because we are constrained by perceived responsibilities to work that ultimately has no value.
Imagine if we could throw off those shackles. That’s worth letting a few things go.
Culture is often difficult to define, and culture change even more so – what concrete actions do we need to take to change a culture?
Despite this apparent difficulty, it is possible to spend an hour or two with a group, and leave with consensus on practical actions for culture change.
This exercise achieves that by make culture change something concrete. We look to the questions we ask everyday as reinforcing values and thus being drivers of culture. Then we challenge ourselves to find better questions, and explore what it will take to adopt those better questions in our specific context.
Questions driving culture
Let’s keep our definition of culture really simple: the sum of our everyday behaviours as a group.
To give an example: typically, you and your colleagues juggle many tasks at once. Multitasking is part of your culture.
What is driving this behaviour though? One strong driver is the questions that are asked in your group. For instance, in this environment, you probably find people explicitly asking something like “can you take this on?” The multitasking behaviour is a natural response to that question. Especially if all parties are, consciously or otherwise, implicitly asking themselves “how do we get everything done?”
Now let’s assume that you want to change your multitasking culture to one where people limit their work in progress to become more productive overall.
Making change more concrete
To change the behaviour, we can look for the driving questions and change those.
For instance, we might aim to change “how do we get everything done?” to “how do we do a great job of the most important things?”
And that is the heart of the change. If everyone is asking themselves, consciously or otherwise “how do we do a great job of the most important things?”, their behaviours will follow that question. In this case (and with training and support as required), we expect they will try to identify priorities, understand success and deliver on that before moving on to the next thing. People can helpfully answer “no” to the old question “can you take this on?”, but more importantly, that question will no longer be asked as frequently, because it will cease to make sense.
However, that’s still not as concrete a recipe as we would like. The exercise (below) helps us get down to the concrete actions required in a given context to change one driving question to another.
Before we go any further, though, a reminder that questions do not exist in isolation, and that we must tackle consistent set of questions simultaneously:
Today’s orthodoxy has institutionalised a set of internally consistent but dysfunctional beliefs. This has created a tightly interlocking and self-reinforcing system, a system from which it is very difficult to break free. Even when we change one piece, the other pieces hold us back by blocking the benefits of our change. When our change fails to produce benefits, we revert to our old approaches.
Donald G. Reinertsen, The Principles of Product Development Flow
This exercise can be run with the group whose culture we are looking to change.
At the end of the exercise, you will have a list of concrete actions that can be taken to change driving questions, and will have identified potential blockers to plan around.
- Observe the group and its behaviours
- Identify instances of counter-productive behaviours
- Analyse these behaviours to propose driving questions
- Pair current, undesirable driving questions with new, desirable driving questions
- Find examples to illustrate why each question should change
You should have something like the table below:
The exercise can then be run as follows:
- Discuss the premise of changing culture by changing questions
- Share your first example of a pair of driving questions, and the instance of the behaviour (this should be an instance widely understood and accepted by the group)
- Work through the other question pairs in your list, and ask the group to come up with examples themselves. They will generally do so enthusiastically! It’s unlikely, but if they don’t, you have your prepared examples to fall back on.
- Because you won’t be able to solve everything in this session, prioritise as a group (through dot voting, etc) the question pairs to focus on (no more than 3 for the first session). Allow 30 mins to 1 hour to get to this point.
- Now for each question pair, run an “anchors and engines” exercise to identify – in the group’s context – the potential blockers (“anchors”) and the supporting factors or concrete actions (“engines”). Take 15-30 minutes per pair. Synthesise individual contributions into themes.
You now have a set of concrete actions to support, and real issues that might hinder, the type of culture change you are seeking to achieve. It might look something like:
Of course, effort remains to make this change happen, but it can be directed very precisely, and that is valuable when dealing with culture.
I helped organise this event with assistance from sponsors ThoughtWorks and Curtin University (among numerous other generous sponsors). It was a great event, with important and challenging problems presented, innovative solution concepts delivered, and new relationships formed between individuals and organisations in health and technology.
Please refer to the report and the catalogue of products for detailed information on this event, and resources for hackathons in general. Health Hack is an Open Knowledge Foundation Australia event, so is predicated on sharing open source deliverables.
Some Highlights and Lessons Learned
We focussed on curated problems for this event, approaching a large number of potential “problem owners” with a checklist to recruit those with the most appropriate challenges for the weekend hackathon format. We then worked with the problem owners to shape their challenges and pitches for the “ideas market”. This was a very substantial effort (primarily by the fabulous Diana Adorno) in the lead-up to the weekend, but the well-formed problems were key to the success of the hack.
We attracted a diverse set of participants, with skills ranging from design, to software development, to data science, and these individuals organised themselves into teams around the problems most suited to their collective skill set. As organisers, we made only one substitution to balance teams.
We started with fewer participants than expected, because the drop-off rate from registrations was substantially higher (50%) than previous years at other sites (30%). However, attrition over the weekend was virtually zero, as the participants were uniformly enthusiastic and energised by their challenges.
The ideas market built great energy around the challenges and the potential for the weekend. We posted the challenges around the room prior to the event. Then the problems owners took turns to pitch in just 2 minutes each from their challenge posters. The pitches were clear and concise, and the cumulative effect was really energising. When the pitches were done, participants had time to walk the room, seek more information from problem owners, and organise their own teams.
Coaching and regular check-ins on team progress helped keep the teams focussed on solving key problems and having a demonstrable product at the end of the weekend. No team failed to showcase. However, we had feedback that access to more coaching would have been valuable.
The venue at Curtin University Chemistry Precinct was ideal, with team tables, breakout spaces and bean bags, and surrounded by gardens. However, it was the only Health Hack venue not in the CBD of the host city, and this may have presented transport challenges (though we didn’t collect any data on this). The plan at the time was to rotate the venue through various supporting institutions in future years.
Food trucks and coffee vans were a great way to service participants! Although it required some coordination ahead of the event, and may not be possible in CBD sites, it was very easy on the weekend, and lots of fun.
For more, see the full report.