22 rules of generative AI

Thinking about adopting, incorporating or building generative AI products? Here are some things to think about, depending on your role or roles.

I assume you’re bringing your own application based on an understanding of an opportunity or problem that involves creating, combining or transforming some kind of digital content. If you don’t have that understanding of a customer problem and why other solutions are not suitable, go and get it! Digital content may mean text, code, images, sound, video, 3D, etc, for digital consumption, or it may mean digitized designs for real world products or services such as code (again), recipes, blueprints, etc. Some of this may also be relevant for how you use other people’s generative AI tools in your own work.

Product strategy and management roles

1. Know what input you have to an AI product or feature that’s difficult to replicate. This is generally proprietary data, but it may be an algorithm tuned in-house, access to compute resources, or a particularly responsive deployment process, etc. This separates competitive differentiators from competitive parity features.

2. Interrogate the role of data. Do you need historical data to start, can you generate what you need through experimentation, or can you leverage your proprietary data with open source data, modelling techniques or SaaS products? Work with you technical leads to understand the multitude of mathematical and ML techniques available to ensure data adds the most value for the least effort.

3. Understand where to use open source or Commercial Off-The-Shelf (COTS) software for parity features, but also understand the risks of COTS including roadmaps, implementation, operations and data.

4. Recognise that functional performance of AI features is uncertain at the outset and variable in operation, which creates delivery risk. Address this by: creating a safe experimentation environment, supporting dual discovery (creating knowledge) and development (creating software) tracks with a continuous delivery approach, and – perhaps the hardest part – actually responding to change.

Design roles

5. Design for failure, and loss of vigilance in the face of rare failures. Failure can mean outputs that are nonsensical, fabricated, incorrect, or – depending on scope and training data – harmful.

6. Learn the affordances of AI technologies so you understand how to incorporate them into user experiences, and can effectively communicate their function to your users.

7. Study various emerging UX patterns. My quick take: generative AI may be used as a discrete tool with (considering #5) predictable results for the user, such as replacing the background in a photo, it may be used as a collaborator, reliant on a dialogue or back-and-forth iterative design or search process between the user and AI, such as ChatGPT, or it may be used as an author, producing a nearly finished work that the user then edits to their satisfaction (which comes with risk of subtle undetected errors).

8. Consider what role the AI is playing in the collaborator pattern – is it designer, builder, tester, or will the user decide? There is value in generating novel options to explore as a designer, in expediting complex workflows as a builder, and in verifying or validating solutions to some level of fidelity as a tester. However, for testing, remember you can not inspect quality into a product, and consider building in quality from the start.

9. Design for explainability, to help users understand how their actions influence the output. (This overlaps heavily with #6)

10. More and more stakeholders will want to know what goes into their AI products. If you haven’t already, start on your labelling scheme for AI features, which may include: intended use, data ingredients and production process, warnings, reporting process, and so on, with reference to risk and governance below.

Data science and data engineering roles

11. Work in short cycles in multidisciplinary product teams to address end-to-end delivery risks.

12. Quantify the functional performance of systems, the satisfaction of guardrails, and confidence in these measures for to support product decisions.

13. Make it technically easy and safe to work with and combine rich data.

14. Implement and automate a data governance model that enables delivery of data products and AI features to the support business strategy (i.e., a governance model that captures the concerns of other rules and stakeholders here).

Architecture and software engineering roles

15. Understand that each AI solution is narrow, but composable with other digital services. In this respect, treat each AI solution as a distinct service until a compelling case is made for consolidation. (Note that, as above, product management should be aware of how to make use of existing solutions.)

16. Consolidate AI platform services at the right level of abstraction. The implementation of AI services may be somewhat consistent, or it may be completely idiosyncratic depending on the solution requirements and available techniques. The right level of abstraction may be emergent and big up-front design may be risky.

17. Use continuous delivery for short feedback cycles and delivery that is both iterative – to reduce risk from knowledge gaps – and responsive – to reduce the risk of a changing world.

18. Continuous delivery necessitates a robust testing and monitoring strategy. Make use of test pyramids for both code and data for economical and timely quality assurance.

Risk and governance roles

19. Privacy and data security are the foundation on which everything else is built.

20. Generative AI solutions, like other AI solutions, may also perpetuate harmful content, biases or correlations in their historical training data.

21. Understand that current generative AI solutions may be subject to some or all of the following legal and ethical issues, depending on their source data, training and deployment as a service: privacy, copyright or other violation regarding collection of training data, outputs that plagiarise or create “digital forgeries” of training data, whether the aggregation and intermediation of individual creators at scale is monopoly behaviour and whether original creators should be compensated, that training data may include harmful content (which may be replicated into harmful outputs), that people may have been exposed to harmful content in a moderation process, and that storing data and the compute for training and inference may have substantial environmental costs.

22. Develop strategies to address the further structural failure modes of AI solutions, such as: misalignment with user goals, deployment into ethically unsound applications, the issue of illusory progress where small gains may look promising but never cross the required threshold, the magnification of rare failures at scale and the resolution of any liability for those failures.

Conclusion

These are the type of role-based considerations I alluded to in Reasoning About Machine Creativity. The list is far from complete, and the reader would doubtless benefit from sources and references! I intended to write this post in one shot, which I did in 90 minutes while hitting the target 22 rules without significant editing, so I will return after some reflection. Let me know if these considerations are helpful in your roles.

Data Mesh Radio

I joined Scott Hirleman for an episode (#95) of the Data Mesh Radio podcast. Scott does great work connecting and educating the data mesh community, and we had fun talking about:

  • Fitness functions to define “what good looks like” for data mesh and guide the evolution of analytic data architecture and operating model
  • Team topologies as a system for organisational design that is sympathetic to data mesh
  • Driving a delivery program through use cases
  • Thin slicing and evolution of products

My episode is #95 Measuring Your Data Mesh Journey Progress with Fitness Functions

Data mesh: a lean perspective

Data mesh can be understood as a response to lean wastes identified in data organisations. I paired with Ned Letcher to present this perspective at the LAST Conference 2021, which was much delayed due to COVID restrictions.

Lean wastes including overproduction, inventory, etc, may be concealed and made more difficult to address by centralised data systems and team architectures. Conversely, data mesh may make these wastes visible where they exist and provides mechanisms for reducing these wastes.

The presentation is written up in this article, and the slides are also available.

Guiding the Evolution of Data Mesh with Fitness Functions

I presented this webinar with Zhamak Dehghani – see the recording Guiding the Evolution of Data Mesh with Fitness Functions. There was great engagement with the topic and we captured some questions and further thoughts on this mini-blog post, published a little later.

This presentation brought together the idea of architectural fitness functions from the book Building Evolutionary Architectures with the core data mesh principles and logical architecture.

Our thoughts around guiding fitness functions included the below. These high-level measurable objectives were supported by a range of proposed metrics. This table is a handy summary; check out the webinar for more.

Domain Ownership
Scaling sources and consumers
Truthfulness
Domain autonomy
Reduced accidental complexity
Data as a Product
Serving users’ needs
Ease of discovery
Evaluation of quality
Service levels
Self-Serve Data Platform
Abstraction of complexity
Domain team autonomy
Protocols enable an ecosystem
Automation
Federated Computational Governance
Governance for common good
Degree of decentralisation
Interoperability
Increasing returns from scale

Scaling Change

Once upon a time, scaling production may have been enough to be competitive. Now, the most competitive organisations scale change to continually improve customer experience. How can we use what we’ve learned scaling production to scale change?

Metaphors for scaling
Metaphors for scaling

I recently presented a talk titled “Scaling Change”. In the talk I explore the connections between scaling production, sustaining software development, and scaling change, using metaphors, maths and management heuristics. The same model of change applies from organisational, marketing, design and technology perspectives.  How can factories, home loans and nightclubs help us to think about and manage change at scale?

Read on with the spoiler post if you’d rather get right to the heart of the talk.

Scaling Change Spoiler

When software engineers think about scaling, they think in terms of the order of complexity, or “Big-O“, of a process or system. Whereas production is O(N) and can be scaled by shifting variable costs to fixed, I contend that change is O(N2) due to the interaction of each new change with all previous changes. We could visualise this as a triangular matrix heat map of the interaction cost of each pair of changes (where darker shading is higher cost).

Change heatmap
Change interaction heatmap

The thing about change being O(N2) is that the old production management heuristics of shifting variable cost to fixed no longer work, because the dominant mode is interaction cost. Instead we use the following management heuristics:

Socialise

Socialising change
Socialising change

We take a variable cost hit for each change to help it play more nicely with every other change. This reduces the cost coefficient but not the number of interactions (N2).

Screen

Screening change
Screening change

We only take in the most valuable changes. Screening half our changes (N/2) reduces change interactions by three quarters (N2/4).

Seclude

Secluding change
Secluding change

We arrange changes into separate spaces and prevent interaction between spaces. Using n spaces reduces the interactions to N2/n.

Surrender

Surrendering change
Surrendering change

Like screening, but at the other end. We actively manage out changes to reduce interactions. Surrendering half our changes (N/2) reduces change interactions by three quarters (N2/4).

Scenarios

Where do we see these approaches being used? Just some examples:

  • Start-ups screen or surrender changes and hence are more agile than incumbents because they have less history of change.
  • Product managers screen changes in design and seclude changes across a portfolio, for example the separate apps of Facebook/ Messenger/ Instagram/ Hyperlapse/ Layout/ Boomerang/ etc
  • To manage technical debt, good developers socialise via refactoring, better seclude through architecture, and the best surrender
  • In hiring, candidates are screened and socialised through rigorous recruitment and training processes
  • Brand architectures also seclude changes – Unilever’s Dove can campaign for real beauty while Axe/Lynx offends Dove’s targets (and many others).

See Also

The life-changing magic of tidying your work

Surprise! Managing work in a large organisation is a lot like keeping your belongings in check at home.

Get it wrong at home and you have mess and clutter. Get it wrong in the organisation and you have excessive work in progress (WIP), retarding responsiveness, pulverising productivity, and eroding engagement.

Reading Marie Kondo’s The Life-Changing Magic of Tidying Up (Amazon), I was struck by a number of observations about tidying personal belongings that resonated with how individuals, teams and organisations manage their work.

First, reading TLCMOTU helped me tidy my things better. Second, it reinforced lean and agile management principles.

I won’t review the book here. Maybe the methods and ideas resonate with you, maybe they don’t. However, because I think tidying is something that everyone can relate to, I will compare some of KonMari’s (as Marie Kondo is known) explanations of the management of personal belongings with the management of work in organisations. The translation heuristic is to replace stuff with work, and clutter with excessive WIP, to highlight the parallels.

I’d love to know if you find the comparison useful.

On the complexity of work storage systems

KonMari writes:

Most people realise that clutter is caused by too much stuff. But why do we have too much stuff? Usually it is because we do not accurately grasp how much we actually own. And we fail to grasp how much we own because our storage methods are too complex.

Organisations typically employ complex storage methods for their work: portfolio and project management systems with myriad arcane properties, intricate plans, baselines and revisions, budget and planning cycle constraints, capitalisation constraints, fractional resource allocations, and restricted access to specialists who are removed from the outcomes but embrace the management complexity.

And this is just the work that’s stored where it should be. Then there’s all the work that’s squirrelled away into nooks and crannies that has to be teased out by thorough investigation (see below).

Because organisations don’t comprehend the extent of their work, they invent ever-more complex systems to stuff work into storage maximise utilisation of capacity, which continues to hide the extent of the work.

Thus, we fail to grasp how much work is held in the organisation, and the result is excessive WIP, which inflates lead times and reduces productivity, failing customers and leaving workers disengaged. Simplifying the storage of work – as simple as cards on a wall, with the information we actually need to deliver outcomes – allows us to comprehend the work we hold, and allows us to better manage WIP for responsiveness and productivity.

On making things visible

KonMari observes that you cannot accurately assess how much stuff you have without seeing it all in one place. She recommends searching the whole house first, bringing everything to the one location, and spreading the items out on the floor to gain visibility.

Making work visible, in one place, to all stakeholders is a tenet of agile and lean delivery. It reveals amazing insights, many unanticipated, about the volume, variety and value (or lack of) of work in progress. The shared view helps build empathy and collaboration between stakeholders and delivery teams. You may need to search extensively within the organisation to discover all the work, but understanding of the sources of demand (as below) will guide you. A great resource for ideas and examples of approaches is Agile Board Hacks.

So get your work on cards on a wall so you can see the extent of your WIP.

On categories

KonMari observes that items in one category are stored in multiple different places, spread out around the house. Categories she identifies include clothes, books, etc. She contends that it’s not possible to assess what you want to keep and discard without seeing the sum of your belongings in each category. Consequently, she recommends thinking in terms of category, rather than place.

If we think organisationally in terms of place, we think of silos – projects, teams, functions. We can’t use these storage units to properly assess the work we hold in the organisation. Internal silos don’t reflect how we serve customers.

Instead, if we think organisationally in terms of category, we are thinking strategically. With a cascading decomposition of strategy, driven by the customer, we can assess the work in the organisation at every level for strategic alignment (strategy being emergent as well as explicit). Strategy could be enterprise level themes, or the desired customer journey at a product team level.

With work mapped against strategy, we can see in one place the sum of efforts to execute a given branch of strategy, and hence assess what to keep and what to discard. We further can assess whether the entire portfolio of work is sufficiently aligned and diversified to execute strategy.

So use your card wall to identify how work strategically serves your customers.

On joy

KonMari writes:

The best way to choose what to keep and what to throw away is to … ask: ‘Does this spark joy?’ If it does keep it. If not, throw it out.

We may ask of each piece of work: ‘Is this work valuable?’ ‘Is it aligned to the purpose of the organisation?’ ‘Is it something customers want?’ If it is, keep it. If not, throw it out.

KonMari demonstrates why this is effective by taking the process to its logical conclusion. If you’ve discarded everything that doesn’t spark joy, then everything you have, everything you interact with, does spark joy.

What better way to spark joy in your people than to reduce or eliminate work with no value and no purpose?

On discarding first

KonMari observes that storage considerations interrupt the process of discarding. She recommends that discarding comes first, and storage comes second, and the activities remain distinct. If you start to think about where to put something before you have decided whether to keep or discard it, you will stop discarding.

Prioritisation is the act of discarding work we do not intend to pursue. Prioritisation comes first, based purely on value, before implementation considerations. Sequencing can be done with knowledge of effort and other dependencies. Then scheduling, given capacity and other constraints, is the process of deciding which “drawers” to put work in.

On putting things away

KonMari observes that mess and clutter is a result of not putting things away. Consequently she recommends that storage systems should make it easy to put things away, not easy to get them out.

Excessive WIP may also be caused by a failure to rapidly stop work (or perceived inability to do so). Organisational approaches to work should reduce the effort needed to stop work. For instance, with continuous delivery, a product is releasable at all times, and can therefore be stopped after any deployment. Work should be easily stoppable in preference to easily startable. (This could also be framed as “stop starting and start finishing”.)

Further, while many organisations aim for responsiveness with a stoppable workforce (of contractors), they should instead aim for a stoppable portfolio, and workforce responsiveness will follow.

On letting things go

A client of KonMari’s comments:

Up to now, I believed it was important to do things that added to my life …  I realised for the first time that letting go is even more important than adding.

I have written about the importance of letting go of work from the perspective of via negativa management in Dumbbell Delivery; Antifragile Software, and managing socialisation costs in Your Software is a Nightclub.

However, KonMari also observes that, beyond the mechanics of managing stuff (or work), there is a psychological cost of clutter (or excessive WIP). Her clients often report feeling constrained by perceived responsibility to stuff that brings them no joy. I suspect the same is true in the organisation: we fail to recognise and embrace possibilities because we are constrained by perceived responsibilities to work that ultimately has no value.

Imagine if we could throw off those shackles. That’s worth letting a few things go.