Working with the grain of today’s AI

Today’s AI is powerful, but far from omnipotent. It’s a technology that, like any other, can be misapplied. Here I share some thoughts about appropriate and inappropriate uses, with reference to historically enduring AI applications, expanding on the final section of my YOW! Tech Leaders Summit talk.

Historically enduring AI applications

In my nearly three decades of experience, there are four primary applications for data & AI solutions:

1. Describe the world with data
2. Predict the future or other unknowns with models
3. Explore varied scenarios (through multiple predictions)
4. Solve for feasible or best actions or designs

While the applications endure, new tools have been added over time, unlocking new domains, such as moving from physics to business documents.

The last few years in particular show Generative AI augmenting or replacing a variety of historical techniques across all of the enduring applications (speaker notes on my journey in the source slide). Nonetheless, the applications endure.

We’ll use Large Language Models (LLMs) as the archetypal AI technology of today.

Testing with the grain

An important underlying consideration for all these applications is how to test any AI system. LLM text generation is deliberately non-deterministic, to provide more varied responses for user engagement and to avoid getting stuck in word loops in word loops in word loops.

Non-deterministic software components are not new, however, and have been employed for decades to drive user engagement and to avoid getting stuck in sub-optimal solutions, for instance:

• Random behaviours in games for engagement and emergent scenarios
• Randomisation of simulations that show sensitive dependence on initial conditions (aka the butterfly effect) to assure the generalizability of results, including: climate, car crashes and contact centres.
• Randomized searches for good designs in these non-linear domains,
• Machine learning training, including stochastic gradient descent, as a well-known example of randomized search for good design,
• Machine learning inference on novel inputs, of which LLM text generation is a current example.

All of these illustrate similar non-determinism to working with LLMs. The approach to testing has also been consistent historically across all of these examples, namely:

1. Modular system design that allows deterministic and non-deterministic modules to be tested and verified in known circumstances,
2. Calibrate the whole system behavior against known examples, such as real-world experiments or historical data, and
3. Trust and monitor when the application is deployed in novel scenarios, including an iterative feedback loop to improve the implementation.

Testing today’s AI is consistent with this historical approach, with these key call-outs:

1. Decompose into modules: LLM-based prototypes are often highly coupled, either due to poor problem definition or rapid solution code generation, so deliberate effort is required to decompose them into testable modules,
2. Evals is the term we use for calibrating the whole system behavior, and a critical part of testing any LLM-based system of any consequence.
3. The input and output space is huge and adversarial, so we must make a special effort to constrain inputs and outputs, and monitor extremely closely compared to historical applications with fewer degrees of freedom.

Antifragile AI Architectures includes patterns for testable AI solutions, and other concepts echoed below.

Describe the world

How AI works: LLMs represent text by compressing input text through a bottleneck, and regenerating likely output text from the compressed representation.

Going with the grain means: making use of this compression bottleneck for comparing text or searching for similar text, as this “embedding” technique is powerful and generalisable.

Going against the grain means: relying on this lossy compression for factual retrieval in high-stakes situations, as you will get something plausible but not necessarily factual from the text regeneration step. This becomes worse if results are fed through LLMs multiple times, as they lose fidelity on each cycle.

Predict unknowns

How AI works: LLMs predict likely text completions from their prompts. Prompts can include the user input, plus context and instructions, some of which may be visible to the user and some of which may be hidden, or “system”. They complete text by sampling from the next word (“token”) probability distributions they learned during their training, typically using further randomisation (“temperature”) for varied outputs or user engagement.

Going with the grain means: predicting quite complex “text” completions, which with a broad definition of text, includes any business document or artefact, for internal or for customer use, from slide decks to source code. Useful business documents can be generated easily with today’s tools, synthesising and transforming with natural language instructions many different inputs, to create in seconds or minutes products that may have taken hours or days previously. Many of these tasks can be described as translation problems, revealing the architectural heritage of LLMs as language translators.

You might also use LLMs in phases of developing a traditional machine learning predictor, such as a classifier (where the LLM output text identifies categories). Using an LLM simplifies feature engineering, and the training loop becomes tuning the prompt, so it can be more responsive than traditional ML approaches. But you still need an eval set in any application with consequences, and you might also give up the advantages of carefully designed features and loss functions to improve speed, cost and explainability at scale. You might also prefer that a machine takes care of the tedious learning optimisation, rather than subject humans to a prompt-tuning hamster wheel, so when the evals are sufficiently robust, you might consider transitioning to traditional ML. So building traditional predictors can go with or against the grain.

Going against the grain means: assuming that a human thought process went into producing any document. The LLM text generation process does not reflect human reasoning or writing (as should be apparent from how it works above). While tools are designed to given the impression of human-like thinking, research shows tricks like “reasoning traces” are just more loops of guessing likely text, and have a variable connection to better performance.

When complex text completion is done, you may simply have an artefact. You may not have a coherent thought process, or someone who struggled with the thinking and can talk to it. The artefact is probably longer, more complex, and harder to change than it needs to be, and probably bears hallmarks of “AI slop”. Nor is the AI that generated the artefact invested or accountable for taking action on the contents.

In short, if the process is important, as well as the product, use AI judiciously.

You also want to be very careful about guardrails if you expose this document-generation capability to customers, in any environment that could admit adversarial inputs, because an LLM only sees text to complete, and can be subverted by any part of the context it consumes.

Explore varied scenarios

How it works: Explore can be thought of as “making many predictions”. This can be done in an ad-hoc fashion to probe and understand a certain space, or in a structured fashion to quantify sensitivity to various factors, or solve for feasible or best designs. Explore is the bridge between predict and solve.

Going with the grain means: taking an iterative approach to generating business documents, rather than trying to “one-shot” results. It also means leaning into the inherent randomized nature of LLM text generation, and generating multiple triangulating business documents to assess the repeatability of one-shot results, if they have some consequence.

Going against the grain means: assuming one single solution for any complex LLM output, which we may do due to availability, automation, anchoring, framing or other biases. When I’ve done repeatability and robustness tests, I’ve been surprised how much the outputs can vary across identical runs or by making small variations in inputs – you should check this too for any application of consequence, because you may accept a sub-optimal, wrong or harmful solution.

Solve for good outcomes

How it works: LLMs, augmented with harnesses and tools, can solve for good actions, designs or products by searching the space of possible solutions and orchestrating the testing of candidate solutions.

Going with the grain means: conducting cheap searches (with bounded downside) for high-value solutions. A great example of this is generating prototype source code from natural language specs. This will work best for common solution scenarios that are well-represented in the model training data. The more tooling and harnessing you can deploy to either reduce the dimensionality of the search (for example, in coding, the use of a DSL) or constrain the space of feasible answers and/or provide some feedback signal to improve the fitness of the next iteration of solution (for example, in coding, many functional and cross-functional deterministic tests), the more likely this approach is to work and be cost-effective.

Going against the grain means: forgetting that any application in this space is a constrained satisfaction or optimization problem. If we don’t a priori have a good definition of success to solve for, we might be confusing solve with predict or explore, so be sure to revise how to work with or against the grain in those cases. When we don’t have a clear definition of success before we start, we’re more likely to accept any answer generated by an LLM, due to biases like availability, automation, anchoring, framing, etc, and therefore less likely to generate high-performing answer. Endless right answers are compensated by endless mediocre and wrong answers. Again, see explore recommendations.

If we can define success up-front, then we need a way to test it, and we will probably need to test each of the many candidate solutions an LLM generates. Testing like this may be unfeasible or non-viable in the real world, for reasons of cost or safety. Testing in a virtual environment is also only feasible or viable for certain classes of problems, such as coding, or by using physics simulators, etc. Not being able to test effectively means solve is a non-starter, and we’re back at explore and hope.

Now that we’ve defined success and can evaluate it, we might find we can’t guarantee LLMs will produce correct solutions, and even if they can, we might find that many problems will be better solved with other mathematical optimization techniques, techniques that will be much more efficient or explainable than LLMs. AI-assisted coding is somewhat unique in that we can easily test the solutions LLMs propose, but not many real world applications have this desirable property, and we should be cautious of assuming they do.

In this respect, we must remember that any verified solution is typically only part of a containing validation cycle. For instance, in product development, we can verify that generated code satisfied requirements with tests, but we can only validate it meets customer needs by bringing real world customers into the loop. No matter how much code we can generate and how fast we can do so, if doing so reduces the effectiveness of the outer validation loop, then it is a net negative application.

Finally, we must critically evaluate our ability to constraint adversarial inputs in our input supply chain for solutions, as we can’t feasibly test for all potential exploits in solutions with sufficient degrees of freedom, like software in general.

Grainy resolution

I feel like I’ve come at these questions from a few angles recently (see the Leading data and AI teams resources for the other perspectives), so apologies for any repetition, but these different framings have helped me articulate the opportunity and challenge of working with today’s AI more fully; take what works for you.