Category: Machine Learning

Though but a footnote in the epic of coding with AI, I though it worth musing on my recent experience with Bard. Bard currently uses the LaMDA model, which is capable of generating code, but not optimised for it. The story might be different with Codey as protagonist (or is that antagonist?)

I didn’t produce any software I could use in this session, but I did learn that Bard was at its most useful when prompted with open questions or very specific concepts. I also learned that generated code may be sabotaged by plausible but incorrect domain logic and conceptual fault lines that occur between contexts.

Also, given that safetydave.net contributes 0.0000005% of tokens to the C4 dataset which makes up some of Bard’s training data, I thought I should see some return!

Act 1

In which we meet a domain.

Take 1

I started naively with a reasonably complex ask: how do I calculate the swept path of a vehicle in python?

I’ve done swept path analysis before, but in manufacturing, and I certainly don’t have a Python solution handy. Even if I only got the broad outlines of a solution, an answer or parts of an answer to this question would save a bunch of time.

This was far too wide an ask. I got code that looks like it’s for a curve that a vehicle might follow, but that’s just one part of the solution, and the more I looked at it, the more issues I found.

# The vehicle's position
x = np.array([0, 0])
y = np.array([0, 0])

# The vehicle's orientation
theta = 0

# The vehicle's velocity
v = 1

# The vehicle's wheelbase
w = 2

# Calculate the vehicle's position at each time step
for t in np.arange(0, 10, 0.1):
    x += v * np.cos(theta) * t
    y += v * np.sin(theta) * t
    theta += v / w * t

First, vehicle is a word that identifies a whole domain of assumptions, including how steering works. But this domain logic is weird; assuming theta represents direction of travel, that’s not how steering works, in this universe at least. Export to Colab feature is nice, so let’s see how the code runs. Spoiler: it doesn’t.

As a user of Python, I could diagnose the runtime error and fix it (the initial values of x and y could be defined with dtype=float), but on closer inspection, the += operator overwrites rather than appends, and, wait a minute, x and y are treated both as vectors and scalars. As the domain logic is still bizarre, is there any purpose in fixing these conceptual faults in the code?

Take 2

A secondary prompt to provide more guidance: I would like to consider the area swept out by the corners of the vehicle, can you update the solution?

This only resulted in more trigonometric prestidigitation; more domain dislogic. I use the term dislogic a bit like the term truthiness. There is some related logic to the code, but it is “apart” from the logic we need.

Take 3

I tried much more explicit guidance by elaborating on domain concepts in the third prompt. This lead to more detailed but still fundamentally flawed and unusable solution. I decided this path wouldn’t lead to a happy ending.

Morals of Act 1

Don’t assume too much domain expertise. Bard has learned on generic public examples. Use your domain expertise to break the problem into smaller chunks.

Also, don’t expect conceptual consistency throughout. LLMs like Bard, as next-token predictors, don’t necessarily ensure conceptual consistency in their output.

Act 2

In which I choose a path to follow.

Take 1

I decided to focus on one part of the solution; getting the curve right. I reset Bard’s context.

I want a python function to create a curve between two points. The function arguments should be the start and end points and the tangent to the curve at the start and end points

Nice linear interpolation, shame about the tangents (which, while present as arguments, were totally ignored in the function body).

And the above could only be generated after fixing more errors preventing the code from running. The affordances of tuples and numpy.ndarray were confused, and the coordinates weren’t passed correctly to the plot method. The syntax was fine, but the code was riven with conceptual fault lines between contexts – what looked OK in one or other context in isolation caused problems when the contexts were brought together. The bugs were fairly obvious in this case, but in general could be subtle and difficult to detect.

Still, after minor adjustments, it’s a curve that meets some of the requirements. This is more useful than what we got in Act 1.

Take 2

I augmented the initial prompt.

The curve tangent should match the tangents of the start and end point supplied as arguments. Please define points and tangents as numpy arrays

Now we have two nice linear interpolations, but the tangent is not the tangent of the curve! (It’s more like we have linearly interpolated the position and orientation of the vehicle.)

This code, presumably due to specifying numpy, worked straight out of the box.

Take 3

Maybe I can establish the link between the tangent and the curve to which it is tangent?

In the create curve function, the tangent should be calculated as unit vector in the direction from one point to the next on the curve. This calculated tangent should equal the tangent argument at both the start and end of the curve.

Nope, that was too naively optimistic. I was forgetting the Morals of Act 1.

Take 4

Let’s recall the Morals of Act 1, and try to link to concepts with better public support, such as Bézier curves.

The create curve function should be updated to return a bezier curve rather than a linear interpolation between start and end point

The result looked Bézier-ish, but the formula wasn’t transcribed faithfully, with the calculated points quickly racing away to infinity (well outside the convex hull of the control points).

# Calculate the point at the current index.
point = (1 - i / num_points)**3 * start_point + 3 * (1 - i / num_points)**2 * i / num_points * start_tangent + 3 * (1 - i / num_points) * (i / num_points)**2 * end_tangent + i**3 * end_point

However, I think we’re getting closer.

Take 5

Time to reset Bard’s context again and go all in on Bézier.

I would like a python function to create a bezier curve

Then,

Please update the solution to plot the curve with matplotlib

And with a few manual tweaks that are becoming familiar…

… we finally get a curve with the desired properties from Take 1!

However, this is curve is not necessarily going to represent the path of a vehicle, which has a minimum radius of curvature (the vehicle’s turning circle) that a Bézier curve can’t guarantee. Based on the experience above, it’s unclear to me how I would incorporate the curvature requirement with further prompts.

Taking stock: after much prompting, I have a method to create a class of curve I can’t directly use in the solution. (And the code for which I ended up looking up anyway.)

Moral of Act 2

Don’t forget the Morals of Act 1.

In addition, being able to directly reference the key concepts in a generic domain, rather than trying to explain the logic, may be key to getting closer to a correct solution in a few shots.

If you’re not familiar enough with the key concepts to reference them directly, you might use a more open prompt initially, such as: How do you create a smooth curve? This might point you in the right direction…

There are many ways to create a smooth curve. Here are a few examples:

* Use a Bezier curve. A Bezier curve is a parametric curve that is defined by a set of control points. The curve is smooth because it is a polynomial function of the control points.

* ... [goes on to reference Splines, Fourier series and B-Splines] ...

Act 3

Because we need 3 acts. In which I back up and broaden the domain.

Take 1

Considering whether the word vehicle in the initial prompt had given Bard the wrong steer, I tried the more generic prompt: how do I calculate the swept path of an object in python?

This gave helpful introductory commentary on breaking the problem down, and a nearly usable solution.

# Define the circle's geometry.
points = np.array(...)

# Define the circle's motion.
path = np.array(...)

# Calculate the swept path.
swept_path = np.zeros((len(points), len(path)))
for i in range(len(points)):
    for j in range(len(path)):
        swept_path[i][j] = points[i] + path[j]

But one that still needed an expert review to ensure values ended up in the all the right places.

Below we can see different coloured “circles” drawn at each position in the path.

This is pretty trivial though – it’s just organised vector addition – did I need AI for that?

Moral of Act 3

Keeping it simple increases the chance of success, but you should balance this against whether a simple solution provides sufficient value.

Concluding the saga, for now

I tried to use Bard to deliver large chunks of a complex solution, rather than as a smarter autocomplete for finer details, or as an aid to understanding existing or proposed solutions. In the time I spent prompting Bard, I would have got further writing code directly. However, I have a lot of scope to improve my prompting.

With expertise in the domain and the code, I was able to diagnose and correct the issues in Bard’s solutions, but I suspect that someone who lacked one or both of those areas of expertise couldn’t recover quickly. In some respects, developing software is about recovering quickly from errors – we can’t avoid making mistakes, but we set up feedback loops to detect them quickly, and over time we become vigilant to more of the types of mistakes we are likely to make. Does an AI coding assistant like Bard help us recover quickly from mistakes? I didn’t actually ask Bard to help much in this session, so that question needs further work to resolve, possibly taking the angle of AI-aided test-first development.

What I did learn that Bard was at its most useful when prompted with open questions or very specific concepts with public data support. I also learned that generated code is likely to be sabotaged by domain dislogic and conceptual fault lines between contexts.

Over time, we’ll figure out how to make AI a better protagonist and antagonist in our coding stories; for me, this was an interesting way to introduce a new character.