So-Called "AI" Cannot Program
I contend that it is impossible for so-called "AI", by which I mean the crop of convolutional neural network-based pattern-filling tools that is currently in the throes of a hype cycle that puts crypto and NFTs to shame, to meaningfully do programming. As more and more of the mind bogglingly rich tech oligarchs lead their followers into the delusion that these tools are useful in any way remotely comparable to their cost, I think it's worth taking the time to articulate exactly why this is, even if it's already intuitively clear to the more thoughtful practitioners of the craft.
We must begin, as always, with a clarity of the term under discussion. "Programming", in its broadest sense, is the act of making a computer do something—but "do something" is itself vague, so we'll need to dig a little deeper into that. To understand it, please bear with me as I establish some useful terminology.
A Brief Meander into the Philosophy of Language
Let's talk about syntax and semantics. If you're a programmer, you may be familiar with these terms as they're used to discuss programming languages. If you're not, that's fine too. I'll do my best to explain them in brief.
Syntax and semantics are two related ways of talking about an abstract structure. Although the concepts were originally developed to describe human languages, they can be used by analogy for all sorts of things, which is what I'm building up to here. They're very useful particularly for understanding how humans[1] relate to those structures and relate those structures to the world as they understand it.
"Syntax" is simply the structure itself, in all the technical detail of how it fits together. The syntax of a human language is the way its words and sentences fit together, what's "allowed" and "not allowed" by the subtle and mercurial rules we all internalized as children (or adults, for second languages and beyond). In English, "ran boy" is not a valid sentence due to the language's semantics, although most native speakers would probably guess that it means "the boy ran". Noam Chomsky wrote "colorless green ideas sleep furiously" as an example of a sentence that is syntactically valid despite being nonsense.
The fact that that sentence is nonsense is the domain of "semantics". Semantics are the human interpretation of a structure, the meaning we ascribe to it and to the best of our ability share with those around us. For example, although the syntax of the sentences "みなさんはピッコロさんが大好きだよ!" and "Everyone loves Piccolo!"[2] are completely different, their meaning—their semantic content—is exactly the same. And that meaning is something that humans bring to them, not something that is in any way intrinsically associated with those particular wiggly lines in that particular order.
Although syntax is a very useful concept in its own right, in this post I really want to focus on semantics. So remember: semantics is meaning, and specifically it's meaning applied to a structure from outside by humans.