This is the only post on this blog that an LLM was used for in any way. I had whisper generate a transcript from the audio above and then tried to have claude edit that transcript into a blog post. I naively thought it would oneshot that (it's basically text style transfer and that's been a thing for a long time) but the first two attempts were unusably bad and even the third one, after prompting that it should stick to the transcript and stop using em-dashes it didn't do a great job. Anyway if you want the real, 100% human-generated version watch the youtube video.

Pulling Words Out of a Hat

Say you want to make a thing that can generate text. Your goal is to generate new text that is indistinguishable from text a human wrote. How do you go about doing that?

What you do is you go and buy a bookstore. You buy out the whole bookstore, the whole inventory. You shut the door, closed for business, and then you take all the books and put them in a big shredder. You shred them into little pieces where each piece is a word or a couple of words long. Then you take all those pieces of your shredded books and put them in a big mixer, or a big tumbler, like you're picking lottery numbers, and you mix them all around and repeatedly pull pieces of paper out at random. You pull a piece of paper out at random that has a couple of words on it and paste it down on your paper. Then you pull another piece, another couple of words at random, and paste it after that. And another, and you keep doing that until you fill the page up.

That is a way to generate text. And if you looked at that, it would be recognizable as text, certainly a lot more recognizable than if you were doing it a letter at a time. You'd look at it and say, okay, that's definitely English. But it's not grammatical, it's not saying anything. It's gibberish. If you have a spectrum between baby talk and actual English spoken by a real person, it's better than baby talk, but definitely not saying anything.

The technical term for those little pieces of paper is n-grams. An n-gram is just a couple of words. A two-gram is two words, a three-gram is three words, and so on. Those pieces of paper are a real thing in actual computers, and they're called n-grams.

Sorting Words Out of a Hat

What we want is to have the words make sense one after another. Right now there's no relationship between a word and the word that comes after it. When we pick another snippet, the likelihood that we pick that one isn't influenced at all by what came before it.

So what if we make longer snippets. Say each snippet, instead of being two words long, is four words long. We pick a snippet at random, put it down on the paper, then look at the two words at the end and go find, maybe we have some filing system that lets us do this quickly, a snippet where the two words at the beginning are the same as the two words at the end of the one we just picked, and we paste that down. Then we take the two words at the end of that one and find another that matches, and put that down, and so on. That gives us some dependency between the thing that came before and the thing that came after.

When you do that, you do get text that actually looks like it's kind of saying something. It goes off track very quickly, it meanders, a few words at a time will kind of make sense in isolation, but then it wanders off, losing its train of thought. There isn't actually a train of thought, but the experience of reading it feels like it's wandering off its train of thought. Still, it's definitely an improvement over just picking snippets totally at random. Now the sentences kind of look like sentences, they kind of look grammatical. We don't have a system with any inbuilt notion of grammar, but just by having those dependencies there, the way the probabilities work out, you get something that's kind of grammatical.

This thing is called an n-gram Markov chain. The dependency on the thing that came before is what makes it a Markov chain. This was somewhat popular on Twitter, within a certain corner of Twitter, in the middle 2010s. horse_ebooks worked like that. There were also a lot of people who made one of these using their own tweets as the source material, as the training data (training data is just another word for source material). They generated text that kind of looks like real text, and often it's funny, because it has no idea of concepts. There's no semantics. You get topics that seem unrelated mashed together, but in a kind of grammatical way, and that's often funny.

So that's an improvement, but the text still isn't really saying anything. We've gotten to the point where we're generating grammatical sentences, but there's no content there, the sentences aren't going anywhere. So what if we lean on that dependency more? Instead of matching on the previous two words, what if we go farther back? Well, if we just say we have to have four words of overlap, we wouldn't be able to pick anything, very quickly we'd end up in a dead end where there isn't anything whose beginning four words are the same as the ending four words of this one. When you get to that position, you can either stop or pick another thing at random, but you've reached a dead end.

So what if, instead of a hard and fast exact match, we have multiple kinds of matching and assign a score based on how many of them matched? Say we look at the prior one word, and the prior two words, and the prior three words but skipping. Imagine with our pieces of paper we cut holes in them. Say we have a six word long piece of paper and we cut a hole where the second word is. Anything can go there, that's a wildcard. Those are called skip-grams. The hole we're cutting is the skip, we're skipping that word. So we can take that same piece of text and generate a bunch of different versions of it, the forward version, the two word version, the skip one, the skip two, and multiple of them can potentially match. The more that match, the better we're doing for that particular match. We add up how many matches this particular sequence got versus how many this other one got, and so on, and pick the one with the best score.

If we do that, we're depending on more of the context, more of the things coming before in the text, and by having the scoring thing, this lets us depend on more context without quickly running out of source material, because we have this fuzzier notion of matching with the scoring. And if we do that, we do get text that makes more sense, that's less full of non-sequiturs. But the trade-off is that by adding more dependency, these things tend to get stuck in loops. The technical term for the sequences that get you stuck in a loop is attractors, sequences of words that loop back on themselves. You'll get "he opened the book and," and then what's likely to come after "and"? Well, it could be "the book." And what's likely to come after "and the book"? Well, "and the book" again. So you get "and the book and the book and the book," stuck in a loop.

To avoid getting stuck in loops like that, we want to add some randomness. So instead of just picking the one with the highest score, we can rank the scores, add up the total of all the scores, and divide each score by that total, which turns them all into probabilities between zero and one. Then we pick at random with those probabilities. If something has a normalized score of 0.2, twenty percent of the time we'll pick that one. If one has a normalized score of 0.01, one percent of the time we'll pick that one, and so on. We're injecting some randomness in, but biasing the randomness with the scores, so things with higher scores are more likely to get picked, but they're not guaranteed. At each step in a loop, there's some chance we get knocked out of it. That is actually how Markov chains work in practice.

That solves the looping problem, and the skips do extend our training data somewhat, in that each snippet can do multiple duties. But we're still in a situation where the more dependency we add on the prior context, more skips, longer n-grams, more overlap required, the more likely we are to just be regurgitating something that's already in our source material. We don't want that. We want to stretch our source material further, somehow.

The Meaning of a Word is the Company It Keeps

One thing we could do is say, well, we can probably replace words with their synonyms. If we have a particular word, instead of saying it has to match this exact word, we say it can match this word or any other word listed as a synonym of it in a thesaurus. That extends our source material somewhat, and we get text that's still kind of consistent and not totally nonsensical, while doing less regurgitating of the books we shredded.

But then, what's in the thesaurus is kind of limited, the list of words listed as synonyms for a particular word isn't very long. And really, because we're adding things up at the end of the day to generate a score, what we want isn't to just ask is this an exact synonym, we want to ask how synonymous is this. Instead of a binary synonym or not, it's like, well, it's half a synonym, it's a tenth of a synonym. If something is half a synonym, when we're adding up the matches we can give it half points.

So how might we come up with this notion of half a synonym? What does it even mean for two words to be synonyms? One idea, going back to the Anglo-analytic tradition, which is where the line of thought that leads to LLMs comes from, is the old phrase that the meaning of a word is the company it keeps. That's John Rupert Firth. The meaning of a word is the context in which it's used. There's a similar idea from Saussure in the continental tradition, that word meanings are usage context.

How do we turn that into numbers? We can take all of our text, all of our books. We've shredded one bookstore, but we have an unlimited budget, so let's go buy another one, this time not shredded yet. Then we take a sliding window, called a context window, of some length, let's say a hundred words. We open a book, start at the beginning, take the first hundred words, and make a big table for adding up. For each pair of words in that hundred words, word 1 and 2, 1 and 3, 1 and 4, and so on, we add up the number of times that pair occurs. "The rain in Spain" turns into the-rain, the-in, the-Spain, and so on, row and column, add one, add one, add one. We do that for all pairs in that window, then move the window over one word, so it starts at the second word in the book, and do it again, for every word in the book, for every book in the bookstore. We end up with this giant table of word pairs and counts.

That table gives us a notion of similarity for free. To compute similarity between word A and word B, we take the row for A and the row for B, subtract those two rows column by column and add up the differences, then go through again and add the two rows column by column and add up the sums, and take the differences divided by the sums. That gives us a number between zero and one, a similarity score. So instead of asking whether a word is a listed synonym, we go into this big table, look up the score, and add that to the score for the snippet, and that gives us our fuzzy notion of similarity. If snippets aren't exact synonyms but have similar kinds of words, they somewhat match. Because we're adding up scores over multiple words, a large number of moderate partial matches can add up to the same as one exact match, which before this, the exact match got points and the non-exact got no points at all.

The rows in that big table are called word vectors. The word vectors used in practice are a compressed version, you don't use the whole row, you crunch it down to maybe a few hundred dimensions instead of a dimension for every word in the dictionary. There are two approaches to that compression. One uses a linear algebra technique called PCA, and in this context that approach is called LSA, latent semantic analysis. The other uses a single layer neural network called word2vec, which is the more famous of the two, and the newer one. To digress a little, a single layer neural network of the kind word2vec is turns out to be a first order approximation of PCA, so word2vec is a first order approximation of LSA, and that goes back to a result from the 1980s. But anyway, word2vec is the famous one. It's just compression though, we're taking our co-occurrence table and compressing it so it's easier to work with. If the compression works well, our relative scores are preserved. That's actually the objective of the compression, to maintain those relative similarity scores.

Attention

Now we have something that can generate text that doesn't get stuck in loops, looks relatively grammatical, and kind of makes sense over a short stretch. If you try to generate a whole page it's going to meander all over the place, but tweet length, that works fine. And it doesn't regurgitate, usually, a book from our source material, in most cases it's not going to go and regurgitate Harry Potter. But the limitation is we can't generate a whole page and have it be coherent. It kind of, for a single paragraph, looks like it has a train of thought, but a whole page meanders and doesn't know what it's trying to say.

So we want longer range consistency, and this is the same game we've been playing this whole time, the trade-off between novelty and consistency, where we want things to not just be regurgitating, but also to make sense. We can have it be more consistent by having it look farther back in what it's previously generated, and having longer range dependencies, longer snippets, but with more holes in them so we're more likely to get a match, even though they're longer.

Extend the skip idea, and instead of n-grams with holes, come up with templates. The way we've been cutting out holes is kind of a template already, a three-skip seven-gram is a template. But what if we apply that template to everything we've generated so far, and go back and look at the original context, and see if that long template matches the prior context in the book we're pulling from. Those templates are called attention masks. And now we're getting to attention, which if you've heard of large language models, was the big deal.

We've been advancing forward in time the way this technology actually developed historically, and we're getting up to about 2017 now. If we pick the right templates, based on the most recent stuff we just generated, we can pull semantic information from the previous context in the book without ending up regurgitating the whole book. You can think of that template as semantic context, especially combined with our word vectors so synonyms count too. Pick the right templates, and that gives us another way to cheat on the trade-off between regurgitating and generating nonsense, by strategically depending on only certain things from way back in the context.

How do we pick those templates? In real large language models, that's done using stochastic gradient descent, which I'm not going to get into here since I'm trying to avoid math beyond basic arithmetic. There are plenty of good explanations of how neural networks and gradient descent work, go look at one of those. But roughly, it's an optimization algorithm, we're trying to minimize a loss function, and the loss function is predicting the next word, so we're finding templates that do a good job at accurately choosing the next word. It's a great big tournament for all the templates, and the ones that win are the ones we keep. There's another algorithm that's more tournament-like called genetic algorithms, more intuitive if you don't have a math background, but that's not actually what these things use. Genetic algorithms is less efficient than stochastic gradient descent, though they're solving the same problem.

One more thing, large language models don't use words exactly, they use tokens, which are almost but not quite the same as words. There are plenty of explainers online about how byte pair encoding works to generate tokens. For our purposes, tokens are words. If our context is a million tokens long, our templates can potentially be a million long, so the range of possible templates, if holes could be anywhere, is astronomical. We need to be strategic about which ones we pick, using an optimization algorithm like gradient descent.

We can also choose which templates to use based on the recent context. That's done using a neural net, but you can imagine it the same way we've been doing Markov chains, generating the next thing depending on the previous thing, except now it's which template we pick that depends on what we most recently generated. Some templates get used in some contexts, others in other contexts. We pick a template, apply it to the stuff we've generated before, and see which next words come after the matches through the holes in the template, in the words we've generated before. That lets us potentially look extremely far back in the context, but because we're only looking at certain things, not forcing everything that came before to be exactly the same or even synonyms, only certain words have to match, and we're giving things scores so they don't have to be exact matches. That's what captures long-range dependencies, holding on to things we said earlier in the page and having that still influence what we're generating at the bottom of the page. That lets us generate things that are page length instead of paragraph length. That was the big innovation with transformers in 2017.

So now we're getting up to GPT, we've entered the large language model era at this point. Everything up until now was historical, how the technology developed. But once we add these attention masks, these templates, now we're in the realm of modern large language models.

There's one other piece large language models have, besides the attention masks: neural networks. A neural network is a type of regression model. Because we've taken our words and turned them into numeric vectors, we can treat those vectors as data we can do statistics on, and use a statistical model to predict, given these vectors, what vector is most likely to come next. We're still using all the template stuff and tokens to generate the vectors and choose which vectors go into that regression model at the end. So the actual statistics-in-the-normal-sense part is just the smoothing at the end. The stuff actually looking at the context window is our attention masks. The neural network at the end, the MLP, multi-layer perceptron, a classical neural network, is just a matrix we're multiplying the word vectors of the things we've sampled by. We take the things that poke through the templates, put them in a big vector, multiply that by a matrix of parameters fit to accurately predict the next thing. But that MLP at the end isn't where most of the magic happens in a large language model, it's mostly in the attention layers.

So that lets us generate a page of text that kind of makes sense, without much control over what it's about. We get a little control by putting what we type in at the top of the page. If we start with "a long time ago in a galaxy far, far away," we're probably going to get science fiction. If we start with "once upon a time in the dark forest there lived a little old lady," we're probably going to get a fairy tale.

ChatGPT

The thing that turned GPT into ChatGPT was really two things.

One, we also want stylistic control, we want to generate a certain kind of text, not just anything that happened to be in the training data. That's done by a technique called reinforcement learning from human feedback, RLHF. We take some people, our taste testers, usually called labelers, who provide feedback on the text the model generates, thumbs up or thumbs down. We want to generate text that gets thumbs up. There are really two things we can change: which words we attend to, which words poke through the holes in the template, and which words we're more likely to generate, which is the MLP part. We can update both of those, since they're both trained by stochastic gradient descent, and you can do that gradient descent online. Get feedback, use it to update things, see if the feedback got better or worse, do more or less of that accordingly, repeat.

There's also a way to strategically choose which text to generate to get the most useful feedback possible, which gets into probability distributions, bandit optimization, that's a whole separate topic I won't go into here. But basically, the same tournament idea we used for picking attention masks, we can do again, using feedback instead of predicting the next word.

Two, something called InstructGPT. If RLHF lets us make updates based on feedback, what if we have a question and answer type interface, and format it consistently, "Question: ... Answer: ..."? We probably start by generating a bunch of question and answer pairs so that we get an attention mask that looks at questions and says, oh, I need an answer, because a question showed up in one of the holes in the template. If we do that, and give the answers to people, and get feedback on how good the answers were, we can adjust our weights based on that feedback. That gives us something that can be a chatbot, a conversational agent, which is what people were calling these around 2020. Something with a back and forth, that looks far back in context and uses that to pull information out of what we trained on, mostly web pages in GPT's case.

One more detail: it's not actually going back to the books in most cases. There's an approach that does do that, called RAG, which modern chatbots all use, but GPT-3 didn't. The compressed version of all those dependencies is stored in the weights of the network itself, via gradient descent.

That's GPT-3, and that was the thing people went crazy over when it was released, and enthralled Bill Gates enough that Microsoft poured a huge amount of money into scaling it up.

Why They Hallucinate

They hallucinate because you're pasting snippets one after the other that kind of make sense in the same contexts, but that text can have come from different documents about different things, and you end up with context confusion, generating output that's saying half of one thing and half of another in a way that ends up grammatical but wrong. It generates new and wrong things in the same way cut up poetry does. If you've seen cut up poetry, where you take snippets from a newspaper that say whole things and paste them one after another, you get something with a completely different meaning. Even whole sentences or paragraphs, lifted from text where they made sense in context, when put one after the other, might even be grammatical, because of the word vectors, the synonym information, and the attention masks, it's very good at generating things that are grammatical rather than absurd or nonsensical. The reinforcement learning feedback makes it good at generating things that don't sound absurd, it sounds grammatically reasonable, but it's cut up poetry with an editing pass, and it's still total BS. That's why it generates fake citations, citations have a consistent structure, so if you shredded up a bunch of citations and pasted them together, you'd get something that looks like a citation but is just made up. That's where hallucinations come from, they make things up the way cut up poetry makes things up.

Postscripot

So what's changed since GPT-3? Not actually a whole lot. GPT-5 is bigger than GPT-3, but the way it works is fundamentally the same. What's changed is RAG and tools. There's a thing sitting in front of the model that, when it gets input from the user, puts it in the question-answer format. You can also train the network to generate things in a consistent format that mean take what comes after this, treat it as a search query, go out on a search engine and search for it, take the results and paste them back in the context window as if they'd been generated by the model, then continue. Sometimes it's not a search query in text, sometimes it's another one of those word vectors, and there are efficient ways to take a vector and find similar matches, called approximate nearest neighbor search. Regular old search engines have been using word vectors like that for ten years, even pre-LLM, when you type a query into Google it gets turned into one of those vectors and does an approximate nearest neighbor search. That accounts for most of the improvement in chatbots since GPT-4.

The other thing they added is tools generally. The big one, the reason chatbots mostly stopped failing at how many R's are in strawberry or basic arithmetic, is more of this reinforcement learning training, teaching the model to recognize when it's in a situation where nothing it generates directly is going to get good feedback, but it can generate Python code, run it, and put the output back in the context window the same way search results get pasted in, and continue from that. Generate a ton of synthetic training data like that, and you get improvement. That, plus probably some higher quality training data, rare books and things they've spent a lot of money on, is pretty much the whole story of improvement in chatbots since GPT-4.

Agents

That's been the hot thing over the past year, agents. Agents are more of the tool use thing, except you only do tool use. The model generates something that's interpreted as "go out on the internet and perform this action," and the harness, the thing running the model, and by the way the harness is just a normal piece of software, no machine learning or AI in the harness itself, sees that and goes and does it, then takes the result and pastes it back in, which causes the model to generate more text, which eventually is interpreted as "now go do this," and it does, and so on.

If you have an environment where you can run agents like this, in a harness, and get feedback on whether it achieved a goal, in a synthetic environment, people sometimes call these training gyms, a contained environment where it's not actually going out on the internet but getting feedback as if it had, then you can do reinforcement learning, if you have absurd amounts of energy and GPUs to throw at it. Generate tons of these fake contexts, update the parameters based on the feedback, until it's able to get to the goal state.

The only real application for this is coding. Coding is a special case because there's a lot of training data that looks like a sequence of steps like this: commit histories. When somebody writes a piece of software, every edit they make gets recorded, so you have a history of all the edits that person made, and that's agent training data. What it's generating isn't the code, it's the sequence of edits that add up to the code, edits based on prior context. That makes it reasonably good at reproducing software that already exists. It can't do anything new, but if what you're trying to build can be built by applying patches that have already been applied by other people, over and over, in the right order, you have a decent shot a coding agent can generate it. It's actually quite uncommon for it to zero-shot a piece of software just from a prompt, usually it generates a bunch of patches then gets stuck because there wasn't a sequence of patches in the training data that gets to the result, and somebody has to go in and either make the edits themselves or explain what to do differently. If you're trying to do something with many examples in the training data, or you're willing to spend a lot of time giving it feedback, you can get software out of it, usually pretty bad quality, but not everyone cares about quality, most software is made to get thrown out anyway.

Using them for booking hotel reservations and stuff like that isn't really a thing in practice. There was this thing called OpenClaw, maybe about a year ago, that was really in vogue, a lot of people were using it, and I have not seen any actual use cases for it, almost everybody who was using it a year ago isn't using it anymore. The idea was it's your AI butler, it can do anything you want. But if I've done a good job explaining this, it should be obvious that using one of these as your AI butler is like using a Ouija board as your AI butler, having web pages and books that have already been written, and rolls of the dice, take control of your affairs. Not Jesus take the wheel, fate take the wheel. Not really a good idea, and it hasn't really caught on. What's caught on is coding agents, which again are a special case, and they work as well as they do because of these commit histories.

So anyway, I hope that was helpful, I hope it was understandable, especially if you don't have a computer science or math background. That is how large language models work.