LLMs: screenwriters vs characters

From Amit’s Thoughts

I often see someone say "the AI said XYZ" but my understanding is that what an LLM like ChatGPT shows us is not the the LLM answering, but something slightly different. I am going to post my current understanding, with the expectation that my readers will correct me (xkcd 386 and/or Cunningham's Law).

When we chat with an LLM, we might see something like:

User: Hello. My name is John. Assistant: Hello John.

I think it's useful to separate the different parts of the system here. The underlying LLM ("foundational model") does completion. It predicts the next word/token given a sequence:

User: Hello. My name is John. Assistant: Hello

User: Hello. My name is John. Assistant: Hello John

User: Hello. My name is John. Assistant: Hello John .

[Note for those of you reading in an RSS reader: I'm distinguishing my input from the LLM's output, and some RSS readers may not show the different formatting.]

Let's keep going a few more steps.

User: Hello. My name is John. Assistant: Hello John. My name is

The LLM will make predictions about the next word. But it doesn't only predict one possible word; it assigns a probability to each possible next word. Here's the table of the next possible words (using token-explorer, command line uv run -p 3.12 main.py --input <(echo -n 'User: Hello. My name is John. Assistant:')):

So that's interesting. In this simple model, it thinks John is the most likely next word, because that's the only name it has seen so far. It also has lots of common names in the list. It assigns a probability of (seeing My name is John | given My name is) at 43%.

If have it pick also instead of John, the next probability table is:

Given that the only name it has seen so far as John, it assigns a probability of (seeing My name is also John | given My name is also) at 90%. This is pretty good. There aren't many other words that make sense here.

I want to pause a moment and point out a few things:

• The LLM is predicting the probability of the next word. It's not choosing a word.
• The LLM does not inherently know the difference between what the user wrote and what the LLM wrote.
• The LLM is stateless. It doesn't remember how far it has completed. The state is the transcript itself.

What can we do with these observations?

One thing is that we can build a chat app:

1. Start with a string like "User:"
2. Let the user type in a string, and append it to the string.
3. Feed the entire string to the LLM and get a prediction of the next word.
4. Choose the highest probability word and append it to the string.
5. Repeat. The LLM will add one word at a time to make its response.

Let's try it. Let's always pick the top word, running uv run -p 3.12 main.py --input <(echo -n 'User: What is a baby cow called? Assistant:'):

• User: What is a young sheep called? Assistant: A young sheep is called a lamb.

If I run it again, again picking the top word every time, I will get the same answer:

• User: What is a young sheep called? Assistant: A young sheep is called a lamb.

This chatbot is "deterministic". It will produce the same answer every time. If we want some variety, we can change step 4: instead of always picking the highest probability word, we can sometimes pick one of the other words. How often we do that is controlled by a "temperature" setting in some chatbots. I'll try running it multiple times, picking different paths each time:

• User: What is a young sheep called? Assistant: A young sheep is called a lamb or a calf.
• User: What is a young sheep called? Assistant: A young sheep is called an ewe.
• User: What is a young sheep called? Assistant: A young sheep is known as a lamb.
• User: What is a young sheep called? Assistant: A young sheep is a lamb. A lamb is a small, fluffy, woolly animal that is usually around 10-12 months old.

That has potential. We could write a multi-chat bot that prints out multiple responses to a user's question. Or we could put some conditions on the output, like choosing short words, or words that start with a vowel, or words that don't contain the letter e.

Another thing we can do with the chatbot is put something at the beginning of the string that influences how it answers. With the small model I'm using, I wasn't able to get much variation:

• The following is a transcript between a user and an assistant named Alexa. User: Hello. What is your name? Assistant: Hello, my name is Alexa.
• The following is a transcript between a user and an assistant named Jacob. User: Hello. What is your name? Assistant: Hello, I'm your assistant, Jacob.

But with a more capable model we should be able to insert instructions such as The assistant gives snarky answers or The assistant gives verbose answers or The assistant always rhymes. Note that we are telling the LLM about the character. The "answers" we get from the LLM aren't us talking to the LLM. We're talking to a character that it's writing a script for.

Another thing we can play with is changing the text that the LLM wrote. The LLM doesn't distinguish between the text the user wrote and the text it wrote. After all, it doesn't actually write any text. It only predicts word probabilities, and then we (our chatbot code) inserts that text. That means we can go back and change the text from the chatbot, and have it complete the altered text. With the small model I'm playing with, it's not very interesting:

• User: What is a young sheep called? Assistant: A young sheep is not called a "young sheep"

But with a larger model, we could have some interesting possibilities for inserting "thoughts" into the LLM's output (gaslighting?). Or we could go back and change one thing in the past (time travel?) and see how the conversation evolves differently.

The LLM is stateless. The state is held "outside" the LLM, in the chatbot. We can copy or modify that state. One thing we can do with this is a "transporter accident" like Will vs Tom Riker in Star Trek. We can make a copy of the state, then let the chatbot continue. But then we can restore that state, change something, and have the chatbot continue from that modified state.

The commercial services like ChatGPT, Claude, Gemini, etc., do not give you this level of access. They present the LLM as a simple chatbot. Their systems have many more layers than what I'm presenting here (such as RLHF / instruction training), but if we have direct access to an LLM like Llama or DeepSeek, there are so many more ways to put the ingredients together than a simple chatbot.

After learning about these things, I think the main thing I learned is that the screenwriter and the character are different. Think of the LLM as the one who is writing the script. It is not the character in the script. The LLM predicts what the character will say, but it is not the one we are "talking" to. The commercial chatbots make it hard to see this difference, but we can see it by playing with the underlying LLM technology.

The next time you hear "the AI said XYZ", read it as "the AI predicted that a fictional character would say XYZ".

Labels: structure