LLMs: screenwriters vs characters #
I often see someone say "the AI said XYZ" but my understanding is that what an LLM like ChatGPT shows us is not the the LLM answering, but something slightly different. I am going to post my current understanding, with the expectation that my readers will correct me (xkcd 386 and/or Cunningham's Law).
When we chat with an LLM, we might see something like:
User: Hello. My name is John.
Assistant: Hello John.
I think it's useful to separate the different parts of the system here. The underlying LLM ("foundational model") does completion. It predicts the next word/token given a sequence:
User: Hello. My name is John. Assistant: Hello
User: Hello. My name is John. Assistant: Hello John
User: Hello. My name is John. Assistant: Hello John .
[Note for those of you reading in an RSS reader: I'm distinguishing my input from the LLM's output, and some RSS readers may not show the different formatting.]
Labels: structure