LLMs and shaders #
In the last post I described how chat interfaces to LLMs are giving a limited view of what the LLMs could do. In this post I wanted to compare this to graphics pipelines.
Graphics pipelines used to be "fixed function", where the graphics system (such as Silicon Graphics hardware) would offer you the graphics features:
- you always got a matrix multiply (translate, scale, rotate, perspective, etc.)
- you always got gouraud shading
- you always got their standard lighting model (ambient, diffuse, specular, emission, shininess)
- you got to choose up to 8 lights with glLight() with a handful of styles
- you always got a color per object, but later they offered a texture lookup
- you got to control the fog level with glFog() (density, color, etc.)
But what if you wanted a different model like phong shading? Too bad! What if you wanted any effect other than fog? Too bad! What if you wanted more than 8 lights? Too bad! What if you wanted 2 textures? Too bad! At least until they added a feature to enable a second texture lookup. If you wanted anything else, you needed to implement your own graphics pipeline and give up on hardware acceleration.
Over time, these fixed features were replaced by "shaders". The vertex shader would handle the overall structure with matrix multiply, clipping, fog, colors. Once we could control it, people came up with cloth, grass, water, terrain, animation, procedural geometry, and lots of other things. The fragment shader would handle fine details with texture lookup and lighting. Once we could control it, people came up with custom lighting, organic textures, animation, procedural art, and lots of other things.
Labels: speculation, structure
LLMs: screenwriters vs characters #
I often see someone say "the AI said XYZ" but my understanding is that what an LLM like ChatGPT shows us is not the the LLM answering, but something slightly different. I am going to post my current understanding, with the expectation that my readers will correct me (xkcd 386 and/or Cunningham's Law).
When we chat with an LLM, we might see something like:
User: Hello. My name is John. Assistant: Hello John.
I think it's useful to separate the different parts of the system here. The underlying LLM ("foundational model") does completion. It predicts the next word/token given a sequence:
User: Hello. My name is John. Assistant: Hello
User: Hello. My name is John. Assistant: Hello John
User: Hello. My name is John. Assistant: Hello John .
[Note for those of you reading in an RSS reader: I'm distinguishing my input from the LLM's output, and some RSS readers may not show the different formatting.]
Labels: structure
Emacs per-project colors #
For several years now I've been using color in my Emacs buffers to mark whether it's a "work" project or a "personal" project. I think I first showed this in a blog post I wrote in 2017, when I had switched to spaceline's mode line. Since then I've extended that color to also work on the tab line, current line, and header line. I wanted to make it work with the cursor color but couldn't figure that out.
I saw a question on stack exchange today that made me realize I could extend this to have one color per project, similar to how rainbow-identifiers works.
Labels: emacs
Emacs Tree-sitter custom highlighting, part 3 #
In the last post I described how, for a particular project with certain naming conventions, I wanted to use emacs tree-sitter font-locking to highlight variables that might be used in the wrong context:
I found a way to make it work for simple expressions but it was relying on regular expression matching rather than tree-sitter parse trees. I set a goal of handling these types of expressions:
elevation_t[t] // good elevation_t[r] // bug elevation_t[t_from_r[r]] // good elevation_t[r_from_t[t]] // bug elevation_t[obj.t] // good elevation_t[obj.r] // bug elevation_t[t_from_r(r)] // good elevation_t[r_from_t(t)] // bug elevation_t[x.t_fn(x)] // good elevation_t[x.r_fn(x)] // bug elevation_t[x.t_arr[i]] // good elevation_t[x.r_arr[i]] // bug elevation_t[(t)] // good elevation_t[(r)] // bug
Labels: emacs
Emacs Tree-sitter custom highlighting, part 2 #
In the last post I described my initial attempt at customizing font-lock using Emacs 29 tree-sitter. Regular font locking uses regular expression matching. Tree sitter allows combining regular expression matching with parse tree matching, like "variable declaration" or "function call". The ideas I came up with were mostly things that I could have implemented with regular expressions alone, but I wanted to experiment with the tree sitter approach. Here's the effect of conventional highlighting, using color for keywords and other syntax:
Labels: emacs