Using an LLM to query my notes #

robot asks a question holding a book
"robot asks a question holding a book" image from stable diffusion

I occasionally play with LLMs (large language models). Last time I had tried Simon Willison's llm software to search over my notes. Today I tried amaiya/onprem to ask questions about my notes. The documentation is pretty good but I made a few changes for my configuration, so I decided to write them down here. I'm running these on my Mac M1 and not on my Linux or Windows machines.

First step, get the thing installed:

cd ~/Projects/machine-learning/
mkdir onprem
cd onprem

# need to use python 3.9 through 3.11 for torch support; 3.12 doesn't support it yet
# check the version I have installed
python3 --version
# if python3 isn't a reasonable version, then install or choose a different python
# before doing the next step

python3.11 -m venv venv
source venv/bin/activate

# install onprem itself, which also installs torch
pip install onprem
mkdir data