Clean history, git LFS

2025-07-05 15:36:34 +00:00
commit 70423ce723
141 changed files with 8484 additions and 0 deletions
@@ -0,0 +1,111 @@
+++
+categories = ["software","technology"]
+tags = ["llm","docker"]
+date = 2025-03-12T10:54:00-06:00
+description = ""
+draft = false
+slug = "local-llm-basic-setup"
+title = "🦙 Self-hosting LLM with Ollama & OpenWebUI"
+author = "nicholas"
+++
+
+I impulsively purchased a NVIDIA RTX 4060 at my local Best Buy. I thought it would be fun to run my own LLMs.In this post, I deploy only a minimal local LLM with web UI.
+
+## LLMs
+LLMs continue to improve apace, getting smaller, more efficient, smarter, generally better in every way, every day, relentlessly and with such speed that I really cannot even pretend to have kept up with their developments to any meaningful degree. I merely absorb the general *sense of things*, passively, allowing the vibes of the techno-accelerationist-sphere wash over me. Every week a new set of capabilities are launched with a new version of a new model using a new paradigm. Exciting, truly what a time to be alive. Around the time of this writing, Deepseek released its R1 model which one can run on relatively low-spec hardware (this is true of many earlier models by other organizations, but none produced the same frenzied excitement as this one), so I hopped on the hype wagon and now I have a somewhat powerful reasoning model running on my computer. Running my own LLM locally is just for fun, and as of the time I write this confers no "practical" benefit to me to over using ChatGPT or Deepseek over the web.
+
+## Docker Machine
+Typically I run all of my containers on my "server" machine. I have a few related posts about this: [Server build pt. I](), [Server build pt. II](). However, running LLMs locally will require more powerful hardware than my server machine contains. Luckily my [other machine]() does contain powerful enough components, so I can easily run some smaller LLMs e.g. quantized and distilled models.
+
+
+I wrote a Docker Compose file that defines a service for running Ollama container, with support for GPU acceleration:
+### Docker Compose File
+```yaml {filename="compose.yml"}
+services:
+  ollama:
+    container_name: ollama
+    image: ollama/ollama
+    expose:
+     - 11434/tcp
+    ports:
+     - 11434:11434/tcp
+    deploy:
+      resources:
+        reservations:
+          devices:
+          - driver: nvidia
+            capabilities: ["gpu"]
+            count: all
+    volumes:
+      - ollama:/root/.ollama
+    restart: unless-stopped
+
+volumes:
+  ollama:
+```
+### Running the Container
+To run the container, I simply:
+```shell
+docker compose up -d
+```
+This will start the container in headless mode.
+
+### Interacting with LLM - CLI
+{{< image 
+    src="images/ollama-docker-desktop.JPG" 
+    caption="Docker Desktop running Ollama" >}}
+
+Now I can access the model via CLI. I happen to be using Docker Desktop on Windows, which allows me to access the container's shell via the UI. Here I can download different models and interact with them using the CLI. I demonstrate a few commands in the image, `ollama --version` to get the Ollama version, `ollama list` returns a list of the models installed (I already installed a few models to test). I can run a specific model with `ollama run <model name>` or `ollama run <model id>`. It works, incredible!
+
+However, this is a poor way to interact with an LLM, and I can do better by running a web UI that will connect to Ollama API backend.
+
+## OpenWebUI
+{{< image 
+    src="OpenWebUI" 
+    caption="images/openwebui.jpg" >}}
+
+OpenWebUI is a web interface which I can use to interact with and manage my local models. I will also be running this in Docker. 
+
+### Docker Compose File
+Here is the Docker compose file I used to run OpenWebUI:
+```yaml
+services:
+  openwebui:
+    restart: unless-stopped
+    image: ghcr.io/open-webui/open-webui:cuda
+    #network_mode: host
+    ports:
+      - "3000:8080"
+    volumes:
+      - ./data:/app/backend/data
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+```
+### Ollama API Connection
+Since I am running Ollama on port 1134 on the same machine, I configure the Ollama API connection to point to local machine:
+{{< image 
+    src="images/ollama-openwebui.JPG" 
+    caption="OpenWebUI ↔ Ollama Connection" >}}
+
+## GPU Usage
+This screenshot shows that the model is indeed using my GPU to do all manner of linear algebra operations, etc., which eventually results in it generating some output text.
+
+{{< image 
+    src="images/deepseek-r1-7b-GPU-usage.jpg" 
+    caption="GPU Usage - Deepseek r1:7b" >}}
+
+## Realtime Usage
+{{< video
+  src="videos/deepseek-r1-7b-GPU.mp4"
+  width="100%"
+  autoplay=true
+  loop=true
+>}}
+
+## Potential Uses
+I can integrate locally-hosted LLM APIs with new software projects. Who knows.