web-log/container/site/content/posts/local-llm-setup/index.md

+++ categories = ["software"] tags = ["llm","docker"] date = 2025-03-12T10:54:00-06:00 description = "Deploying a private local language-model stack with an NVIDIA GPU, Ollama, Open WebUI, and Docker Compose." draft = false slug = "local-llm-basic-setup" title = "🦙 Self-hosting LLM with Ollama & OpenWebUI" author = "nicholas" +++

I impulsively purchased a NVIDIA RTX 4060 at my local Best Buy. I thought it would be fun to run my own LLMs.In this post, I deploy only a minimal local LLM with web UI.

LLMs

LLMs continue to improve apace, getting smaller, more efficient, smarter, generally better in every way, every day, relentlessly and with such speed that I really cannot even pretend to have kept up with their developments to any meaningful degree. I merely absorb the general sense of things, passively, allowing the vibes of the techno-accelerationist-sphere wash over me. Every week a new set of capabilities are launched with a new version of a new model using a new paradigm. Exciting, truly what a time to be alive. Around the time of this writing, Deepseek released its R1 model which one can run on relatively low-spec hardware (this is true of many earlier models by other organizations, but none produced the same frenzied excitement as this one), so I hopped on the hype wagon and now I have a somewhat powerful reasoning model running on my computer. Running my own LLM locally is just for fun, and as of the time I write this confers no "practical" benefit to me to over using ChatGPT or Deepseek over the web.

Docker Machine

Typically I run all of my containers on my "server" machine. I have a few related posts about this:

• [Server build pt. I]({{< relref "posts/nas-build-pt-i">}})
• [Server build pt. II]({{< relref "posts/nas-build-pt-ii">}})

However, running LLMs locally will require more powerful hardware than my server machine contains. Luckily my [other machine]({{< relref "posts/pc-build">}}) does contain powerful enough components, so I can easily run some smaller LLMs e.g. quantized and distilled models.

I wrote a Docker Compose file that defines a service for running Ollama container, with support for GPU acceleration:

Docker Compose File

services:
  ollama:
    container_name: ollama
    image: ollama/ollama
    expose:
     - 11434/tcp
    ports:
     - 11434:11434/tcp
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            capabilities: ["gpu"]
            count: all
    volumes:
      - ollama:/root/.ollama
    restart: unless-stopped

volumes:
  ollama:

Running the Container

To run the container, I simply:

docker compose up -d

This will start the container in headless mode.

Interacting with LLM - CLI

{{< image src="images/ollama-docker-desktop.JPG" caption="Docker Desktop running Ollama" >}}

Now I can access the model via CLI. I happen to be using Docker Desktop on Windows, which allows me to access the container's shell via the UI. Here I can download different models and interact with them using the CLI. I demonstrate a few commands in the image, ollama --version to get the Ollama version, ollama list returns a list of the models installed (I already installed a few models to test). I can run a specific model with ollama run <model name> or ollama run <model id>. It works, incredible!

However, this is a poor way to interact with an LLM, and I can do better by running a web UI that will connect to Ollama API backend.

OpenWebUI

{{< image src="OpenWebUI" caption="images/openwebui.jpg" >}}

OpenWebUI is a web interface which I can use to interact with and manage my local models. I will also be running this in Docker.

Docker Compose File

Here is the Docker compose file I used to run OpenWebUI:

services:
  openwebui:
    restart: unless-stopped
    image: ghcr.io/open-webui/open-webui:cuda
    #network_mode: host
    ports:
      - "3000:8080"
    volumes:
      - ./data:/app/backend/data
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Ollama API Connection

Since I am running Ollama on port 1134 on the same machine, I configure the Ollama API connection to point to local machine: {{< image src="images/ollama-openwebui.JPG" caption="OpenWebUI ↔ Ollama Connection" >}}

GPU Usage

This screenshot shows that the model is indeed using my GPU to do all manner of linear algebra operations, etc., which eventually results in it generating some output text.

{{< image src="images/deepseek-r1-7b-GPU-usage.jpg" caption="GPU Usage - Deepseek r1:7b" >}}

Realtime Usage

{{< video src="videos/deepseek-r1-7b-GPU.mp4" width="100%" autoplay=true loop=true

}}

Potential Uses

I can integrate locally-hosted LLM APIs with new software projects. Who knows.