Local AI on Your Home Network: How to Combine Ollama, Docker, and Open-WebUI on Your Server (Without Your Data Leaving the House)

We all know that AI has fundamentally changed the way we work. However, if you regularly analyze sensitive documents, debug internal scripts, or simply value true data sovereignty, commercial cloud services quickly hit a wall. Why route every prompt and configuration snippet through third-party servers when your own home infrastructure is already running 24/7?

The good news: running modern Large Language Models (LLMs) like Llama 3.2, Mistral, or Qwen locally no longer requires a data center or hours of wrestling with broken Python dependencies. By combining Ollama (as a lightweight backend for the models) and Open-WebUI (as a full-featured web interface in the style of ChatGPT), you can spin up a completely self-contained AI environment on your Linux server in just a few minutes.

What does this mean in practice? And what does a clean, maintainable setup look like?

Why This Exact Stack?

If you already run services like Nextcloud, Paperless-ngx, or Vaultwarden in containers, you are looking for two things above all else: maintainability and clean isolation. That is exactly where the combination of Ollama and Open-WebUI shines:

• Full Data Control: Not a single token leaves your local network. No training third-party models with your data, no hidden API costs.
• Clean Container Architecture: Both components run isolated in Docker and can be managed effortlessly as a stack in Portainer or directly via docker compose.
• Hardware Flexibility: Whether you have a dedicated NVIDIA GPU in your server or are starting out with a modern CPU and 16 GB of RAM, Ollama scales remarkably well with compact models.
• More Than Just Chat: Open-WebUI comes with built-in local RAG (Retrieval-Augmented Generation). You can drag and drop PDFs, log files, or Markdown documents straight into the chat and analyze them locally.

Prerequisites: What Hardware Do You Need?

Before diving into the configuration, let’s take a realistic look at hardware resources. You do not need a high-end workstation to get started:

1. CPU-Only Setup (The Pragmatic Entry Point): A modern multi-core CPU (e.g., Intel Core i5/i7 or AMD Ryzen) and at least 16 GB RAM are plenty to run compact models with 3 to 8 billion parameters (such as llama3.2:3b or mistral:7b) smoothly.
2. GPU Acceleration (For Maximum Speed): If you can pass through a dedicated graphics card (NVIDIA with CUDA or AMD with ROCm), response times drop from a few words per second to near real-time. Larger 14B or 32B models also become viable, provided you have enough VRAM.

Step 1: The Docker Compose Stack

The cleanest approach is to deploy both services together in a docker-compose.yml file (or paste the block directly as a new stack in Portainer). This way, Ollama and Open-WebUI share an internal Docker network—meaning you don’t even have to expose the Ollama port (11434) to the host unless external tools need access.

YAML

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ollama_data:/root/.ollama
    # Optional: Only expose this port if external tools (e.g., IDE plugins) need direct access
    ports:
      - "11434:11434"
    # --- GPU Support (NVIDIA) ---
    # Uncomment the block below if you have an NVIDIA GPU + Container Toolkit installed:
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_AUTH=true
      # Prevent unwanted telemetry or external connection checks
      - SCARF_NO_ANALYTICS=true
      - DO_NOT_TRACK=true
    volumes:
      - open_webui_data:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama_data:
  open_webui_data:

Start the stack from your terminal:

Bash

docker compose up -d

Once the containers are up, open your browser and navigate to http://<YOUR-SERVER-IP>:3000. The first account you create automatically receives administrator privileges.

Step 2: Pulling the Right Models

After your first login, Open-WebUI will be empty because Ollama ships without pre-installed language models by default. You have two ways to pull your first model:

• Directly via the GUI: In Open-WebUI, click on your profile in the bottom left -> Admin Panel -> Settings -> Models. Under “Pull a model from Ollama.com”, simply enter the model tag (e.g., llama3.2:3b).
• Via the Terminal: Using the command line is often faster and gives you direct progress output:

Bash

# Lightweight and extremely fast on CPUs (approx. 2.0 GB)
docker exec -it ollama ollama pull llama3.2:3b

# The all-rounder for complex text and summaries (approx. 4.7 GB)
docker exec -it ollama ollama pull llama3.1:8b

# Specialist for Bash scripts, YAML, Python & Linux configs (approx. 4.7 GB)
docker exec -it ollama ollama pull qwen2.5-coder:7b

Practical Tip on Model Selection: Don’t get fixated on massive parameter counts when running on local hardware. A well-quantized 3B or 7B model handles 80% of daily tasks (proofreading, writing regex, debugging Docker configs, summarizing documents) with impressive quality—without bringing your server to its knees.

Step 3: Securing and Integrating into Your Home Network

To integrate the setup seamlessly into your existing infrastructure, three quick tweaks are well worth the effort:

1. Put a Reverse Proxy in Front: Instead of memorizing port 3000, set up a proxy host in Nginx Proxy Manager (or Traefik) such as ai.home.arpa or your own local subdomain with an SSL certificate. Important: Make sure to enable Websockets Support in Nginx Proxy Manager, otherwise token streaming in the chat interface will fail.
2. Optimize RAM Management (OLLAMA_KEEP_ALIVE): By default, Ollama keeps a loaded model in memory for 5 minutes before unloading it. If your server hosts several other containers, you can add the environment variable OLLAMA_KEEP_ALIVE=2m (or 0 to unload immediately after each response) under the ollama service in your docker-compose.yml.
3. Safe Remote Access: Avoid exposing Open-WebUI directly to the public internet. If you want to access your local AI from your laptop or phone while on the go, route it cleanly through your WireGuard VPN.

Conclusion

All in all, local AI is no longer just a weekend tinkering project—it is a solid, practical addition to any serious homelab. With Ollama and Open-WebUI, you get a fast, privacy-first workspace that stays entirely under your control, from the first prompt to the last byte on your drive.

What’s your take on this? Are you already running LLMs locally on your own hardware, or do you still rely on cloud APIs?