Four GPUs, Zero Cloud: Running LLMs Across a Heterogeneous Home GPU Cluster

The problem: one GPU becomes the whole queue

If you already own a gaming GPU, the first local model is the easy part. The harder question is what happens when you want a second model, a longer conversation, or an image generation job while the first model is busy.

A card has a fixed memory budget. Loading another model does not expand it. Neither does changing the number printed on the context-window slider.

Cloud GPU rental can solve a particular capacity problem. For my home setup, I want a different default: use the hardware I own, keep inference at home, and give each machine a clear job. That still costs electricity, hardware, storage, and maintenance. “Zero cloud” here means no rented inference GPUs, not a free system or a network disconnected from the internet.

My cluster is three machines and four GPUs in one home in California. Windows and Ubuntu both feature in the story. So do NVIDIA and Intel. This is a practical architecture for separate serving endpoints, not a claim that I have built one giant GPU.

If you have a workstation and an older GPU-equipped desktop, you may already have the beginnings of this arrangement. Start by separating jobs before attempting distributed inference.

The cluster: count the memory correctly

Here is the hardware I am working with, using manufacturer-rated GPU memory capacities.

Intel specifies 32GB per B70. For these card models, the arithmetic is 32 + 24 + 32 + 32 = 120GB of rated VRAM across four GPUs. That is a sum of manufacturer capacities, not a measurement of currently available memory.

Check the installed GPU model and dedicated-memory readout before using the rated inventory to plan a deployment. Drivers, desktop applications, and resident models determine how much of that capacity is actually available.

Rated memory is an inventory number. Available memory is a runtime number. Schedulable capacity additionally depends on a working backend, device selection, and other running jobs.

The network is an eero Max 7 plus TP-Link Deco BE63 setup, with a multi-gigabit LAN and partial 10 GbE. Tailscale is the remote-access boundary in this design. The examples below keep inference ports private.

Architecture: give each endpoint a job

My useful unit of work is a request sent to a model server. Open WebUI provides one front door; the servers behind it remain separate.

Browser on the tailnet
        |
        v
Open WebUI on OG3090
        |
        +--> OG3090: Ollama / NVIDIA serving
        +--> 5090 Beast: NVIDIA serving endpoint
        +--> 14600K: llama.cpp / SYCL endpoint A
        +--> 14600K: llama.cpp / SYCL endpoint B

For the 3090, an 8B-class model is a sensible starting point. A 32B Q4 model is a candidate at modest context: the weight estimate below is roughly 19.2GB, before cache and runtime allocations. It is not a promise that every “32B” file fits in 24GB.

The 5090 gives larger models and longer contexts more room. A 32B Q5 candidate has approximately 22.8GB of weights; a Q6 candidate is closer to 26.3GB. Either requires a real memory-budget check. Desktop applications and ComfyUI also compete for that card.

A conventional dense 70B Q4_K_M model is about 42GB of weights by the estimate below. It does not fit fully in the 5090's 32GB. CPU offload, a smaller quantization, or supported multi-GPU sharding changes the experiment, with different quality and performance tradeoffs. I would not call any of those a measured win without testing.

For the B70s, I would first run independent SYCL workers, selecting one physical card per process. Each 32GB card makes a 32B Q4 model a memory candidate, assuming backend support and enough headroom. Two cards in one machine can also be an explicit sharding experiment. That is a different configuration from two independent workers.

The 3090's memory does not automatically join the 5090's over Tailscale. Open WebUI is a model selector and client, not a VRAM pool or an automatic GPU scheduler. Independent endpoints let me run separate workloads concurrently; they do not make one model fit everywhere.

VRAM math that actually matters

The first estimate is weight storage:

weight GB = parameter count × effective bits per weight / 8 / 1,000,000,000

“Four-bit” is shorthand. Quantization blocks also store scales and, for some formats, minima. GGML's block definitions give Q8_0 34 bytes per 32 values, or 8.5 bits per weight; Q6_K uses 210 bytes per 256 values, or 6.5625 bits. Pure Q4_K and Q5_K blocks use 4.5 and 5.5 bits respectively.

The _M formats mix tensor precisions. For a rough planning table I use 4.8 bits for Q4_K_M and 5.7 bits for Q5_K_M. Those are explicitly chosen approximation inputs, not fixed format guarantees. The llama.cpp quantizer allows mixed types and overrides; the actual GGUF file and loader report settle the question.

These estimates assume exactly 8, 32, or 70 billion parameters. Real model names round their parameter counts, and some tensors remain at higher precision. Metadata, alignment, vision projectors, and execution buffers are outside this table. A mixture-of-experts model needs a separate calculation for stored parameters; its active parameter count is not its weight-storage requirement.

Here is the 32B Q4 calculation: 32 × 4.8 / 8 = 19.2GB, approximately 17.88GiB. GB means decimal billions of bytes; GiB means powers of 1024. Compare actual byte counts or consistently converted units, not two labels that happen to look similar.

Then budget the cache. For a conventional decoder with grouped-query attention, a simple unquantized KV-cache estimate is:

KV bytes = 2 × layers × KV heads × head dimension
           × cached tokens × bytes per element × concurrent sequences

As a worked example, assume 32 layers, 8 KV heads, head dimension 128, FP16 cache, and one sequence. This is an illustrative architecture, not a specification for every 8B model:

2 × 32 × 8 × 128 × 8,192 × 2 = 1,073,741,824 bytes = 1GiB
At 32,768 tokens: 4GiB
At 4 concurrent 8,192-token sequences: approximately 4GiB

Different attention designs, shared prefixes, sliding windows, cache quantization, and runtime allocation policies change that result. Weight quantization does not automatically quantize the KV cache.

My final budget is weights plus cache plus execution buffers plus everything else using the card. I start with one request and a short context, inspect the loader's memory allocations, then increase context or concurrency separately.

These are planning estimates, not measured fit or throughput results for this reference configuration. For a real deployment, record the exact model artifact and revision, file size, context, cache type, offloaded layers, selected physical GPU, and peak dedicated memory.

Networking without exposed ports

I want one reachable UI and narrowly reachable backends. In the reference setup, inference services bind to loopback; Tailscale Serve publishes their HTTP interfaces inside the tailnet. Serve supplies the private HTTPS reverse proxy. Funnel is not part of this design.

Open WebUI keeps its login enabled. Tailnet policy should permit my devices to reach the UI and permit OG3090 to reach the worker APIs, rather than letting every tailnet member call every backend. Private addressing by itself is not an access policy.

Before calling a deployment private, check its effective tailnet policy, existing listeners, firewall rules, and allow/deny behavior from real clients. The reference recipe here is not a security audit of an existing installation.

Wake-on-LAN needs a second path. A sleeping machine cannot receive a command through its inactive Tailscale client. I reach an always-on machine over Tailscale, then have that machine send the magic packet on the sleeping machine's local network. Tailscale's WoL guide explains this relay arrangement.

Confirm the waking NIC’s MAC address, local broadcast subnet, BIOS/driver settings, and successful wake from the specific sleep or shutdown state you intend to use.

A reference setup you can inspect

Save the following examples under guides/heterogeneous-gpu-cluster/. This is a reference recipe checked against upstream interfaces, rather than an export of my running services. Start on an unused test host or adapt ports deliberately.

The examples use readable image tags and illustrative model paths. Before relying on them, pin tested container digests and a llama.cpp release, record driver/runtime versions, and reproduce a completion on the target hardware.

OG3090: Ollama and Open WebUI

Prerequisites are Docker Engine with Compose v2, NVIDIA Container Toolkit configured for Docker, and Tailscale on the Ubuntu host. The NVIDIA reservation exposes the local GPU to Ollama. Named volumes keep models and UI data across container restarts.

Save as compose.yaml:

services:
  ollama:
    image: ollama/ollama:latest
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    restart: unless-stopped
    depends_on: [ollama]
    ports:
      - "127.0.0.1:3000:8080"
    environment:
      OLLAMA_BASE_URL: http://ollama:11434
      WEBUI_SECRET_KEY: ${WEBUI_SECRET_KEY:?Set WEBUI_SECRET_KEY in .env}
    volumes:
      - open-webui:/app/backend/data

volumes:
  ollama:
  open-webui:

The bridge-network URL names the Ollama service, not the container's own localhost. Open WebUI documents the image, persistent volume, and connection configuration. The floating tags make the example readable; substitute tested digests for a repeatable deployment.

# In the guide directory on the Ubuntu test host:
if [ ! -e .env ]; then
  umask 077
  printf 'WEBUI_SECRET_KEY=%s\n' "$(openssl rand -hex 32)" > .env
fi
docker compose config --quiet
docker compose up -d
docker compose exec ollama ollama pull llama3.1:8b
docker compose exec ollama ollama run llama3.1:8b \
  "Reply with exactly LOCAL_OK"
docker compose exec ollama ollama ps
tailscale serve --bg --https=443 http://127.0.0.1:3000
tailscale serve status

Use the HTTPS URL printed by Serve to create the first admin account. Keep .env out of Git. ollama ps is part of the check: a completed answer alone does not prove the intended GPU served it.

NVIDIA throughput experiments: vLLM

I would keep the baseline in Ollama and use a separate vLLM experiment when batching or a particular supported model format matters. Do not start both servers with large allocations on the same occupied GPU.

On a Linux NVIDIA host, this small BF16 model is a baseline for the official vLLM Docker interface, not a 70B demonstration:

docker run --rm --gpus all --shm-size 4g \
  -p 127.0.0.1:8000:8000 \
  -v vllm-cache:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-0.6B \
  --served-model-name nvidia-test \
  --max-model-len 4096 --gpu-memory-utilization 0.8
tailscale serve --bg --https=8443 http://127.0.0.1:8000

My 5090 host runs Windows. A Linux Docker command is not native Windows vLLM support; the example requires a separately validated environment with CUDA access, such as WSL2 with a supported container setup.

The vLLM example requires a CUDA-capable Linux environment. On a Windows workstation, establish and test the WSL2/container path separately before adopting it.

Intel Arc: native Windows llama.cpp with SYCL

I would start with a Windows SYCL release and its matching oneAPI runtime, following the upstream backend instructions. Build and runtime versions need to agree. A generic CUDA container will not discover an Intel GPU.

In an initialized oneAPI command prompt, run sycl-ls and the release's device-list utility. Then, in a PowerShell session that inherits that runtime environment:

# From the extracted SYCL release directory; use your actual GGUF path.
$env:ONEAPI_DEVICE_SELECTOR = 'level_zero:0'
.\llama-server.exe -m 'S:\models\model-Q4_K_M.gguf' `
  --alias arc-a --host 127.0.0.1 --port 8081 `
  --split-mode none --main-gpu 0 --n-gpu-layers 99 `
  --ctx-size 4096 --parallel 1

For a second worker, use another initialized terminal, the verified second B70 selector, alias arc-b, and port 8082. Do not assume index 1 is the second B70: an integrated GPU can appear in the enumeration. Observe each physical card under separate requests.

On the Windows host, Serve can proxy 8081 on tailnet HTTPS port 8443 and 8082 on 8444. Add each printed endpoint with /v1 under Open WebUI's admin API connections. Use server-side connections so OG3090 is the API client.

Test an answer, not just a model list

For an Arc endpoint, this local PowerShell request exercises generation:

$body = @{
  model = 'arc-a'
  messages = @(@{role = 'user'; content = 'Reply with exactly LOCAL_OK'})
  max_tokens = 32
  temperature = 0
} | ConvertTo-Json -Depth 5
Invoke-RestMethod -Uri 'http://127.0.0.1:8081/v1/chat/completions' `
  -Method Post -ContentType 'application/json' -Body $body

Repeat from the UI host using the private HTTPS endpoint. Save the answer, server logs, selected device, and observed dedicated GPU memory. A successful /v1/models response only confirms discovery. Finally test a client that policy should deny.

Honest gotchas

Intel is the part I would document most carefully. My reference path is llama.cpp plus SYCL, but it would be inaccurate to claim that Ollama and vLLM have no Intel route: Ollama now documents Vulkan support, and vLLM documents Intel XPU support. Those capabilities do not establish B70 compatibility, correctness, or speed for a particular installed version.

Windows paths, WSL paths, and Docker volumes are three different views of storage. I would keep active model files near the inference process and use the Storage Spaces machine as the library, rather than silently introducing network file loading into every test. Network speed matters for copying models; it does not turn separate cards into shared memory.

An idle workstation is not an always-available server. After wake, check that the serving process and runtime environment returned. Disable unavailable UI connections deliberately; a stopped worker should produce a visible failure, not silently send the request to a cloud service.

If I were starting again, I would get one small model working on one card first, write down its exact artifact and launch command, and only then add the next endpoint. The inventory is interesting. A reproducible answer from the intended GPU is the result that matters.

What's next

The next guide will go deeper on VRAM and quantization: actual GGUF sizes, quality checks between quantizations, KV-cache growth, and measured memory at different contexts. I want to show the point where a model fits comfortably, where it merely loads, and where a bigger context turns a workable configuration into an out-of-memory error.