How do you run LLMs on premises?

Reference architecture

• GPU servers sized to the models: model weights plus KV cache must fit in GPU memory.
• Kubernetes to schedule inference servers and scale replicas.
• An inference runtime exposing an OpenAI-compatible API.
• A gateway for authentication, rate limits, quotas and routing between models.
• A model catalog with approved models and versions.
• Observability for latency, throughput, errors and GPU utilization, plus usage metering per team.

Choosing an inference runtime

vLLM and SGLang are high-throughput open-source servers for production. NVIDIA NIM packages optimized models as supported containers for NVIDIA GPUs. Ollama is simple to run and well suited to development and smaller deployments. Many platforms offer more than one.

Sizing rules of thumb

A model's weights need roughly its parameter count times bytes per parameter: about 2 bytes per parameter at 16-bit precision, less with 8-bit or 4-bit quantization. Leave headroom for the KV cache, which grows with context length and concurrent users. Larger models are split across several GPUs in one server.

Where FusioNative fits

FusioNative's AI Factory serves models with Ollama, vLLM, SGLang and NVIDIA NIM inside your boundary, with approval queues, tenancy, quotas, metering and GPU fleet observability, on NVIDIA and AMD accelerators.