Running LLMs Locally with Ollama

AI-generated content

Running large language models locally gives you privacy, unlimited usage, and complete control. Ollama makes it simpler than ever.

Why Run Locally?

• Privacy: Your data never leaves your machine
• No rate limits: Use as much as you want
• Offline capable: Works without internet
• Customizable: Fine-tune models for your specific needs
• Cost effective: After hardware purchase, inference is free

Hardware Requirements

Minimum (7B models)

• CPU: 4+ cores
• RAM: 8 GB
• Storage: 5 GB per model
• GPU: Not required (but helps)

Recommended (13B-70B models)

• CPU: 8+ cores
• RAM: 16-32 GB
• GPU: NVIDIA RTX 3060+ (12GB+ VRAM) or Apple M-series
• Storage: 10-50 GB depending on model

For Large Models (70B+)

• GPU: Multiple high-VRAM cards or Apple M2 Ultra
• RAM: 64+ GB (for CPU offloading)
• Storage: 40-100 GB

Installation

Linux

curl -fsSL https://ollama.com/install.sh | sh

macOS

brew install ollama

Windows

Download from ollama.com

Docker

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 ollama/ollama

Getting Started

Pull a Model

# Start the server
ollama serve

# In another terminal:
ollama pull llama3.1:8b
ollama pull mistral:7b
ollama pull codellama:13b

Chat with a Model

ollama run llama3.1:8b

API Usage

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Explain quantum computing in simple terms",
  "stream": false
}'

Python Integration

import requests

response = requests.post("http://localhost:11434/api/generate", json={
    "model": "llama3.1:8b",
    "prompt": "Write a haiku about debugging",
    "stream": False
})

print(response.json()["response"])

Model Selection Guide

For General Use

For Code

For Advanced Reasoning

Quantization Explained

Models come in different precision levels:

• FP16: Full precision, best quality, largest size
• Q8_0: 8-bit, minimal quality loss, ~50% size
• Q5_K_M: 5-bit mixed, good balance, ~33% size
• Q4_K_M: 4-bit mixed, best speed, ~27% size
• Q3_K_S: 3-bit, smallest, noticeable quality drop

Most users prefer Q4_K_M or Q5_K_M for the best quality/speed tradeoff.

# Pull a specific quantization
ollama pull llama3.1:8b-q4_K_M

Advanced Features

Ollama Files (Custom Models)

Create a Modelfile to customize models:

FROM llama3.1:8b

SYSTEM """You are a helpful coding assistant. 
Always provide code examples when explaining concepts."""

PARAMETER temperature 0.7
PARAMETER num_ctx 4096
PARAMETER stop "</s>"

ENV OLLAMA_CONTEXT_LENGTH 8192

ollama create my-coder -f Modelfile
ollama run my-coder "Explain async/await"

RAG with Ollama

Combine Ollama with vector databases for document question answering:

# Use ollama with LangChain
pip install langchain ollama

# Python RAG example
from langchain_community.llms import Ollama
from langchain_community.vectorstores import Chroma

llm = Ollama(model="llama3.1:8b")
# Load documents, embed, query...

Multi-Modal Models

Some models support images:

ollama pull llava:7b

curl http://localhost:11434/api/generate -d '{
  "model": "llava:7b",
  "prompt": "Describe this image",
  "images": ["base64_encoded_image"]
}'

Performance Tips

1. Use GPU offloading: OLLAMA_GPU_MEMORY=0.9 (90% of VRAM)
2. Adjust context length: Longer = more accurate but slower
3. Use batching: Process multiple prompts simultaneously
4. Enable NUMA: numactl --interleave=all ollama serve
5. Monitor GPU: nvtop or nvidia-smi for real-time stats

Troubleshooting

Out of memory

# Reduce VRAM usage
OLLAMA_GPU_MEMORY=0.7 ollama serve

# Use more CPU offloading
OLLAMA_NUM_PARALLEL=1 ollama serve

Slow inference

# Check GPU is being used
ollama ps  # Shows active models

# Verify CUDA is working
nvidia-smi

Model not downloading

# Check internet connection
# Use proxy if behind firewall
http_proxy=http://your-proxy:8080 ollama pull llama3.1:8b

Next Steps

• Explore ollama.ai/library for more models
• Try Open WebUI for a better interface
• Learn about LoRA fine-tuning to customize models
• Set up Dify for workflow automation

This post was generated with AI assistance. Review and customize it to match your experience.