SAYLOR Build a compatibility plan from the GPU, kernel driver, CUDA runtime needs, Ollama version, model size, and available VRAM instead of chasing version numbers blindly.
The result you're building
A versioned local-AI compatibility plan showing GPU support, driver/runtime path, Ollama execution backend, model memory fit, context/concurrency limits, and a measured proof that inference uses the intended hardware.
Use this guide when
- Ollama falls back to CPU, crashes, runs slowly, or stops after a driver/kernel update.
- Choosing a model/context for limited VRAM on Ubuntu.
Do not use it as a substitute for
- Do not install a full CUDA toolkit just because
nvidia-smishows a CUDA compatibility version. - Do not chase latest versions without checking GPU architecture and Ollama support.
Before you change anything
- Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- GPU model/PCI ID and VRAM.
- Kernel, bound driver,
nvidia-smi, and Ollama version/logs. - Model file/quantization/parameter size and intended context.
- RAM, swap, concurrency, and measured CPU/GPU utilization.
Understand the system before fixing it
Driver support and toolkit installation are separate
Prebuilt inference software can ship required CUDA libraries; host driver compatibility matters, while a system CUDA compiler toolkit may be unnecessary.
Model fit is more than file size
Weights plus KV cache, context, batch, runtime buffers, and concurrent requests determine VRAM/RAM use.
Logs and utilization prove acceleration
A fast-looking response does not prove GPU use. Record Ollama logs, processor assignment, memory, and tokens/second.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
nvidia-smi fails | Driver/kernel | GPU binding and kernel log | Fix host driver before Ollama. |
nvidia-smi works; Ollama uses CPU | Runtime discovery | Ollama service log and environment | Unsupported GPU/runtime path, service environment, or library discovery. |
| Loads then OOM/crashes | Memory fit | Model/context/concurrency vs VRAM/RAM | Reduce model/quantization/context/concurrency; toolkit reinstall is irrelevant. |
| GPU used but slow | Offload/workload | Processor split, context, tokens/s, thermals | Partial offload, oversized context, throttling, or prompt processing dominates. |
Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
Step 01 — Establish host GPU health
Why: Inference cannot exceed a broken driver boundary.
Do: Verify GPU binding, driver communication, VRAM, kernel errors, and a stable driver package.
lspci -nnk | grep -A4 -E 'VGA|3D|Display'
nvidia-smi
journalctl -k -b | grep -iE 'nvrm|xid|nvidia'Read the result: A healthy nvidia-smi and no fresh Xid errors allow runtime testing.
Next: Record versions rather than assuming the displayed CUDA number is installed toolkit.
Step 02 — Verify Ollama service context
Why: The service user/environment differs from the interactive shell.
Do: Check version, unit configuration, environment overrides, permissions, and logs while loading a tiny model.
ollama --version
systemctl cat ollama
journalctl -u ollama --no-pager --pager-endRead the result: GPU discovery or library errors in the service log identify runtime integration.
Next: Fix the service boundary without exporting global unsafe paths.
Step 03 — Build a memory budget
Why: Random model trials cause avoidable OOM and thrash.
Do: Record model quantization/file size, required context, concurrency, free VRAM/RAM, and reserve for desktop/runtime. Start with smaller context and one request.
Read the result: If weights cannot fit, expect CPU/RAM offload and lower speed; if KV cache pushes over budget, context/concurrency is the lever.
Next: Choose the smallest model that meets the task.
Step 04 — Measure one controlled inference
Why: Subjective speed cannot distinguish CPU fallback, partial offload, or cache effects.
Do: Run the same prompt twice; record load time, prompt/eval tokens and durations, processor assignment, VRAM, CPU, and temperature.
Read the result: GPU memory/utilization rise with logs indicating GPU placement proves acceleration.
Next: Change one variable at a time.
Step 05 — Tune context and concurrency
Why: Large default context consumes memory even when prompts are short.
Do: Increase context only for a measured need; test one then multiple concurrent requests under memory and latency limits.
Read the result: Stop before swapping/OOM or unacceptable p95 latency.
Next: Persist explicit settings with model/service version.
Step 06 — Prove restart and fallback behavior
Why: A configuration that works only in the current shell is not deployable.
Do: Restart service and host if driver changed; rerun health/model/test, then intentionally test a smaller fallback model.
Read the result: Service must return to known performance without manual exports.
Next: Document upgrade and rollback versions.
Worked example
Evidence collected
nvidia-smiand logs show partial GPU offload, not driver failure.- Context is set to 32768 and two requests run concurrently.
- System RAM and swap grow sharply; GPU VRAM is full.
- A smaller quantized 7B/8B model fits and runs fully on GPU.
Decision: The bottleneck is working-set size and concurrency, not missing CUDA installation.
Actions taken
- Reduced context to the actual workload need and concurrency to one.
- Selected a quantization/model that fits with desktop reserve.
- Benchmarked tokens/second, p95 latency, VRAM, and restart behavior.
Why this example matters: Compatibility is proven by the running stack and workload budget, not by matching version strings alone.
Verify, recover, and hand off
Completion tests
- A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Host GPU/driver remain healthy under load.
- Ollama service detects and uses intended backend after restart.
- Chosen model/context/concurrency stay within VRAM/RAM reserves.
- Measured tokens/second and p95 latency meet the task target.
- A smaller fallback model and rollback version are documented.
Rollback or safe recovery
- Return to known-good Ubuntu/NVIDIA package and kernel combination.
- Restore prior Ollama unit environment/model settings.
- Reduce to smaller model/context and single concurrency to regain service before further tuning.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| GPU not visible only to service | Unit user/environment/device permission differs. | Compare service unit and logs with interactive execution. |
| OOM after longer chat | KV cache/context growth exceeds budget. | Reduce context, concurrency, or model size/quantization. |
| Performance falls over time | Thermal throttling, memory pressure, or concurrent queue. | Monitor temperature, clocks, RAM/swap, and request count. |
| Update breaks GPU use | Driver/runtime/Ollama regression. | Pin/revert one layer and reproduce with benchmark. |
Reusable handoff record
- Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- GPU/driver/kernel/Ollama/model versions.
- Memory budget and chosen model/context/concurrency.
- Service configuration and log evidence of backend.
- Benchmark prompt and performance/resource results.
- Restart, fallback, upgrade, and rollback procedure.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| context | object | Versioned environment, target, and requested outcome. |
| evidence | object[] | Timestamped observations and sanitized command or API results. |
| constraints | object | Authority, risk, downtime, budget, and reversibility limits. |
| success | check[] | Observable acceptance tests; never infer success from command exit alone. |
Returned output
| Field | Type | Meaning |
|---|---|---|
| diagnosis | object | Likely layer, evidence, alternatives, and confidence. |
| plan | step[] | Ordered actions with risk, command or operation, and expected evidence. |
| verification | check[] | Pass/fail checks that prove the requested outcome. |
| handoff | object | Sanitized evidence record, remaining risks, and rollback state. |
Agent refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.
Official reference starting points