The result you are building

Finished Result:

A versioned local-AI compatibility plan showing GPU support, driver/runtime path, Ollama execution backend, model memory fit, context/concurrency limits, and a measured proof that inference uses the intended hardware.

Use this guide when

Ollama falls back to CPU, crashes, runs slowly, or stops after a driver/kernel update.
Choosing a model/context for limited VRAM on Ubuntu.

Do not use it as a substitute for

Do not install a full CUDA toolkit just because nvidia-smi shows a CUDA compatibility version.
Do not chase latest versions without checking GPU architecture and Ollama support.

Before you change anything

Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.

GPU model/PCI ID and VRAM.
Kernel, bound driver, nvidia-smi, and Ollama version/logs.
Model file/quantization/parameter size and intended context.
RAM, swap, concurrency, and measured CPU/GPU utilization.

Stop Before Proceeding:

Stop when the kernel logs GPU Xid faults, hardware instability, or repeated system crashes. Stabilize driver/hardware with a known-good workload before blaming model configuration.

Understand the system before fixing it

Driver support and toolkit installation are separate Prebuilt inference software can ship required CUDA libraries; host driver compatibility matters, while a system CUDA compiler toolkit may be unnecessary.

Model fit is more than file size Weights plus KV cache, context, batch, runtime buffers, and concurrent requests determine VRAM/RAM use.

Logs and utilization prove acceleration A fast-looking response does not prove GPU use. Record Ollama logs, processor assignment, memory, and tokens/second.

Evidence-to-decision map

  Start with the row that most closely matches the evidence. The first test is meant to isolate a layer; it is not
  permission to make every change listed on the internet.

    Evidence                  Likely layer       First decisive check           What the result means

    `nvidia-smi` fails        Driver/kernel      GPU binding and kernel log     Fix host driver before Ollama.

    `nvidia-smi` works;       Runtime            Ollama service log and         Unsupported GPU/runtime path, service
    Ollama uses CPU           discovery          environment                    environment, or library discovery.

    Loads then                Memory fit         Model/context/concurrency vs   Reduce
    OOM/crashes                                  VRAM/RAM                       model/quantization/context/concurrency;
                                                                                toolkit reinstall is irrelevant.

    GPU used but slow         Offload/workloa    Processor split, context,      Partial offload, oversized context, throttling, or
                              d                  tokens/s, thermals             prompt processing dominates.

Step-by-step procedure

Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.

01 Establish host GPU health Why: Inference cannot exceed a broken driver boundary.

Do: Verify GPU binding, driver communication, VRAM, kernel errors, and a stable driver package.

Read the result: A healthy nvidia-smi and no fresh Xid errors allow runtime testing.

Next: Record versions rather than assuming the displayed CUDA number is installed toolkit.

02 Verify Ollama service context Why: The service user/environment differs from the interactive shell.

Do: Check version, unit configuration, environment overrides, permissions, and logs while loading a tiny model.

ollama --version systemctl cat ollama journalctl -u ollama --no-pager --pager-end

Read the result: GPU discovery or library errors in the service log identify runtime integration.

Next: Fix the service boundary without exporting global unsafe paths.

03 Build a memory budget Why: Random model trials cause avoidable OOM and thrash.

Do: Record model quantization/file size, required context, concurrency, free VRAM/RAM, and reserve for desktop/runtime. Start with smaller context and one request.

Read the result: If weights cannot fit, expect CPU/RAM offload and lower speed; if KV cache pushes over budget, context/concurrency is the lever.

Next: Choose the smallest model that meets the task.

04 Measure one controlled inference Why: Subjective speed cannot distinguish CPU fallback, partial offload, or cache effects.

Do: Run the same prompt twice; record load time, prompt/eval tokens and durations, processor assignment, VRAM, CPU, and temperature.

Read the result: GPU memory/utilization rise with logs indicating GPU placement proves acceleration.

Next: Change one variable at a time.

05 Tune context and concurrency Why: Large default context consumes memory even when prompts are short.

Do: Increase context only for a measured need; test one then multiple concurrent requests under memory and latency limits.

Read the result: Stop before swapping/OOM or unacceptable p95 latency.

Next: Persist explicit settings with model/service version.

06 Prove restart and fallback behavior Why: A configuration that works only in the current shell is not deployable.

Do: Restart service and host if driver changed; rerun health/model/test, then intentionally test a smaller fallback model.

Read the result: Service must return to known performance without manual exports.

Next: Document upgrade and rollback versions.

Worked example

Starting Problem:

An 8 GB RTX 4060 loads a 14B model but responses crawl and the desktop stutters.

Evidence collected

nvidia-smi and logs show partial GPU offload, not driver failure.
Context is set to 32768 and two requests run concurrently.
System RAM and swap grow sharply; GPU VRAM is full.
A smaller quantized 7B/8B model fits and runs fully on GPU.

Decision The bottleneck is working-set size and concurrency, not missing CUDA installation.

Actions taken

Reduced context to the actual workload need and concurrency to one.
Selected a quantization/model that fits with desktop reserve.
Benchmarked tokens/second, p95 latency, VRAM, and restart behavior.

Proof Of Completion:

Ollama logs show intended GPU placement; no swap thrash/OOM; measured latency meets target after service restart.

Why this example matters Compatibility is proven by the running stack and workload budget, not by matching version strings alone.

Verify, recover, and hand off

Completion tests A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.

Host GPU/driver remain healthy under load.
Ollama service detects and uses intended backend after restart.
Chosen model/context/concurrency stay within VRAM/RAM reserves.
Measured tokens/second and p95 latency meet the task target.
A smaller fallback model and rollback version are documented.

Rollback or safe recovery

Return to known-good Ubuntu/NVIDIA package and kernel combination.
Restore prior Ollama unit environment/model settings.
Reduce to smaller model/context and single concurrency to regain service before further tuning.

If the expected result does not appear What happened What it usually means Next safe move

GPU not visible only to service Unit user/environment/device Compare service unit and logs with interactive permission differs. execution.

OOM after longer chat KV cache/context growth exceeds Reduce context, concurrency, or model budget. size/quantization.

Performance falls over time Thermal throttling, memory pressure, Monitor temperature, clocks, RAM/swap, and or concurrent queue. request count.

Update breaks GPU use Driver/runtime/Ollama regression. Pin/revert one layer and reproduce with benchmark.

Reusable handoff record Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.

GPU/driver/kernel/Ollama/model versions.
Memory budget and chosen model/context/concurrency.
Service configuration and log evidence of backend.
Benchmark prompt and performance/resource results.
Restart, fallback, upgrade, and rollback procedure.

Agent delivery contract

  Required inputs
    Field                        Type               Requirement

    context                      object             Versioned environment, target, and requested outcome.

    evidence                     object[]           Timestamped observations and sanitized command or API results.

    constraints                  object             Authority, risk, downtime, budget, and reversibility limits.

    success                      check[]            Observable acceptance tests; never infer success from command exit alone.

  Returned output
    Field                        Type               Meaning

    diagnosis                    object             Likely layer, evidence, alternatives, and confidence.

    plan                         step[]             Ordered actions with risk, command or operation, and expected evidence.

    verification                 check[]            Pass/fail checks that prove the requested outcome.

    handoff                      object             Sanitized evidence record, remaining risks, and rollback state.

  Agent refusal and escalation rules
  • Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
  • Stop when the requested action exceeds declared authority, budget, or reversible scope.
  • Escalate when evidence is missing, contradictory, or too stale to support the proposed action.

  Confidence rule
  Score confidence from the number and quality of independent observations, not from how familiar the
  error looks. Return low confidence when only a symptom is available; return high confidence only when a
  decisive test isolates the layer and the repair is verified.

Official reference starting points

https://docs.ollama.com/gpu
https://docs.ollama.com/troubleshooting
https://docs.ollama.com/context-length
https://docs.nvidia.com/deploy/cuda-compatibility/

CUDA, Driver and Ollama Compatibility Guide