SAYLOR The result you are building
Finished result: A measured Ubuntu local-AI service sized to CPU/RAM/GPU/VRAM and workload, bound to intended interfaces, protected from unauthorized access, versioned, monitored, backed up where needed, and recoverable with a smaller fallback model.
Use this guide when
- Keeping sensitive prompts/data local or avoiding per-call cloud cost.
- Choosing Ollama/model/context/concurrency for actual hardware.
Do not use it as a substitute for
- Do not call a system private if models, prompts, telemetry, browser tools, or update downloads still send data externally.
- Do not expose an unauthenticated model API to LAN/internet.
Before you change anything
- CPU, RAM, GPU/VRAM, disk, Ubuntu/kernel/driver.
- Task quality target, model class, context, concurrency, and latency.
- Network/data-flow/privacy boundary and allowed external services.
- Storage, updates, logs, users, backup, and threat model.
Understand the system before fixing it
Privacy is an end-to-end data-flow property. Local inference does not make web retrieval, plugins, logs, analytics, or backups local.
Memory budget drives model choice. Weights, KV cache, runtime buffers, desktop reserve, and concurrency determine fit.
Quality must be tested on your tasks. Parameter count and benchmarks do not prove acceptable coding/research/extraction behavior for your workload.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Model will not load/OOM | Capacity | Weights/context/concurrency vs VRAM/RAM | Smaller/quantized model or lower context/concurrency. |
| Runs on CPU unexpectedly | GPU runtime | Driver, service log, processor assignment | Fix backend discovery before tuning model. |
| Fast but poor result | Model/task | Representative eval with expected outcomes | Choose better-fit model/prompt/RAG; speed is not success. |
| Other machines can connect | Exposure | Listener/firewall/auth test | Bind loopback or secure proxy; do not rely on obscurity. |
Step-by-step procedure
01 Map privacy and workload
Why: Hardware sizing without task/data boundary is meaningless. Do: List input/output data classes, external flows, retention, users, task cases, quality/latency/concurrency targets. Read the result: Any external dependency becomes explicit exception. Next: Create representative eval set.
02 Inventory hardware and reserve
Why: Using all RAM/VRAM destabilizes desktop/service. Do: Record usable VRAM/RAM/disk, driver health, and reserve; estimate weights plus context/KV and concurrent requests. Read the result: Select one small baseline model that safely fits. Next: Avoid swap as performance plan.
03 Install/version service safely
Why: Ad hoc shell launches drift and expose interfaces. Do: Use supported package/service, pin model digest/tag and configuration, run unprivileged, bind loopback, set storage and resource limits. Read the result: Service restarts with same version/settings. Next: Record update/rollback.
04 Benchmark representative tasks
Why: Synthetic speed alone misses quality. Do: Run fixed prompts/documents/code cases; measure correctness, refusal, hallucination, tokens/s, load, p95 latency, RAM/VRAM, and power/thermal behavior. Read the result: Choose model/settings by minimum quality under resource target. Next: Compare smaller fallback.
05 Secure interfaces and data
Why: Local APIs are often unauthenticated. Do: Firewall loopback-only or put authenticated TLS proxy for approved clients; restrict model files/logs, disable unnecessary telemetry/integrations, and sanitize retention. Read the result: Unauthorized LAN/user access fails. Next: Test tool/plugin egress separately.
06 Operate and recover
Why: Models/services consume disk and can fail after updates. Do: Monitor health/resource/disk/errors, back up configs/eval/RAG indexes as needed, test model rollback and service restart, and maintain fallback. Read the result: Update only after eval and privacy diff. Next: Document restore.
Worked example
Starting problem: An 8 GB GPU user wants a 14B model at 32k context with two simultaneous chats.
Evidence collected
- Model partially offloads to CPU.
- KV/context pushes RAM into swap.
- Latency misses target and desktop stutters.
- An 8B quantized model at 8k passes most task evals and fits GPU.
Decision: Requested configuration exceeds practical memory/latency budget; toolkit reinstall will not change fit.
Actions taken
- Selected model/context/concurrency from measured tasks.
- Bound service to loopback and set limits.
- Kept 14B as optional slower batch profile and 8B as default/fallback.
Proof of completion: Default profile meets task threshold and p95 latency without swap; GPU use/logs confirmed; unauthorized LAN request fails; restart/rollback pass.
Why this example matters: The end result is a stable service matched to real work, not the largest model that can barely start.
Verify, recover, and hand off
Completion tests
- Representative task eval meets minimum quality.
- p95 latency and RAM/VRAM stay within reserve under allowed concurrency.
- Service uses intended backend and restarts reproducibly.
- Unauthorized network/user access is denied.
- Data-flow audit shows only approved external flows/retention.
- Fallback model and rollback are tested.
Rollback or safe recovery
- Restore prior service config/model version and restart.
- Switch to smaller fallback model on OOM/performance regression.
- Close proxy/firewall exposure immediately on auth/privacy incident.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Model list disappears after restart | Service storage/user/path differs | Inspect unit environment and model directory ownership. |
| LAN cannot connect intentionally | Loopback bind/firewall/proxy not configured | Add authenticated proxy and narrow rule; never bind public without auth. |
| Quality falls after update | Model/digest/prompt/runtime changed | Run versioned eval and revert bundle. |
| Disk fills | Model/cache/log retention unbounded | Inventory and remove only unused verified artifacts; add alerts. |
Reusable handoff record
- Privacy/data-flow and task requirements.
- Hardware/memory/concurrency budget.
- Pinned service/model/config and network policy.
- Quality/performance/resource evaluation.
- Monitoring, backup, fallback, update, and rollback runbook.
For agents
This guide also defines a structured diagnose/propose/execute contract for building an agent-facing tool around this workflow (a design reference, not a live endpoint on this site today):
Required inputs: context, evidence, constraints, success. Returned output: diagnosis, plan, verification, handoff.
Refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: score confidence from the number and quality of independent observations, not from how familiar the error looks.
References
- https://docs.ollama.com/linux
- https://docs.ollama.com/gpu
- https://docs.ollama.com/context-length
- https://docs.ollama.com/troubleshooting