SAYLOR Design a local Ubuntu AI deployment around actual CPU, RAM, GPU VRAM, model size, context, concurrency, privacy, and recovery needs.
The result you're building
A measured Ubuntu local-AI service sized to CPU/RAM/GPU/VRAM and workload, bound to intended interfaces, protected from unauthorized access, versioned, monitored, backed up where needed, and recoverable with a smaller fallback model.
Use this guide when
- Keeping sensitive prompts/data local or avoiding per-call cloud cost.
- Choosing Ollama/model/context/concurrency for actual hardware.
Do not use it as a substitute for
- Do not call a system private if models, prompts, telemetry, browser tools, or update downloads still send data externally.
- Do not expose an unauthenticated model API to LAN/internet.
Before you change anything
- Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- CPU, RAM, GPU/VRAM, disk, Ubuntu/kernel/driver.
- Task quality target, model class, context, concurrency, and latency.
- Network/data-flow/privacy boundary and allowed external services.
- Storage, updates, logs, users, backup, and threat model.
Understand the system before fixing it
Privacy is an end-to-end data-flow property
Local inference does not make web retrieval, plugins, logs, analytics, or backups local.
Memory budget drives model choice
Weights, KV cache, runtime buffers, desktop reserve, and concurrency determine fit.
Quality must be tested on your tasks
Parameter count and benchmarks do not prove acceptable coding/research/extraction behavior for your workload.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Model will not load/OOM | Capacity | Weights/context/concurrency vs VRAM/RAM | Smaller/quantized model or lower context/concurrency. |
| Runs on CPU unexpectedly | GPU runtime | Driver, service log, processor assignment | Fix backend discovery before tuning model. |
| Fast but poor result | Model/task | Representative eval with expected outcomes | Choose better-fit model/prompt/RAG; speed is not success. |
| Other machines can connect | Exposure | Listener/firewall/auth test | Bind loopback or secure proxy; do not rely on obscurity. |
Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
Step 01 — Map privacy and workload
Why: Hardware sizing without task/data boundary is meaningless.
Do: List input/output data classes, external flows, retention, users, task cases, quality/latency/concurrency targets.
Read the result: Any external dependency becomes explicit exception.
Next: Create representative eval set.
Step 02 — Inventory hardware and reserve
Why: Using all RAM/VRAM destabilizes desktop/service.
Do: Record usable VRAM/RAM/disk, driver health, and reserve; estimate weights plus context/KV and concurrent requests.
Read the result: Select one small baseline model that safely fits.
Next: Avoid swap as performance plan.
Step 03 — Install/version service safely
Why: Ad hoc shell launches drift and expose interfaces.
Do: Use supported package/service, pin model digest/tag and configuration, run unprivileged, bind loopback, set storage and resource limits.
Read the result: Service restarts with same version/settings.
Next: Record update/rollback.
Step 04 — Benchmark representative tasks
Why: Synthetic speed alone misses quality.
Do: Run fixed prompts/documents/code cases; measure correctness, refusal, hallucination, tokens/s, load, p95 latency, RAM/VRAM, and power/thermal behavior.
Read the result: Choose model/settings by minimum quality under resource target.
Next: Compare smaller fallback.
Step 05 — Secure interfaces and data
Why: Local APIs are often unauthenticated.
Do: Firewall loopback-only or put authenticated TLS proxy for approved clients; restrict model files/logs, disable unnecessary telemetry/integrations, and sanitize retention.
Read the result: Unauthorized LAN/user access fails.
Next: Test tool/plugin egress separately.
Step 06 — Operate and recover
Why: Models/services consume disk and can fail after updates.
Do: Monitor health/resource/disk/errors, back up configs/eval/RAG indexes as needed, test model rollback and service restart, and maintain fallback.
Read the result: Update only after eval and privacy diff.
Next: Document restore.
Worked example
Evidence collected
- Model partially offloads to CPU.
- KV/context pushes RAM into swap.
- Latency misses target and desktop stutters.
- An 8B quantized model at 8k passes most task evals and fits GPU.
Decision: Requested configuration exceeds practical memory/latency budget; toolkit reinstall will not change fit.
Actions taken
- Selected model/context/concurrency from measured tasks.
- Bound service to loopback and set limits.
- Kept 14B as optional slower batch profile and 8B as default/fallback.
Why this example matters: The end result is a stable service matched to real work, not the largest model that can barely start.
Verify, recover, and hand off
Completion tests
- A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Representative task eval meets minimum quality.
- p95 latency and RAM/VRAM stay within reserve under allowed concurrency.
- Service uses intended backend and restarts reproducibly.
- Unauthorized network/user access is denied.
- Data-flow audit shows only approved external flows/retention.
- Fallback model and rollback are tested.
Rollback or safe recovery
- Restore prior service config/model version and restart.
- Switch to smaller fallback model on OOM/performance regression.
- Close proxy/firewall exposure immediately on auth/privacy incident.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Model list disappears after restart | Service storage/user/path differs. | Inspect unit environment and model directory ownership. |
| LAN cannot connect intentionally | Loopback bind/firewall/proxy not configured. | Add authenticated proxy and narrow rule; never bind public without auth. |
| Quality falls after update | Model/digest/prompt/runtime changed. | Run versioned eval and revert bundle. |
| Disk fills | Model/cache/log retention unbounded. | Inventory and remove only unused verified artifacts; add alerts. |
Reusable handoff record
- Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- Privacy/data-flow and task requirements.
- Hardware/memory/concurrency budget.
- Pinned service/model/config and network policy.
- Quality/performance/resource evaluation.
- Monitoring, backup, fallback, update, and rollback runbook.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| context | object | Versioned environment, target, and requested outcome. |
| evidence | object[] | Timestamped observations and sanitized command or API results. |
| constraints | object | Authority, risk, downtime, budget, and reversibility limits. |
| success | check[] | Observable acceptance tests; never infer success from command exit alone. |
Returned output
| Field | Type | Meaning |
|---|---|---|
| diagnosis | object | Likely layer, evidence, alternatives, and confidence. |
| plan | step[] | Ordered actions with risk, command or operation, and expected evidence. |
| verification | check[] | Pass/fail checks that prove the requested outcome. |
| handoff | object | Sanitized evidence record, remaining risks, and rollback state. |
Agent refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.
Official reference starting points