SAYLOR Evaluate and contain the operational, security, privacy, financial, legal, and exit risk of a provider, API, model, data source, plugin, or managed service.
The result you're building
A dependency register and control plan that classifies criticality, verifies authority and data flow, measures reliability/cost/change behavior, tests failure and exit, and prevents one vendor from silently controlling the product.
Use this guide when
- A product depends on third-party APIs, cloud, AI models, data, payments, plugins, or contractors.
- Provider pricing, terms, versions, or reliability may change.
- Customer commitments rely on vendor behavior.
Do not use it as a substitute for
- Relying on a logo, sales claim, or status page as complete diligence.
- Calling a backup provider a failover before data, auth, schemas, semantics, capacity, and switching are tested.
Before you change anything
- Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
- Vendor/service owner, contract/terms/version, support, billing identity, and renewal/termination.
- Data fields/classes, regions, retention, training/use, subprocessors, credentials, and deletion/export.
- API/schema/version/deprecation, quotas, prices, latency, errors, status, and incident history.
- Architecture coupling, critical journeys, cache/queue, failover, reconciliation, and manual workaround.
- Exit export, migration, deletion proof, credential revocation, cost, RTO/RPO, and test evidence.
Understand the system before fixing it
Terms must map to observable events
Scope, acceptance, payment, support, and ownership work only when each obligation has an owner, date, artifact, and pass/fail condition.
Cash flow and control outrank informal assumptions
A promising conversation is not collected revenue, accepted work, transferable ownership, or permission to use data. Record the actual state.
Criticality follows substitutability and consequence
A small cheap API can be critical if one missing response blocks payment, identity, deployment, or customer commitments.
Failover can create correctness risk
Two vendors may return different identifiers, units, freshness, ranking, or semantics. Normalize and reconcile before automatic switching.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Provider 200 response breaks parser | Schema/change | Validate schema and compare version/changelog | Contract changed without compatible handling. |
| Costs jump unexpectedly | Pricing/usage | Reconcile work units, retries, tiers, and new prices | No spend cap, change alert, or unit-cost model exists. |
| Fallback disagrees with primary | Semantics/data | Compare source, time, units, IDs, and methodology | Providers are not interchangeable despite similar fields. |
| Cannot delete customer data | Lifecycle/contract | Run export/deletion and obtain evidence | Exit or privacy capability was assumed, not tested. |
| Vendor outage breaches client promise | Dependency/S LA | Map vendor limits to customer SLO and workaround | Downstream commitment exceeds upstream capability without buffer. |
Step-by-step procedure
Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.
Step 01 — Register owner, purpose, and criticality
Why: A precise boundary prevents a plausible fix from solving the wrong problem.
Do: Identify business owner, technical owner, service/result, critical journeys, data, alternatives, switching time, maximum outage/loss, and customer commitments.
Read the result: Each dependency has a consequence-based tier and review date.
Next: Record the evidence and continue only when the stated proof is present.
Step 02 — Map data, identity, and authority
Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.
Do: Document inputs/outputs, personal/client/secret/financial classes, regions, retention, provider use/training, subprocessors, credentials, scopes, and deletion/export.
Read the result: No data or privilege is sent beyond authorized purpose.
Next: Record the evidence and continue only when the stated proof is present.
Step 03 — Verify technical contract
Why: Inconsistent inputs create false differences and make later comparisons unreliable.
Do: Pin versions, schemas, units, IDs, errors, quotas, rate limits, latency, freshness, support, deprecation, and status/incident channels. Add contract tests.
Read the result: Breaking change or quota risk is detected before production failure.
Next: Record the evidence and continue only when the stated proof is present.
Step 04 — Model economics and capacity
Why: A decisive test reduces trial-and-error and limits unnecessary change.
Do: Calculate full cost by accepted result including retries, egress, minimums, support, failure, growth, and price-change sensitivity. Set spend and quota alerts.
Read the result: Usage growth and provider price changes remain within approved exposure.
Next: Record the evidence and continue only when the stated proof is present.
Step 05 — Design failure containment
Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.
Do: Use timeouts, backpressure, cache/stale policy, queue, circuit state, manual workaround, and clear user errors. Do not retry storms or silently lower correctness.
Read the result: Provider slowdown cannot exhaust the whole product.
Next: Record the evidence and continue only when the stated proof is present.
Step 06 — Test alternatives and reconciliation
Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.
Do: Exercise failover with production-like data, auth, volume, semantics, freshness, and return; compare results and record switching/rollback.
Read the result: Fallback meets the required outcome or is labeled a manual/degraded option.
Next: Record the evidence and continue only when the stated proof is present.
Step 07 — Prove exit and lifecycle
Why: A result is not complete until it remains observable and repeatable after the immediate fix.
Do: Export data/config, migrate a canary, revoke credentials, verify provider deletion, recover artifacts/keys, calculate time/cost, and document contract renewal/termination.
Read the result: The business can leave within declared time and preserve required service/data.
Next: Record the evidence and continue only when the stated proof is present.
Operational worksheet
Evidence record
- Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
- Vendor/service owner, contract/terms/version, support, billing identity, and renewal/termination.
- Data fields/classes, regions, retention, training/use, subprocessors, credentials, and deletion/export.
- API/schema/version/deprecation, quotas, prices, latency, errors, status, and incident history.
- Architecture coupling, critical journeys, cache/queue, failover, reconciliation, and manual workaround.
- Exit export, migration, deletion proof, credential revocation, cost, RTO/RPO, and test evidence.
Acceptance scoreboard
- Every dependency has owner, criticality, consequence, substitutability, and review trigger.
- Data, identity, region, retention, provider use, subprocessors, deletion, and credentials are approved.
- Version, schema, units, IDs, errors, quotas, freshness, deprecation, and support are tested.
- Cost by accepted result, growth, price, quota, and spend exposure are modeled.
- Timeout, backpressure, cache/queue, incident, fallback, and reconciliation tests pass.
- Exit export, migration, revocation, deletion, recovery, time, cost, and contract lifecycle are proven.
Minimum handoff record
- Versioned vendor and api dependency risk scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Worked example
Evidence collected
- All subscriptions use one provider account.
- No raw event replay or second provider normalization exists.
- Customer alerts promise near-real-time delivery.
- Retry storms consume the remaining quota.
Decision: The dependency controls cost and reliability beyond the promise. Add admission/retry control, durable cursor/replay, measured alternative provider, and revise the SLO or price.
Actions taken
- Mapped provider quota and launch burst demand.
- Added capped backoff and durable event checkpoints.
- Tested a second provider against the same signatures/slots.
- Added cost and lag alerts plus degraded-mode disclosure.
Why this example matters: The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.
Verify, recover, and hand off
Completion tests
- A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
- Every dependency has owner, criticality, consequence, substitutability, and review trigger.
- Data, identity, region, retention, provider use, subprocessors, deletion, and credentials are approved.
- Version, schema, units, IDs, errors, quotas, freshness, deprecation, and support are tested.
- Cost by accepted result, growth, price, quota, and spend exposure are modeled.
- Timeout, backpressure, cache/queue, incident, fallback, and reconciliation tests pass.
- Exit export, migration, revocation, deletion, recovery, time, cost, and contract lifecycle are proven.
Rollback or safe recovery
- Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
- Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
- Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
- Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Provider 200 response breaks parser | Contract changed without compatible handling. | Validate schema and compare version/changelog |
| Costs jump unexpectedly | No spend cap, change alert, or unit-cost model exists. | Reconcile work units, retries, tiers, and new prices |
| Fallback disagrees with primary | Providers are not interchangeable despite similar fields. | Compare source, time, units, IDs, and methodology |
| Cannot delete customer data | Exit or privacy capability was assumed, not tested. | Run export/deletion and obtain evidence |
Reusable handoff record
- Versioned vendor and api dependency risk scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| target | object | Versioned environment, resource, identity, or workflow being evaluated. |
| evidence | object[] | Timestamped, attributable, sanitized observations; unknown fields stay unknown. |
| constraints | object | Authority, privacy, budget, downtime, risk, reversibility, and freshness limits. |
| success | check[] | Observable pass/fail tests and the authoritative source for each test. |
Agent refusal and escalation rules
- Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
- Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
- Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
- Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.
Confidence rule: Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low confidence; a controlled test that isolates the layer and passes verification can support high confidence.
Official reference starting points