SAYLOR Rotate passwords, API keys, tokens, certificates, signing keys, and wallet-adjacent credentials without exposing values or breaking every dependent service at once.
The result you're building
A secret inventory and phased rotation plan that identifies owners and consumers, creates scoped replacement credentials, supports overlap where safe, updates and verifies dependencies, revokes old access, and proves no stale copy remains active.
Use this guide when
- A credential is expiring, overprivileged, shared, leaked, or due for routine rotation.
- Several services depend on one secret.
- You need evidence of revocation and service continuity.
Do not use it as a substitute for
- Pasting secret values into tickets, chat, logs, scripts, or PDFs.
- Revoking the only credential before replacement access and recovery are tested.
Before you change anything
- Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
- Secret identifier/type, owner, issuer, purpose, privileges, scope, environment, and expiry.
- Consumers, delivery method, storage locations, versions, and last-use evidence.
- Rotation/revocation API, overlap support, propagation delay, and emergency access.
- Service health, auth errors, audit logs, and canary identities.
- Leak containment, stale-secret, rollback, revocation, and recovery test evidence.
Understand the system before fixing it
Observe before mutating
Capture state, logs, versions, ownership, and dependency health before restarting, reinstalling, deleting, or rotating anything.
Recovery must be exercised
A backup, rollback command, or spare endpoint is only a claim until a controlled restore or failover test proves it works.
Rotate identity and privilege, not just bytes
A new secret with the same shared owner and excessive scope preserves the original control weakness.
Overlap and compromise require different sequencing
Routine rotation can use dual validity; compromise may require immediate revoke, isolation, and accepting controlled downtime.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| New key works in one service only | Consumer inventory | Compare secret version and auth errors across consumers | A hidden consumer, cache, or deployment still uses the old credential. |
| Old key remains accepted | Revocation/propagation | Test old credential through issuer and each endpoint | Revocation was not completed or edge caches lag. |
| Rotation causes outage | Sequencing | Inspect create-distribute-reload -verify-revoke order | Old credential was removed before every consumer adopted replacement. |
| Secret appears in logs | Handling | Scan logs/artifacts and rotate again if exposed | Delivery or error path leaks sensitive values. |
| No owner can rotate | Ownership/recovery | Verify issuer account, break-glass, and recovery contacts | Credential lifecycle depends on an unavailable personal account. |
Step-by-step procedure
Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.
Step 01 — Inventory by identifier, never value
Why: A precise boundary prevents a plausible fix from solving the wrong problem.
Do: Record secret ID, type, issuer, owner, purpose, scope, privilege, consumers, storage/delivery, version, expiry, and emergency recovery without copying raw material.
Read the result: Every active credential and consumer has an accountable owner.
Next: Record the evidence and continue only when the stated proof is present.
Step 02 — Choose routine or compromise procedure
Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.
Do: Assess exposure evidence, blast radius, data/actions at risk, and required containment. Define overlap, downtime, notification, and incident handling accordingly.
Read the result: Sequence matches actual compromise risk.
Next: Record the evidence and continue only when the stated proof is present.
Step 03 — Create a least-privileged replacement
Why: Inconsistent inputs create false differences and make later comparisons unreliable.
Do: Use a new identity or scoped credential where possible, short lifetime, correct audience/network/environment, and protected secret manager delivery.
Read the result: Replacement has only required access and a documented expiry/rotation path.
Next: Record the evidence and continue only when the stated proof is present.
Step 04 — Update consumers in controlled order
Why: A decisive test reduces trial-and-error and limits unnecessary change.
Do: Change test/canary first, then services/jobs/workers/CI/providers with versioned config, reload behavior, and health checks. Avoid echoing values.
Read the result: Each consumer authenticates with the new version and no sensitive log output.
Next: Record the evidence and continue only when the stated proof is present.
Step 05 — Verify business and audit behavior
Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.
Do: Test critical journeys, scheduled jobs, callbacks, signatures, permissions, and audit attribution; monitor old/new last use during overlap.
Read the result: All required operations succeed and unexpected privileges remain denied.
Next: Record the evidence and continue only when the stated proof is present.
Step 06 — Revoke and search for stale use
Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.
Do: Disable/delete old credential at issuer, monitor auth failures and last-use evidence, scan approved stores/configs/repos/logs for identifier or accidental value exposure.
Read the result: Old credential is rejected and no authorized consumer attempts it.
Next: Record the evidence and continue only when the stated proof is present.
Step 07 — Close ownership and next rotation
Why: A result is not complete until it remains observable and repeatable after the immediate fix.
Do: Update inventory, recovery contacts, expiry alerts, rotation automation, incident record, and evidence. Destroy temporary local copies securely.
Read the result: A different authorized operator can perform the next rotation from the runbook.
Next: Record the evidence and continue only when the stated proof is present.
Operational worksheet
Evidence record
- Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
- Secret identifier/type, owner, issuer, purpose, privileges, scope, environment, and expiry.
- Consumers, delivery method, storage locations, versions, and last-use evidence.
- Rotation/revocation API, overlap support, propagation delay, and emergency access.
- Service health, auth errors, audit logs, and canary identities.
- Leak containment, stale-secret, rollback, revocation, and recovery test evidence.
Acceptance scoreboard
- Inventory records secret identity, owner, scope, consumers, storage, expiry, and recovery without values.
- Routine versus compromise sequencing is explicit.
- Replacement reduces unnecessary privilege and uses protected delivery.
- Every consumer proves business behavior and audit identity under the new credential.
- Old credential is revoked and no stale use remains.
- Runbook, alerts, ownership, secret scanning, and next rotation date are complete.
Minimum handoff record
- Versioned credential and secret rotation scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Worked example
Evidence collected
- The full token is exposed.
- Bot can post to a channel.
- The token is hard-coded in a client repo.
- No inventory shows where it is deployed.
Decision: Treat the token as compromised: revoke immediately, inventory deployments, create a replacement through the issuer, and remove it from code/history as appropriate.
Actions taken
- Revoked the exposed token and monitored unauthorized activity.
- Created a replacement stored outside source.
- Updated known consumers with canary verification.
- Scanned repository/logs and added secret detection.
Why this example matters: The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.
Verify, recover, and hand off
Completion tests
- A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
- Inventory records secret identity, owner, scope, consumers, storage, expiry, and recovery without values.
- Routine versus compromise sequencing is explicit.
- Replacement reduces unnecessary privilege and uses protected delivery.
- Every consumer proves business behavior and audit identity under the new credential.
- Old credential is revoked and no stale use remains.
- Runbook, alerts, ownership, secret scanning, and next rotation date are complete.
Rollback or safe recovery
- Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
- Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
- Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
- Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| New key works in one service only | A hidden consumer, cache, or deployment still uses the old credential. | Compare secret version and auth errors across consumers |
| Old key remains accepted | Revocation was not completed or edge caches lag. | Test old credential through issuer and each endpoint |
| Rotation causes outage | Old credential was removed before every consumer adopted replacement. | Inspect create-distribute-reload-verify-revoke order |
| Secret appears in logs | Delivery or error path leaks sensitive values. | Scan logs/artifacts and rotate again if exposed |
Reusable handoff record
- Versioned credential and secret rotation scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| target | object | Versioned environment, resource, identity, or workflow being evaluated. |
| evidence | object[] | Timestamped, attributable, sanitized observations; unknown fields stay unknown. |
| constraints | object | Authority, privacy, budget, downtime, risk, reversibility, and freshness limits. |
| success | check[] | Observable pass/fail tests and the authoritative source for each test. |
Agent refusal and escalation rules
- Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
- Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
- Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
- Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.
Confidence rule: Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low confidence; a controlled test that isolates the layer and passes verification can support high confidence.
Official reference starting points