SAYLOR Build Solana monitoring that survives disconnects, rate limits, missed slots, duplicate events, provider disagreement, and reorg or commitment differences.
The result you're building
A Solana monitor that checkpoints progress, reconnects with bounded backoff, backfills missed history, deduplicates events, tracks commitment/slot/source, detects gaps and provider disagreement, and never treats a WebSocket as durable delivery.
Use this guide when
- Building wallet/token/program/new-launch monitoring.
- Fixing disconnects, duplicate alerts, rate limits, or missed events.
Do not use it as a substitute for
- Do not assume subscription reconnect replays missed events.
- Do not act financially on one unconfirmed provider event without policy and reconciliation.
Before you change anything
- Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- Exact monitored entity/events and acceptable commitment/finality.
- RPC/WS providers, limits, auth, methods and historical backfill path.
- Checkpoint/dedup identifiers and durable storage.
- Gap, fork, latency, cost and failover requirements.
Understand the system before fixing it
WebSockets are notifications, not a ledger
Connections drop and provider buffers are finite. The chain/RPC history is source for backfill.
Commitment changes latency and reorg exposure
Processed/confirmed/finalized serve different use cases; record context and promote/retract state.
Dedup key depends on event type
Signature+instruction/log index or account+slot/version is safer than raw message text.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Disconnect/reconnect | Transport | Last seen slot/signature and provider status | Reconnect then backfill overlap; do not resume blindly. |
| Duplicate events | At-least-once/overlap | Stable event key and payload hash | Upsert/dedup; conflicting duplicate alerts. |
| Missing slot range | Gap | Compare checkpoint with current/first backfill result | Backfill bounded history or declare incomplete. |
| Provider 429 | Rate | Headers/errors/request rate | Throttle/cache/batch/failover; no tight loop. |
| Providers disagree | Commitment/indexing/fork | Compare slot/blockhash/signature status | Wait higher commitment or reconcile chain source. |
Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
Step 01 — Define event and truth semantics
Why: Vague 'new transaction' creates duplicates/misses.
Do: Specify account/program/log/change, filters, event key, required fields, commitment, reorg policy and action threshold.
Read the result: Processed events are provisional unless policy says otherwise.
Next: Separate notification from confirmed record.
Step 02 — Persist checkpoints and raw references
Why: Memory-only cursor disappears on crash.
Do: Store provider, subscription, last observed slot/signature, finalized checkpoint, event key/hash, status and timestamps durably before/with processing.
Read the result: Restart knows overlap range.
Next: Avoid storing secrets/full unnecessary payloads.
Step 03 — Reconnect with backoff and resubscribe
Why: Fast reconnect storms worsen outages.
Do: Heartbeat/staleness detection, exponential backoff+jitter/cap, auth refresh, resubscribe and connection metrics.
Read the result: Connected without event flow is also unhealthy.
Next: Trigger backfill from durable checkpoint minus overlap.
Step 04 — Backfill and deduplicate
Why: Subscription cannot guarantee missed delivery.
Do: Use signatures/history/blocks/account methods appropriate to event, paginate until checkpoint/limit, merge live and historical by stable key, process deterministic order.
Read the result: Every gap closes or is explicitly incomplete.
Next: Rate-limit and cache.
Step 05 — Handle commitment and forks
Why: Provisional events can disappear/change.
Do: Track slot/blockhash/commitment, promote when confirmed/finalized, retract/mark orphaned per policy, and delay irreversible alerts/actions until threshold.
Read the result: Never silently keep orphaned event as final.
Next: Test simulated fork/status change.
Step 06 — Fail over and observe
Why: Secondary provider can disagree or cost more.
Do: Health-score latency/error/slot lag, compare sampled results, use bounded failover, circuit breakers and budgets; alert disconnect/gap/backfill age/duplicate/disagreement.
Read the result: Failover does not double-process.
Next: Run outage/load tests.
Worked example
Evidence collected
- Client reconnects and resubscribes but has no checkpoint/backfill.
- Alerts use raw log text, so duplicates appear on retries.
- Provider offers signature history.
- Monitor cannot prove completeness.
Decision: Transport recovery restored future events but permanently skipped outage window.
Actions taken
- Persisted slot/signature checkpoint and stable instruction event key.
- On reconnect backfilled overlapping signature history and merged live feed.
- Added gap completeness flag and provider-lag alert.
Why this example matters: Durability comes from reconciliation with history, not a reliable-looking socket.
Verify, recover, and hand off
Completion tests
- A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Crash/reconnect resumes from durable checkpoint.
- Outage window backfills with overlap and no duplicates.
- Every event carries slot/commitment/source/observed time.
- Fork/provisional state promotes or retracts correctly.
- 429/outage/provider disagreement stay bounded.
- Completeness, lag, duplicate and cost metrics alert.
Rollback or safe recovery
- Pause downstream actions while continuing safe ingestion/backfill.
- Return to prior provider/client version without resetting checkpoint.
- Reprocess from earlier checkpoint into idempotent sink when parser defect found.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Socket connected, no events | Stale connection/subscription/filter/provider lag. | Heartbeat/slot progress and resubscribe/backfill. |
| Backfill loops | Pagination cursor/order/checkpoint wrong. | Record page cursors and monotonic termination. |
| Alerts duplicate after failover | Provider-specific ID used. | Use chain-stable signature/instruction key. |
| Finalized too slow | Commitment tradeoff. | Emit provisional label then promote; never mislabel certainty. |
Reusable handoff record
- Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- Event/commitment/reorg semantics.
- Durable checkpoint/dedup schema.
- Reconnect/backfill/failover algorithms and budgets.
- Gap/outage/fork/load test results.
- Completeness/latency/error/cost monitoring.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| context | object | Versioned environment, target, and requested outcome. |
| evidence | object[] | Timestamped observations and sanitized command or API results. |
| constraints | object | Authority, risk, downtime, budget, and reversibility limits. |
| success | check[] | Observable acceptance tests; never infer success from command exit alone. |
Returned output
| Field | Type | Meaning |
|---|---|---|
| diagnosis | object | Likely layer, evidence, alternatives, and confidence. |
| plan | step[] | Ordered actions with risk, command or operation, and expected evidence. |
| verification | check[] | Pass/fail checks that prove the requested outcome. |
| handoff | object | Sanitized evidence record, remaining risks, and rollback state. |
Agent refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.
Official reference starting points