Scheduled and Durable Agent Workflows

The result you are building

Finished Result:

A schedule and workflow definition that survives restarts, handles missed or duplicate triggers, resumes from checkpoints, prevents overlapping side effects, and proves cancellation, timeout, recovery, and end-to-end delivery.

Use this guide when

• An agent performs recurring checks, reports, billing, monitoring, follow-up, or conditional work.
• A workflow can outlive one process or require waits and approvals.
• Duplicate or missed execution would affect people, money, production, or data.

Do not use it as a substitute for

• Keeping the only schedule in a desktop process or model conversation.
• Retrying every failure forever or allowing overlapping runs without a policy.

Before you change anything

Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.

• Schedule expression, timezone, DST policy, start/end, and owner.
• Trigger ID, workflow version, task inputs, and deduplication window.
• Step state, timeout, retry/backoff, idempotency, and side effects.
• Concurrency, catch-up, cancellation, pause, kill, and escalation policy.
• Missed, duplicate, delayed, overlapping, restart, cancellation, and recovery test evidence.

Stop Before Proceeding:

Do not activate a recurring side effect until timezone, catch-up, overlap, idempotency, budget, and cancellation behavior are explicit and tested.

Understand the system before fixing it

Capability is not authority A tool may be technically able to act while policy, budget, ownership, or user intent still forbids the action. Keep execution permission explicit and revocable.

Deterministic boundaries contain probabilistic reasoning Models may propose plans, but schemas, allowlists, budgets, approvals, idempotency, and postconditions must decide what can actually happen.

A trigger is not a completed job Record scheduled, started, checkpointed, succeeded, failed, canceled, and delivered states separately.

Time semantics are business logic Timezone, daylight-saving transitions, clock drift, maintenance windows, and missed runs can change who receives what and when.

Evidence-to-decision map

Start with the row that most closely matches the evidence. The first test isolates a layer; it is not permission to
make every available change.

   Evidence                Likely layer    First decisive check              What the result means

   Job runs twice at the   Concurrency/d   Compare trigger IDs, lease, and   Scheduler delivery or failover is at least once and overlap is
   same time               edup            workflow key                      ungoverned.

   Restart loses           Durable state   Recover workflow from             State exists only in process memory.
   progress                                checkpoint and history

   DST changes run         Time            Evaluate schedule across          Local time policy is unspecified or scheduler interprets it
   time                    semantics       timezone transition               differently.

   Cancellation still      Cooperative     Cancel during each side-effect    Queued or child steps do not recheck workflow state.
   sends output            cancellation    boundary

   Failure retries until   Retry policy    Inspect error class and attempt   Permanent and transient failures are not separated.
   budget exhausted                        budget

Step-by-step procedure

Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.

01 Define trigger and calendar semantics Why: A precise boundary prevents a plausible fix from solving the wrong problem.

Do: Specify timezone, daylight-saving behavior, start/end, exclusions, holiday/maintenance rules, missed-run catch-up, and owner. Prefer explicit UTC where business meaning permits.

Read the result: Test dates around DST and exclusions produce expected next runs.

Next: Record the evidence and continue only when the stated proof is present.

02 Create stable workflow identity Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.

Do: Assign schedule ID, workflow type/version, logical run key, trigger time, and input snapshot. Separate retries from new scheduled runs.

Read the result: Duplicate triggers map to one logical run.

Next: Record the evidence and continue only when the stated proof is present.

03 Persist state and checkpoints Why: Inconsistent inputs create false differences and make later comparisons unreliable.

Do: Store step status, attempts, evidence refs, approvals, deadlines, child IDs, and terminal result outside the worker process.

Read the result: A fresh worker reconstructs the exact current state.

Next: Record the evidence and continue only when the stated proof is present.

04 Make side effects idempotent Why: A decisive test reduces trial-and-error and limits unnecessary change.

Do: Use business keys and atomic result storage for sends, payments, writes, and deliveries. Reject conflicting reuse.

Read the result: Replaying a completed or ambiguous step cannot duplicate its effect.

Next: Record the evidence and continue only when the stated proof is present.

Procedure continued 05 Bound retries, waits, and cost Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.

Do: Classify failures, use exponential backoff with jitter, cap attempts/time/spend, set heartbeat and timeout, and escalate permanent blockers.

Read the result: A failed dependency cannot consume unbounded time or money.

Next: Record the evidence and continue only when the stated proof is present.

06 Control overlap, pause, and cancellation Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.

Do: Choose allow, skip, queue, replace, or single-flight overlap. Recheck paused/canceled state before every external effect.

Read the result: Forced overlap and cancellation tests produce the declared result.

Next: Record the evidence and continue only when the stated proof is present.

07 Verify delivery and recovery Why: A result is not complete until it remains observable and repeatable after the immediate fix.

Do: Simulate scheduler duplicate, missed trigger, worker crash, lost heartbeat, dependency recovery, and dead-letter reprocessing. Monitor schedule lag and terminal delivery.

Read the result: The workflow survives failure without silent loss or duplicate side effects.

Next: Record the evidence and continue only when the stated proof is present.

Operational worksheet Evidence record Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.

• Schedule expression, timezone, DST policy, start/end, and owner.
• Trigger ID, workflow version, task inputs, and deduplication window.
• Step state, timeout, retry/backoff, idempotency, and side effects.
• Concurrency, catch-up, cancellation, pause, kill, and escalation policy.
• Missed, duplicate, delayed, overlapping, restart, cancellation, and recovery test evidence.

Acceptance scoreboard

• Timezone, DST, exclusions, catch-up, overlap, and owner behavior are explicit.
• Duplicate triggers resolve to one logical workflow run.
• A fresh worker resumes from durable state without losing approvals or evidence.
• Every external side effect is idempotent and reconcilable.
• Retries, waits, spend, and total workflow lifetime are bounded.
• Pause, cancel, crash, duplicate, and missed-trigger tests pass end to end.

Decision rule SHIP / AUTOMATE GATE Proceed only when every required acceptance check is supported by direct evidence, rollback is available, and the remaining risk is explicitly owned. Unknown is not a pass.

Minimum handoff record

• Versioned scheduled and durable agent workflows scope, owner, exclusions, and success criteria.
• Sanitized evidence snapshot with source, time, version, and confidence.
• Decision map showing rejected alternatives and the decisive tests used.
• Ordered action log with approvals, idempotency keys, outputs, and rollback state.
• Acceptance results, remaining risks, review date, and escalation owner.

Worked example

Starting Problem:

A daily monitoring agent restarts during message delivery and sends the same alert twice.

Evidence collected

• The scheduler redelivers the trigger.
• Run state is stored only in memory.
• Message send has no business key.
• The worker marks completion after sending, leaving a crash window.

Decision Use a durable logical run ID and idempotent delivery record that can reconcile the provider message after restart.

Actions taken

• Persisted run and step state before execution.
• Bound the alert to schedule ID, due time, recipient, and finding digest.
• Stored provider ID and terminal result atomically.
• Added crash-before and crash-after-send tests.

Proof Of Completion:

Repeated triggers and worker crashes produce one logical report and at most one message per approved recipient, with visible recovery history.

Why this example matters The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.

Verify, recover, and hand off

Completion tests A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.

• Timezone, DST, exclusions, catch-up, overlap, and owner behavior are explicit.
• Duplicate triggers resolve to one logical workflow run.
• A fresh worker resumes from durable state without losing approvals or evidence.
• Every external side effect is idempotent and reconcilable.
• Retries, waits, spend, and total workflow lifetime are bounded.
• Pause, cancel, crash, duplicate, and missed-trigger tests pass end to end.

Rollback or safe recovery

• Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
• Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
• Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
• Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.

If the expected result does not appear What happened What it usually means Next safe move

Job runs twice at the same Scheduler delivery or failover is at least Compare trigger IDs, lease, and workflow key time once and overlap is ungoverned.

Restart loses progress State exists only in process memory. Recover workflow from checkpoint and history

DST changes run time Local time policy is unspecified or Evaluate schedule across timezone transition scheduler interprets it differently.

Cancellation still sends output Queued or child steps do not recheck Cancel during each side-effect boundary workflow state.

Reusable handoff record

• Versioned scheduled and durable agent workflows scope, owner, exclusions, and success criteria.
• Sanitized evidence snapshot with source, time, version, and confidence.
• Decision map showing rejected alternatives and the decisive tests used.
• Ordered action log with approvals, idempotency keys, outputs, and rollback state.
• Acceptance results, remaining risks, review date, and escalation owner.

Agent delivery contract

    Required inputs
       Field                             Type              Requirement

       target                            object            Versioned environment, resource, identity, or workflow being evaluated.

       evidence                          object[]          Timestamped, attributable, sanitized observations; unknown fields stay unknown.

       constraints                       object            Authority, privacy, budget, downtime, risk, reversibility, and freshness limits.

       success                           check[]           Observable pass/fail tests and the authoritative source for each test.

    Returned output
       Field                             Type              Requirement

       diagnosis                         object            Likely layer, supporting and conflicting evidence, alternatives, and confidence.

       plan                              step[]            Ordered bounded actions with owner, risk, expected proof, and stop condition.

       verification                      check[]           Observed pass/fail/unknown results, not inferred success from command exit alone.

       handoff                           object            Sanitized evidence record, recovery state, remaining risk, and next review trigger.

    Agent refusal and escalation rules
•
      Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
•
      Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
•
      Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
•
      Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.

    Confidence rule
    Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low
    confidence; a controlled test that isolates the layer and passes verification can support high confidence.

Official reference starting points

• https://cloudevents.io/
• https://docs.temporal.io/
• https://vercel.com/docs/workflow