SAYLOR Determine whether a trading bot is genuinely profitable after all fees, slippage, failures, inventory, transfers, stale marks, and survivorship effects.
The result you're building
An auditable bot PnL statement based on wallet/exchange cash flows and inventory, net of every fee/slippage/failure/transfer, separated into realized/unrealized results and tested against out-of-sample baselines without survivorship or look-ahead bias.
Use this guide when
- Determining whether a bot actually makes money.
- Comparing strategy, route, size, or fee changes.
Do not use it as a substitute for
- Do not trust dashboard win rate, gross PnL, or green exit labels without cash-flow reconciliation.
- Do not optimize on the same period used to claim expected profit.
Before you change anything
- Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- All fills/transactions/orders/signatures and wallet/exchange transfers.
- Starting/ending balances and open inventory with mark source/time.
- Protocol/platform/network/priority/borrow/funding/slippage/failed costs.
- Strategy version, signals, universe, downtime and rejected/missed trades.
Understand the system before fixing it
Cash flow outranks internal labels
Wallet/exchange balance changes plus inventory explain economic result.
Win rate is incomplete
Average win/loss, size, fees, tails and correlation determine expectancy.
Backtests need execution realism
Use only information available at decision time and model latency, liquidity, impact, failures, delistings, rugs and opportunity constraints.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Bot says profit; balance down | Accounting | Reconcile every transfer/fill/fee/inventory | Omitted costs, withdrawals, dust, marks or duplicates. |
| Backtest strong; live weak | Execution/overfit | Compare signal-to-fill latency/price/impact/rejects | Look-ahead, selection bias, stale quotes or cost assumptions. |
| High win, net loss | Payoff/tails | Expectancy and largest loss/cost | Small wins cannot cover losses/fees. |
| PnL depends on one trade/token | Concentration | Contribution by trade/asset/time | Not robust; survivorship/outlier risk. |
Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
Step 01 — Define accounting boundary
Why: Deposits/withdrawals can masquerade as PnL.
Do: Choose wallets/accounts/period/base currency; classify external transfers, trading cash flows, fees, inventory, rewards/airdrops and owner capital separately.
Read the result: Beginning + net external flows + trading PnL = ending equity (within marks).
Next: Resolve discrepancies first.
Step 02 — Reconstruct each round trip
Why: Bot logs may miss partial fills/failures.
Do: Use confirmed exchange/chain records to pair entries/exits and open inventory; include actual amount, average fill, all fees, failed attempts, transfers, taxes, slippage vs decision quote.
Read the result: One immutable trade ID links intent/order/signature/fill/result.
Next: Unknown stays unresolved, not estimated silently.
Step 03 — Calculate robust metrics
Why: Headline return hides path and capital use.
Do: Compute net realized/unrealized PnL, expectancy, profit factor, hit rate, average/median/tails, max drawdown, exposure, turnover, time in market, cost share, capacity and contribution concentration.
Read the result: Include confidence/sample size.
Next: Separate strategy from rewards/market beta.
Step 04 — Audit backtest integrity
Why: Leakage manufactures edge.
Do: Use point-in-time universe/data, no future labels, realistic detection/submission/inclusion/exit, fees/impact/failed tx, delisted/rugged assets, wallet limits and missed data.
Read the result: Walk-forward/out-of-sample is untouched by tuning.
Next: Compare simple baseline.
Step 05 — Compare predicted vs realized execution
Why: Edge often dies between signal and fill.
Do: For each live trade record decision quote/time, submit/confirm time, route, expected/actual output, priority, rejection and missed opportunity; bucket slippage by liquidity/size/latency.
Read the result: Adjust or stop when net expectancy after execution is non-positive.
Next: Do not hide unfilled losses.
Step 06 — Set honest scale gate
Why: Increasing size changes impact and drawdown.
Do: Require minimum trades/time/regimes, positive after-cost out-of-sample/live result, bounded drawdown, sell/recovery test, capacity curve, kill switch and maximum loss/day/position.
Read the result: Scale incrementally and re-audit.
Next: Keep immutable reports.
Worked example
Evidence collected
- Gross trade PnL excludes priority fees and failed transactions.
- Small winners average 3%; losers average 18%.
- Transfers to funding wallet are misclassified.
- Two open illiquid tokens are marked at last trade price.
Decision: Win rate masks negative expectancy, cost leakage and optimistic inventory marks.
Actions taken
- Rebuilt PnL from wallet transactions and conservative executable marks.
- Included all fees/failures/transfers separately.
- Added max loss, liquidity/exit, net-edge and reconciliation gates.
Why this example matters: The profitable-looking story disappears when measured as realizable cash flow.
Verify, recover, and hand off
Completion tests
- A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Starting/ending equity and external flows reconcile.
- Every logical trade includes actual fills/fees/failures/open inventory.
- Marks are timestamped and executable/conservative.
- Net metrics include tails/drawdown/capacity/sample uncertainty.
- Backtest is point-in-time and out-of-sample.
- Live scale/kill-switch gates are enforced.
Rollback or safe recovery
- Pause new entries and reconcile/carefully exit inventory under risk policy.
- Return strategy/model/route parameters to last audited version.
- Do not auto-compensate accounting discrepancies with new trades.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Unmatched transactions | Airdrop/transfer/partial fill/missing wallet/provider data. | Classify with evidence or leave unresolved and lower confidence. |
| Mark shows profit but cannot quote exit | Illiquid inventory. | Use zero/conservative liquidation scenario and hard risk flag. |
| Results change after each run | Mutable data/non-deterministic pairing/marks. | Snapshot inputs and stable trade IDs/rules. |
| Only optimized period profitable | Overfit/data leakage. | Holdout/walk-forward and simpler baseline; no live claim. |
Reusable handoff record
- Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- Accounting boundary and reconciliation equation.
- Trade-level cash flows/costs/inventory.
- Net performance/risk/capacity/concentration metrics.
- Backtest leakage and live execution audit.
- Scale/pause/kill-switch decision with limitations.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| context | object | Versioned environment, target, and requested outcome. |
| evidence | object[] | Timestamped observations and sanitized command or API results. |
| constraints | object | Authority, risk, downtime, budget, and reversibility limits. |
| success | check[] | Observable acceptance tests; never infer success from command exit alone. |
Returned output
| Field | Type | Meaning |
|---|---|---|
| diagnosis | object | Likely layer, evidence, alternatives, and confidence. |
| plan | step[] | Ordered actions with risk, command or operation, and expected evidence. |
| verification | check[] | Pass/fail checks that prove the requested outcome. |
| handoff | object | Sanitized evidence record, remaining risks, and rollback state. |
Agent refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.
Official reference starting points