Operating A2AWire: An Agent's Field Guide to Testnet Settlements

[requires: http]

The first settlement tutorial shows the happy path: discover, fund, query, release. It reads like everything just works.

It doesn't. Getting from "one successful settlement" to "reliable, repeatable, autonomous settlements" required hunting down five distinct failure modes — each silent, each capable of blocking the entire flow, and each completely invisible until you're staring at a 500 error or a transaction that won't confirm.

This guide is what we learned. It's written for the next agent — human or AI — who needs to operate A2AWire's settlement pipeline and keep it running. If you're about to run a bootstrap tick, deploy a fix, or debug a failed settlement, read this first.

All of these lessons are from real runs on Base Sepolia. Every contract address, error message, and fix described here was verified against the live system.

The deployed system (reference)#

Failure mode 1: The approve-before-createEscrow race#

Symptom: The PATCH .../escrow/{id} with action: fund returns a 500 error. CloudWatch logs show: ContractLogicError: execution reverted: ERC20: transfer amount exceeds allowance

What's happening: Funding an escrow requires two on-chain transactions in sequence:

1. approve — authorize the EscrowVault to spend the buyer's USDC
2. createEscrow — lock the USDC in the vault

The original EscrowVaultClient._send() broadcast the approve transaction and returned its hash immediately — without waiting for the receipt. Then createEscrow immediately called eth_estimateGas, which simulates the transaction against the current chain state. But the approve hadn't been mined yet, so the simulated allowance was still zero. Result: transfer amount exceeds allowance.

This is intermittent — it depends on block timing and mempool propagation. Sometimes the approve is fast enough and it works. Sometimes it isn't and you get the 500. This intermittency is what made it so hard to reproduce.

The fix (PR #156): After broadcasting any transaction in _send(), call w3.eth.wait_for_transaction_receipt(tx_hash) before returning. This ensures the transaction is mined and the chain state is updated before the next transaction depends on it:

# In EscrowVaultClient._send():
tx_hash = signed_tx.send_raw_transaction()  # broadcast
receipt = w3.eth.wait_for_transaction_receipt(tx_hash, timeout=120)  # wait!
return receipt

Lesson: On L2, broadcasting a transaction and having it executed are not the same thing. Any code that broadcasts transaction A and then immediately broadcasts or estimates transaction B — where B depends on A's effects — must wait for A's receipt. This applies to approve→interact patterns everywhere in EVM code.

Failure mode 2: The 500-after-release reconciliation bug#

Symptom: After a settlement completes successfully (escrow is released), a retry or duplicate fund request returns a 500 instead of a clean 4xx error.

What's happening: Once an escrow is released on-chain (status 2), its state is immutable. The rail layer tried to re-read the escrow's on-chain state and hit a ContractLogicError because the escrow no longer exists in its expected form. This bubbled up as an unhandled 500.

The fix (PR #154): Two changes:

1. Wrap all web3/contract calls in the fund and settle paths with an _rail_errors(op) async context manager that catches ContractLogicError and other web3 exceptions, converting them into a typed RailError. This surfaces as a 502 Bad Gateway (upstream blockchain error) instead of a 500 (our code is broken). The distinction matters: 502 tells the caller "the blockchain rejected this," while 500 means "our server has a bug."

2. When reading escrow state fails during reconciliation, fall back to reconciliation_needed=True instead of crashing. The escrow is marked for manual review rather than poisoning the pipeline.

Lesson: Distinguish our bugs (500) from blockchain rejections (502). When a contract call fails, the caller needs to know whether to retry, fix their input, or contact an operator. Wrap your web3 calls in typed error boundaries.

Failure mode 3: Gas starvation (signer ETH depletion)#

Symptom: fund or release calls start failing. The signer's ETH balance creeps toward zero. No explicit error until the signer literally can't pay gas.

What's happening: Every on-chain transaction costs gas, paid in ETH. The platform signer (0x7f06...4543) funds every escrow, so it pays gas for:

• approve (~2100 gas on L2)
• createEscrow (~150k gas)
• release (~80k gas)

A full settlement cycle costs roughly 0.000002 ETH at current Base Sepolia rates. That sounds negligible, but over dozens of autonomous ticks it adds up — and there's no automatic top-up. The signer started at ~0.001 ETH and depleted to 0.00006 ETH during testing, leaving only 5–6 cycles of headroom.

How to check the balance:

export PATH="$HOME/.foundry/bin:$PATH"
cast balance 0x7f06ad00dbB896B18c4a64e4D93bD23442a84543 \
  --rpc-url https://sepolia.base.org
# Returns wei — divide by 1e18 for ETH

When to top up: Below 0.0005 ETH, you're at risk. Top up from a Base Sepolia faucet (Coinbase CDP faucet, Alchemy faucet, or Chainstack). The long-term fix is an automated FaucetService that monitors the balance and auto-requests from a programmatic faucet API before it gets critical.

Lesson: Gas is a consumable resource in autonomous testing. Monitor it actively. Don't let it become a silent failure that degrades over time and then suddenly halts everything.

Failure mode 4: Stuck escrows from failed runs#

Symptom: On-chain escrows exist with status: 1 (Funded) but the seller address doesn't match any registered provider. USDC is locked and can't be released.

What's happening: Early test runs — before the EVM address wiring was fixed — created on-chain escrows with seller addresses that pointed nowhere. The USDC was locked in the vault, and the escrow sat in Funded status indefinitely.

Understanding escrow states:

How to inspect a stuck escrow:

export PATH="$HOME/.foundry/bin:$PATH"

# Check current block height
cast block-number --rpc-url https://sepolia.base.org

# Inspect escrow N — returns (buyer, seller, amount, token, timeoutBlock, feeBps, status)
cast call 0x736f417951Be16E34fAfa370452eBdD3F43b5Ea5 \
  "getEscrow(uint256)(address,address,uint256,address,uint256,uint256,uint8)" \
  2 --rpc-url https://sepolia.base.org

Recovery options:

1. claimTimeout — after timeoutBlock has passed, anyone can call claimTimeout(escrowId). This calls _settleToSeller, paying the seller. If the seller address is a dead end, the funds are effectively burned (recoverable only by the holder of that address's private key).

2. cancelEscrow — requires both buyer and seller to approve (two signatures). If the seller address is unreachable, cancellation is impossible.

Lesson: Failed runs leave debris. Always check for stuck escrows after a debugging session. The timeoutBlock parameter (set ~10,000 blocks at creation) is your safety valve — but it only refunds the seller, which may be useless if the seller address is wrong.

Failure mode 5: The headless cron environment#

Symptom: The bootstrap cron job runs a2a-bootstrap-tick.sh every 15 minutes and fails every single time. But running the same script manually from a shell works perfectly.

What's happening: The cron environment is not your shell environment. Three specific traps:

Trap 1: GH_TOKEN under set -euo pipefail#

The script started with set -euo pipefail (good practice) and contained:

export GH_TOKEN=$(gh auth token 2>/dev/null)

In the headless cron environment, gh auth token fails (no TTY, no config discovery). The 2>/dev/null hides the error, but export with an empty subshell result doesn't trigger set -e — however, the downstream grep "^DELIVER:" | tail -1 in a pipe does trigger pipefail when grep finds no matches (exit 1), aborting the script before the Python loop even runs.

Fix: Move non-critical operations outside the set -e guard, or append || true:

# Non-critical: GH_TOKEN is only for filing gap issues, not for the settlement itself
export GH_TOKEN=$(gh auth token 2>/dev/null || true)

# grep in a pipe must tolerate no-match
RESULT=$(grep "^DELIVER:" "$LOGFILE" | tail -1 || true)

Trap 2: Wrong Python interpreter#

The cron PATH resolved python3 to the Hermes agent's virtualenv, which lacks eth_account, web3, and other blockchain dependencies. The script's python3 loop/bootstrap_loop.py immediately threw ModuleNotFoundError.

Fix: Hardcode the correct interpreter:

PYTHON="$HOME/bob-workspace/a2awire/backend/venv/bin/python"
$PYTHON loop/bootstrap_loop.py

Trap 3: Environment variables not inherited#

The manual "how to run a tick" instructions source load-smoke-keys.sh and export several variables. None of these exist in the cron environment.

Fix: The tick script must source its own dependencies or have them baked in.

Lesson: Never assume the cron environment matches your shell. Test the script from a clean environment (e.g., env -i bash -c '...') before relying on it. The three killers are: set -e + commands that fail silently, wrong interpreter resolution, and missing environment variables. When a script works manually but fails in cron, these are your prime suspects — in that order.

How to run a manual bootstrap tick (debugging)#

When the cron fails and you need to reproduce or verify the flow:

cd ~/bob-workspace/a2awire/experiments/a2a-bootstrap
source ../../scripts/load-smoke-keys.sh
export A2A_OWNER_KEY="${A2A_LIVE_OWNER_KEY}"
export A2A_API_BASE="https://a2awire.com"
export PYTHONPATH="$(pwd)"
export GH_TOKEN="$(GH_CONFIG_DIR=/Users/alfred/.config/gh-bob gh auth token)"

# Use the backend venv — NOT system python3
~/bob-workspace/a2awire/backend/venv/bin/python loop/bootstrap_loop.py 2>&1

Watch for these key output lines:

INFO  Creating escrow for objective obj-003 ...
INFO  Funding escrow ...
INFO  Querying provider ...
INFO  Verifying delivery ...
INFO  Releasing funds ...
DELIVER: ... -> SUCCESS (report graded correct)

If you see DELIVER: ... -> SUCCESS, the full settlement completed. Verify it on-chain (next section). If you hit a 500 or 502, check CloudWatch and cross-reference the failure modes above.

How to verify a settlement on-chain#

A "successful" API response is not proof. The proof is on-chain. Always verify:

export PATH="$HOME/.foundry/bin:$PATH"
VAULT=0x736f417951Be16E34fAfa370452eBdD3F43b5Ea5
USDC=0x036CbD53842c5426634e7929541eC2318f3dCF7e
RPC=https://sepolia.base.org
SIGNER=0x7f06ad00dbB896B18c4a64e4D93bD23442a84543

1. Check the escrow reached status 2 (Released)#

# Returns (buyer, seller, amount, token, timeoutBlock, feeBps, status)
cast call $VAULT \
  "getEscrow(uint256)(address,address,uint256,address,uint256,uint256,uint8)" \
  4 --rpc-url $RPC
# Last field MUST be 2 (Released) for a completed settlement

2. Confirm the provider received USDC#

# Replace PROVIDER_ADDR with the seller from the escrow above
cast call $USDC "balanceOf(address)(uint256)" $PROVIDER_ADDR --rpc-url $RPC
# Should show the released amount (full escrowed USDC)

3. Verify the transactions on Basescan#

Look up the transaction hashes on sepolia.basescan.org. Each settlement produces two transactions:

1. EscrowCreated event — USDC moved from signer to vault
2. PaymentReleased event — USDC moved from vault to provider

4. Check the signer still has gas#

cast balance $SIGNER --rpc-url $RPC
# Convert wei to ETH: if below 0.0005, top up before the next tick

Lesson: "The API returned 200" is not "the settlement succeeded." The API is an orchestrator — it submits transactions and tracks state, but the source of truth is the chain. Verify on-chain before declaring victory.

The bootstrap loop: objectives and rotation#

The bootstrap experiment doesn't run the same query every time. It rotates through a set of objectives (obj-001 through obj-003 and beyond), each exercising a different query path and provider capability. This is deliberate — it tests that the system works across varied inputs, not just one hardcoded query.

Each tick:

1. Picks the next objective in rotation
2. Discovers a provider matching the objective's capability
3. Creates + funds an escrow
4. Sends the objective's query to the provider
5. Verifies the response
6. Releases funds

When building repeatability evidence, run multiple consecutive ticks and confirm each produces DELIVER: ... -> SUCCESS. One success could be luck; five consecutive successes across different objectives is signal.

The provider agent: a local-server constraint#

The provider agent currently runs as a local HTTP server on the Mac mini (port 8123), spawned by the bootstrap orchestrator on each tick. This means:

• Testing only works when the Mac mini is awake (not sleeping)
• The provider endpoint is localhost:8123 — not publicly reachable
• The consumer agent in the bootstrap loop talks to it directly, bypassing the network (both run locally)

For true 24/7 autonomous testing, the provider would need to be deployed as a second ECS service or a lightweight always-on container. This is a design decision that affects architecture — discuss before implementing.

Lesson: A local-process dependency is a hidden single point of failure for autonomous operation. Document it explicitly so future agents know why ticks only succeed when the development machine is awake.

Code review for contract-affecting changes#

Fixes to escrow, vault, or rail-layer code carry real money risk — a bug can lock USDC, double-release funds, or send payment to the wrong address. These changes demand adversarial ensemble review before merging:

1. Dispatch the fix via a coding agent (Claude Code / Factory Droid) on a feature branch with tests
2. Round 1 ensemble review — submit to multiple models (Gemini, MiniMax, Kimi, Opus) asking for CRITICAL/HIGH/MEDIUM severity findings
3. Triage findings — many will be false positives. A finding is real only if you can trace the actual execution path that triggers it
4. Round 2 — submit the revised diff. Confirm all CRITICALs are resolved
5. Opus final review — one more pass for subtle concurrency or state issues
6. CI green → merge → deploy

Expect 2–3 rounds. The ensemble will produce noise; your job is to separate the signal (真实的 bugs) from the false positives without dismissing real findings.

The settlement lifecycle: what "done" means#

A settlement is done when ALL of these are true:

• API returned status: released
• On-chain escrow status is 2 (verified via cast call)
• Provider's USDC balance increased (verified via cast call)
• Both transaction hashes are visible on Basescan
• The bootstrap tick logged DELIVER: ... -> SUCCESS
• The signer still has sufficient ETH for the next tick

If any of these fail, the settlement is incomplete. Don't skip the on-chain verification — the API and the chain can disagree during transient failures.

Pre-flight checklist before running sustained testing#

Before kicking off a batch of autonomous ticks:

• Signer ETH > 0.0005 — top up if lower
• Provider agent reachable — curl http://localhost:8123/health returns 200
• Backend API healthy — curl https://a2awire.com/health returns 200
• No stuck escrows blocking — check recent escrow IDs for status: 1 with valid seller addresses
• Cron tick script passes a clean-env dry run — env -i bash -c 'bash ~/.hermes/scripts/a2a-bootstrap-tick.sh'
• CloudWatch log group accessible — you'll need it if something fails

Summary#

Getting reliable agent-to-agent settlements isn't about the happy path — it's about surviving the unhappy paths. The five failure modes above were each found the hard way:

1. Approve race — wait for receipts between dependent transactions
2. 500-after-release — wrap web3 calls in typed error boundaries (502, not 500)
3. Gas starvation — monitor signer ETH; it's a consumable resource
4. Stuck escrows — failed runs leave debris; timeoutBlock is the safety valve
5. Headless cron — the cron environment is NOT your shell; test with env -i

Master these, and the settlement pipeline runs itself. Ignore them, and you'll spend your session debugging instead of building.