# Temprano Watchtower > Temprano Watchtower is a Rust service that accepts signed Tempo transactions, stores them durably, and broadcasts them throughout their validity window. ## Concepts ### Delivery Semantics * Broadcast starts at `valid_after` (or immediately if unset). * Retry continues until: * the transaction is mined, * the validity window expires, * the transaction is provably invalid, * or the user cancels locally. The transaction can fail to be executed for a number of reasons. A non-exhaustive list is: * Sender out-of-funds * Fees too low for transaction to be included * RPC failure The Watchtower will always retry to send the transaction unless it [becomes invalid](#validity-logic). ### Transaction Format The service accepts the same payload as Ethereum raw transaction submission: ``` rawTx = "0x" + hex-encoded signed Tempo tx bytes ``` This is the output of: * `viem.signTransaction()` * Any Tempo wallet signer The server performs: * decoding * signature validation * field extraction * scheduling Notes: * Any valid Tempo transaction is accepted, including ones with custom nonce keys. * The service accepts JSON-RPC `eth_sendRawTransaction` payloads as-is. ### Grouping #### Group Identity Users may run multiple concurrent groups (for example, multiple subscriptions). Group identity is: ``` GroupKey = (sender_address, group_id) ``` Each sender can have arbitrarily many groups. #### How Grouping Works Grouping is based solely on the transaction nonce key. Transaction input data and memo contents are ignored. For every accepted transaction, the group fields are derived from the nonce key: * `group_id` = first 16 bytes of `keccak256(nonce_key_bytes)` All transactions that share a nonce key are placed in the same group for a given sender. Transactions are grouped only when `nonceKey` matches the explicit grouping format described below. ### Nonce Key Format Nonce keys are 32-byte values. Transactions are grouped only when the nonce key matches this explicit format. #### Binary Layout (Big-Endian) ``` 0..3 magic = 0x4E4B4731 // "NKG1" 4 version = 0x01 5 kind = enum (purpose) 6..7 flags = u16 8..15 scope_id = u64 16..19 group_id = u32 20..31 memo = 12 bytes ``` #### Flags Encoding (u16) ``` bits 0..1 scope_id encoding bits 2..3 group_id encoding bits 4..5 memo encoding bits 6..15 reserved (must be 0) ``` Encoding values (same for scope\_id, group\_id, memo): ``` 00 = numeric/raw 01 = ASCII 10 = reserved (undefined) 11 = reserved (undefined) ``` #### Display / Interpretation Rules * `numeric/raw`: * `scope_id` and `group_id` are interpreted as unsigned integers (big-endian) and displayed in decimal. * `memo` is rendered as hex (full 12 bytes, with 0x prefix). * `ASCII`: * Bytes must be printable 7-bit ASCII (0x20..0x7E). * Trailing `0x00` bytes are trimmed for display. * If any non-printable byte is present, render as hex. #### Example ``` magic/version/kind/flags = 4e4b4731 01 02 0001 scope_id (ASCII) = 504159524f4c4c00 // "PAYROLL\0" group_id (numeric) = 00000f42 memo (ASCII) = 4a414e2d323032360000000000 // "JAN-2026" ``` Display: ``` kind=0x02, scope=PAYROLL, group=3906, memo=JAN-2026 ``` ### State Machine Transactions follow this lifecycle: ``` queued → broadcasting ↔ retry_scheduled → terminal ``` Terminal states: * `executed` * `expired` * `invalid` (provably invalid) * `stale_by_nonce` * `canceled_locally` ### Validity Logic #### Time Window * `eligibleAt = valid_after || now` * `expiresAt = valid_before || ∞` #### Provably Invalid (Stop Early) * malformed tx * invalid sender signature * invalid fee payer signature * expired at ingest * nonce mismatch (if transaction is replaced on-chain) #### Dynamic (Retry Until Expiry) * insufficient balance * fee token selection failure * temporary RPC errors ## System Design ### Scheduler #### Behavior * Due transactions are pulled by `next_action_at`. * Database-backed leasing is used for multi-replica safety. * Redis ZSET is used as an accelerator only. * Guaranteed retry continues until expiry. #### Retry Strategy * Near eligibility: 250–500ms attempts. * Backoff up to 5s max. * Endpoint rotation. * Health scoring per RPC endpoint. ### Broadcaster * Fan-out to multiple Tempo RPC endpoints. * Tracks: * accepted * rejected (with reason) * timeout * Stops only on terminal conditions. ### Chain Watcher Tracks: * `(sender, nonce_key) → current_nonce` * receipts for known transactions Transitions: * mined → terminal * nonce advanced past tx → `stale_by_nonce` Notes: * The watcher uses websocket subscriptions when available and falls back to polling. ### Storage Model #### `txs` Table Fields: * `chain_id` * `tx_hash` (unique) * `raw_tx` * `sender` * `fee_payer` * `nonce_key` * `nonce` * `valid_after` * `valid_before` * `eligible_at` * `expires_at` * `status` * `group_id` (16 bytes nullable) * `next_action_at` * leasing fields * timestamps Indexes: * `(chain_id, tx_hash)` unique * `(sender, group_id)` * `(status, next_action_at)` ### Redis Acceleration Keys: * `watchtower:ready:{chain}` → ZSET(tx\_hash, eligibleAt) * `watchtower:retry:{chain}` → ZSET(tx\_hash, nextRetryAt) * Optional inflight/lease keys Redis is rebuildable from the database. Redis is used as a scheduling accelerator; the database remains the source of truth. ### Observability Metrics: * ingest rate * queue depth * retry counts * success/failure rates * time-to-mined Tracing: * full transaction lifecycle Logs: * structured per txHash ### Security * request size limits * strict decoding * rate limiting * no signing * no private key handling ## Configuration The service reads `config.toml` by default. You can override the path with the `CONFIG_PATH` environment variable. The config file supports environment variable interpolation (for example `${DB_HOST}`). ### Sample `config.toml` ```toml [server] bind = "0.0.0.0:8080" [database] url = "postgres://${DB_USER}@${DB_HOST}:${DB_PORT}/${DB_NAME}" [redis] url = "redis://${REDIS_HOST}:${REDIS_PORT}/${REDIS_DB}" [rpc] # Per-chain RPC endpoints. Keys are chain IDs. [rpc.chains] "42431" = ["${RPC_URL}"] [scheduler] poll_interval_ms = 200 lease_ttl_seconds = 30 max_concurrency = 50 retry_min_ms = 250 retry_max_ms = 900000 expiry_soon_window_seconds = 3600 expiry_soon_retry_max_ms = 5000 [broadcaster] fanout = 2 timeout_ms = 2000 [watcher] poll_interval_ms = 1500 use_websocket = true [api] max_body_bytes = 1048576 ``` ### `server` * `bind`: Address and port to listen on (for example `0.0.0.0:8080`). ### `database` * `url`: PostgreSQL connection string. Environment variables may be interpolated. ### `redis` * `url`: Redis connection string. Environment variables may be interpolated. ### `rpc` * `chains`: Map of chain IDs to one or more RPC URLs for each chain. Chain IDs are string keys in the TOML file, and each value is an array of URLs. These endpoints are used by the broadcaster and watcher. ### `scheduler` * `poll_interval_ms`: How often the scheduler scans for due work. * `lease_ttl_seconds`: Duration of DB-backed leases used for multi-replica scheduling. * `max_concurrency`: Maximum number of concurrent scheduler tasks. * `retry_min_ms`: Minimum delay between retry attempts near eligibility. * `retry_max_ms`: Maximum backoff delay between retry attempts. * `expiry_soon_window_seconds`: Window before expiry during which retry cadence is adjusted. * `expiry_soon_retry_max_ms`: Maximum retry delay when a transaction is nearing expiry. ### `broadcaster` * `fanout`: Number of RPC endpoints to broadcast to in parallel. * `timeout_ms`: Per-endpoint broadcast timeout in milliseconds. ### `watcher` * `poll_interval_ms`: How often the watcher polls for updates when websocket subscriptions are unavailable or disabled. * `use_websocket`: Whether to use websocket subscriptions when supported by the RPC endpoint. ### `api` * `max_body_bytes`: Maximum request body size accepted by the API. ## Dependencies This page describes the external services and tooling needed to run Temprano Watchtower in development or production. ### PostgreSQL PostgreSQL stores transactions, scheduler state, and metadata. You must create a database for Temprano Watchtower and provide the credentials [when configuring](./configuration). ### Redis Redis is used to improve scheduling performance. The database remains the source of truth; Redis can be rebuilt from Postgres. ### Reverse Proxy (optional) A reverse proxy is recommended when exposing the service to the Internet. It can terminate TLS, apply rate limits, and manage access controls in front of the Watchtower API. We provide a [sample configuration for nginx](https://github.com/arvina-tech/temprano-watchtower/blob/main/deployment/nginx/temprano-watchtower.conf) as part of this repo. ### Supervisor (optional) A process supervisor is recommended for production deployments to keep the service running and to manage restarts on failure. We provide a [sample configuration for supervisor](https://github.com/arvina-tech/temprano-watchtower/blob/main/deployment/temprano-watchtower.conf) as part of this repo. ## Getting Started This section covers installation requirements, configuration, and how to run and test the service locally. ## Installation See the Dependencies page for required services and optional components like a reverse proxy or supervisor. ### Precompiled Binaries Download the appropriate archive for your platform from the [GitHub releases page](https://github.com/arvina-tech/temprano-watchtower/releases), then: 1. Unpack the archive. 2. Place the `temprano-watchtower` binary somewhere on your `PATH` (or keep it alongside your deployment artifacts). ### Build From Source This project targets Rust 2024 edition and can be built using the latest stable release. Install straight from the GitHub repo in one step: ```bash cargo install --git https://github.com/arvina-tech/temprano-watchtower.git --locked ``` Otherwise, clone the repo and build the binary from source. From the repository root: ```bash cargo build --release ``` The binary will be at `target/release/temprano-watchtower`. You can run it directly: ```bash ./target/release/temprano-watchtower ``` Or install it into your Cargo bin directory: ```bash cargo install --path . ``` ## Common Types ### Transaction Status Transactions can have the following statuses: | Status | Description | | ------------------ | ----------------------------------------- | | `queued` | Transaction is waiting to be broadcast | | `broadcasting` | Transaction is currently being broadcast | | `retry_scheduled` | Broadcast failed, retry is scheduled | | `executed` | Transaction was mined successfully | | `expired` | Transaction's validity window expired | | `invalid` | Transaction was rejected as invalid | | `stale_by_nonce` | Nonce was consumed by another transaction | | `canceled_locally` | Group was canceled via the API | ### TxInfo Object All transaction endpoints return or include `TxInfo` objects: | Field | Type | Description | | ---------------------- | --------- | ------------------------------------------------ | | `chainId` | `number` | Chain ID | | `txHash` | `string` | Transaction hash (hex) | | `type` | `number?` | Transaction type (EIP-2718) | | `sender` | `string` | Sender address (hex) | | `feePayer` | `string?` | Fee payer address if different from sender (hex) | | `nonceKey` | `string` | Nonce key (hex U256) | | `nonce` | `number` | Transaction nonce | | `groupId` | `string?` | Group ID if part of a group (hex) | | `validAfter` | `number?` | Unix timestamp when tx becomes valid | | `validBefore` | `number?` | Unix timestamp when tx expires | | `eligibleAt` | `number` | Unix timestamp when broadcasting begins | | `expiresAt` | `number?` | Unix timestamp when tx expires | | `status` | `string` | Current transaction status | | `nextActionAt` | `number?` | Unix timestamp of next scheduled action | | `attempts` | `number` | Number of broadcast attempts | | `lastError` | `string?` | Last broadcast error message | | `lastBroadcastAt` | `number?` | Unix timestamp of last broadcast | | `receipt` | `object?` | Transaction receipt if executed | | `gas` | `number?` | Gas limit | | `gasPrice` | `string?` | Gas price (for legacy txs) | | `maxFeePerGas` | `string?` | Max fee per gas | | `maxPriorityFeePerGas` | `string?` | Max priority fee per gas | | `input` | `string?` | Transaction input data (hex) | | `calls` | `array?` | Decoded calls for batch transactions | If raw transaction data is not stored (for example after canceling a group locally), fields derived from the raw transaction (`type`, `gas`, `gasPrice`, `maxFeePerGas`, `maxPriorityFeePerGas`, `input`, `calls`) are omitted. ## Groups ### List Groups `GET /v1/groups` Query parameters: | Parameter | Type | Required | Description | | --------- | --------- | -------- | ---------------------------------------------- | | `sender` | `string` | No | Filter by sender address (hex, 20 bytes) | | `chainId` | `number` | No | Filter by chain ID | | `limit` | `number` | No | Max results to return (default: 100, max: 500) | | `active` | `boolean` | No | Return only active (non-terminal) groups | Response example: ```json [ { "chainId": 42431, "groupId": "0x...", "nonceKey": "0x...", "nonceKeyInfo": { "kind": "0x01", "scope": { "encoding": "hex", "value": "0x..." }, "group": { "encoding": "utf8", "value": "my-group" }, "memo": { "encoding": "hex", "value": "0x..." } }, "startAt": 1700000000, "endAt": 1700086400, "nextPaymentAt": 1700043200 } ] ``` #### Response Fields | Field | Type | Description | | --------------- | --------- | ----------------------------------------------- | | `chainId` | `number` | Chain ID | | `groupId` | `string` | Group ID (hex) | | `nonceKey` | `string` | Nonce key (hex U256) | | `nonceKeyInfo` | `object` | Decoded nonce key components | | `startAt` | `number` | Unix timestamp of first transaction eligibility | | `endAt` | `number` | Unix timestamp of last transaction expiration | | `nextPaymentAt` | `number?` | Unix timestamp of next eligible transaction | `endAt` is the largest `eligibleAt` for the group. `nextPaymentAt` is the earliest `eligibleAt` for non-terminal transactions in the group. `active=true` returns groups whose `endAt` is in the future. ### Get Group `GET /v1/senders/{sender}/groups/{groupId}` Get detailed information about a transaction group including member transactions and cancel plan. #### Path Parameters | Parameter | Type | Description | | --------- | -------- | ------------------------------ | | `sender` | `string` | Sender address (hex, 20 bytes) | | `groupId` | `string` | Group ID (hex, 16 bytes) | #### Query Parameters | Parameter | Type | Required | Description | | --------- | -------- | -------- | ------------------------------------------- | | `chainId` | `number` | No | Required if group exists on multiple chains | #### Response ```json { "sender": "0x...", "groupId": "0x...", "nonceKey": "0x...", "nonceKeyInfo": { "kind": "0x01", "scope": { "encoding": "hex", "value": "0x..." }, "group": { "encoding": "utf8", "value": "my-group" }, "memo": { "encoding": "hex", "value": "0x..." } }, "members": [ { "txHash": "0x...", "nonceKey": "0x...", "nonce": 0, "status": "executed" }, { "txHash": "0x...", "nonceKey": "0x...", "nonce": 1, "status": "queued" } ], "cancelPlan": { "nonceKey": "0x...", "nonces": [0, 1, 2], "alreadyInvalidated": false } } ``` #### Response Fields | Field | Type | Description | | ------------------------------- | ---------- | -------------------------------------------- | | `sender` | `string` | Sender address (hex) | | `groupId` | `string` | Group ID (hex) | | `nonceKey` | `string` | Nonce key shared by all members (hex U256) | | `nonceKeyInfo` | `object` | Decoded nonce key components | | `members` | `array` | List of group member transactions | | `members[].txHash` | `string` | Transaction hash (hex) | | `members[].nonceKey` | `string` | Nonce key (hex U256) | | `members[].nonce` | `number` | Transaction nonce | | `members[].status` | `string` | Transaction status | | `cancelPlan` | `object` | Information for canceling the group on-chain | | `cancelPlan.nonceKey` | `string` | Nonce key to use for cancellation | | `cancelPlan.nonces` | `number[]` | Nonces that need to be invalidated | | `cancelPlan.alreadyInvalidated` | `boolean` | True if all nonces are already invalid | To cancel the group, the user must invalidate all nonces for the group's nonce key. All group members share the same nonce key. Transactions are grouped only when `nonceKey` matches the explicit grouping format. ### Cancel Group (Local) `POST /v1/senders/{sender}/groups/{groupId}/cancel` Cancel a group locally. This marks all group transactions as `canceled_locally`, clears stored `raw_tx` data, and removes scheduled retries. This does not affect on-chain state — transactions that have already been broadcast may still be mined. The specification also notes that local cancel: * Removes the group from the scheduler. * Removes the signed transactions from the database while keeping the remaining metadata. #### Headers | Header | Required | Description | | --------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Authorization` | Yes | `Signature ` — Tempo primitive signature bytes over `keccak256(groupId)` signed by the sender. Accepts legacy 65-byte secp256k1 signatures or P256/WebAuthn signatures with a 1-byte type prefix (`0x01`/`0x02`) per the Tempo signature spec. | #### Response ```json { "canceled": 3, "txHashes": ["0x...", "0x...", "0x..."] } ``` #### Errors | Status | Description | | ------ | ------------------------------------------ | | `401` | Missing or invalid authorization signature | | `404` | Group not found | ## Health `GET /health` Returns service status, version, and runtime/config details. If Redis or Postgres are unavailable, the endpoint returns HTTP 503 and `status: "degraded"`. ### Response ```json { "status": "ok", "service": "temprano-watchtower", "version": "0.2.6", "build": { "gitSha": "abc123", "buildTimestamp": "2026-02-03T12:34:56Z" }, "now": 1738612345, "startedAt": 1738610000, "uptimeSeconds": 2345, "chains": [42431], "rpcEndpoints": 2, "scheduler": { "pollIntervalMs": 1000, "leaseTtlSeconds": 30, "maxConcurrency": 10, "retryMinMs": 100, "retryMaxMs": 1000, "expirySoonWindowSeconds": 3600, "expirySoonRetryMaxMs": 5000 }, "watcher": { "pollIntervalMs": 1000, "useWebsocket": false }, "broadcaster": { "fanout": 2, "timeoutMs": 500 }, "api": { "maxBodyBytes": 1048576 }, "dependencies": { "database": { "ok": true }, "redis": { "ok": true } } } ``` `build.gitSha` and `build.buildTimestamp` are omitted when not provided at build time. ## API Reference Base path for REST endpoints is `/v1`. JSON-RPC requests are served on `/rpc`. This section consolidates the REST and JSON-RPC APIs described in the README and the specification. ## JSON-RPC ### Submit Raw Transaction `POST /rpc` Accepts JSON-RPC 2.0 `eth_sendRawTransaction` requests. The service extracts the `chainId` from the transaction, validates it against configured chains, and stores it for broadcasting. #### Request ```json { "jsonrpc": "2.0", "id": 1, "method": "eth_sendRawTransaction", "params": ["0x...signed_tx_hex..."] } ``` | Field | Type | Required | Description | | --------- | -------- | -------- | ------------------------------------------------ | | `jsonrpc` | `string` | Yes | Must be `"2.0"` | | `id` | `any` | No | Request identifier | | `method` | `string` | Yes | Must be `"eth_sendRawTransaction"` | | `params` | `array` | Yes | Array with single hex-encoded signed transaction | #### Response (Success) ```json { "jsonrpc": "2.0", "id": 1, "result": "0x...tx_hash..." } ``` #### Response (Error) ```json { "jsonrpc": "2.0", "id": 1, "error": { "code": -32602, "message": "unsupported chainId 1" } } ``` | Code | Meaning | | -------- | --------------------------------------------------------- | | `-32600` | Invalid request (malformed JSON-RPC) | | `-32601` | Method not found | | `-32602` | Invalid params (bad tx, unsupported chain, expired, etc.) | | `-32603` | Internal error | ## Transactions ### Submit Transactions (Batch) `POST /v1/transactions` Submit one or more signed transactions for broadcasting. #### Request ```json { "chainId": 42431, "transactions": ["0x...signed_tx_1...", "0x...signed_tx_2..."] } ``` | Field | Type | Required | Description | | -------------- | ---------- | -------- | ---------------------------------------- | | `chainId` | `number` | Yes | Target chain ID | | `transactions` | `string[]` | Yes | Array of hex-encoded signed transactions | #### Response ```json { "results": [ { "ok": true, "txHash": "0x...", "sender": "0x...", "nonceKey": "0x...", "nonce": 5, "groupId": "0x...", "eligibleAt": 1700000000, "expiresAt": 1700003600, "status": "queued", "alreadyKnown": false } ] } ``` | Field | Type | Description | | ------------------------ | ---------- | ----------------------------------------------- | | `results` | `array` | Array of results, one per submitted transaction | | `results[].ok` | `boolean` | Whether submission succeeded | | `results[].txHash` | `string?` | Transaction hash (hex) | | `results[].sender` | `string?` | Sender address (hex) | | `results[].nonceKey` | `string?` | Nonce key (hex U256) | | `results[].nonce` | `number?` | Transaction nonce | | `results[].groupId` | `string?` | Group ID if using group nonce key (hex) | | `results[].eligibleAt` | `number?` | Unix timestamp when broadcasting begins | | `results[].expiresAt` | `number?` | Unix timestamp when tx expires | | `results[].status` | `string?` | Initial transaction status | | `results[].alreadyKnown` | `boolean?` | True if tx was already in the system | | `results[].error` | `string?` | Error message if `ok` is false | #### Behavior * Hash-based idempotency: `(chainId, txHash)` is unique, and resubmission returns the existing record. * Static validation performed at ingest: decoding, signature verification, and not already expired. * Dynamic validity (nonce, balance) is handled by the scheduler. ### Get Transaction `GET /v1/transactions/{txHash}` Retrieve a single transaction by hash. #### Path Parameters | Parameter | Type | Description | | --------- | -------- | -------------------------------- | | `txHash` | `string` | Transaction hash (hex, 32 bytes) | #### Query Parameters | Parameter | Type | Required | Description | | --------- | -------- | -------- | ------------------------------------------------------------- | | `chainId` | `number` | No | Filter by chain ID (required if tx exists on multiple chains) | #### Response Returns a `TxInfo` object. #### Errors | Status | Description | | ------ | ------------------------------- | | `400` | Invalid transaction hash format | | `404` | Transaction not found | ### Cancel Transaction (Mark Stale by Nonce) `DELETE /v1/transactions/{txHash}` Mark a transaction as `stale_by_nonce` when its nonce has been consumed by another transaction on-chain. #### Query Parameters | Parameter | Type | Required | Description | | --------- | -------- | -------- | ------------------ | | `chainId` | `number` | No | Filter by chain ID | #### Response Returns the updated `TxInfo` object with status `stale_by_nonce`. #### Errors | Status | Description | | ------ | --------------------------------------------------- | | `400` | Transaction nonce has not been invalidated on-chain | | `400` | Transaction is already in a terminal state | | `404` | Transaction not found | #### Behavior * Fetches the current nonce for the transaction's nonce key. * If the current nonce is higher than the transaction nonce, marks the transaction as `stale_by_nonce`. * Otherwise returns `400`. ### List Transactions `GET /v1/transactions` Query transactions with optional filters. #### Query Parameters | Parameter | Type | Required | Description | | ----------- | --------- | -------- | ------------------------------------------------------------------------ | | `chainId` | `number` | No | Filter by chain ID | | `sender` | `string` | No | Filter by sender address (hex, 20 bytes) | | `groupId` | `string` | No | Filter by group ID (hex, 16 bytes) | | `ungrouped` | `boolean` | No | Return only transactions without a group (cannot combine with `groupId`) | | `status` | `string` | No | Filter by status (can be repeated for multiple statuses) | | `limit` | `number` | No | Max results to return (default: 100, max: 500) | #### Example ``` GET /v1/transactions?sender=0x1234...&status=queued&status=retry_scheduled&chainId=42431&limit=50 ``` #### Response Returns an array of `TxInfo` objects.