Temprano Watchtower

Tempo Watchtower is a Rust service that accepts signed Tempo transactions, stores them durably, and broadcasts them throughout their validity window.

What It Does

• Accepts raw signed Tempo transactions (same format as eth_sendRawTransaction).
• Stores transactions durably for guaranteed delivery.
• Broadcasts as soon as transactions are valid and retries throughout their validity window.
• Groups transactions by nonce key and allows local group cancellation.
• Exposes JSON-RPC and REST APIs for ingestion and querying.

Hosted Endpoint

Watchtower is available at https://watchtower.temprano.io.

Where To Go Next

• Start with the Getting Started section for installation and configuration.
• Use the API Reference for request and response formats.
• Read Concepts and System Design for guarantees, grouping logic, and internal behavior.

Getting Started

This section covers installation requirements, configuration, and how to run and test the service locally.

Installation

See the Dependencies page for required services and optional components like a reverse proxy or supervisor.

Precompiled Binaries

Download the appropriate archive for your platform from the GitHub releases page, then:

1. Unpack the archive.
2. Place the tempo-watchtower binary somewhere on your PATH (or keep it alongside your deployment artifacts).

Build From Source

This project targets Rust 2024 edition and can be built using the latest stable release.

Install straight from the GitHub repo in one step:

cargo install --git https://github.com/arvina-tech/tempo-watchtower.git --locked

Otherwise, clone the repo and build the binary from source. From the repository root:

cargo build --release

The binary will be at target/release/tempo-watchtower. You can run it directly:

./target/release/tempo-watchtower

Or install it into your Cargo bin directory:

cargo install --path .

Dependencies

This page describes the external services and tooling needed to run Tempo Watchtower in development or production.

PostgreSQL

PostgreSQL stores transactions, scheduler state, and metadata. You must create a database for Tempo Watchtower and provide the credentials when configuring.

Redis

Redis is used to improve scheduling performance. The database remains the source of truth; Redis can be rebuilt from Postgres.

Reverse Proxy (optional)

A reverse proxy is recommended when exposing the service to the Internet. It can terminate TLS, apply rate limits, and manage access controls in front of the Watchtower API. We provide a sample configuration for nginx as part of this repo.

Supervisor (optional)

A process supervisor is recommended for production deployments to keep the service running and to manage restarts on failure. We provide a sample configuration for supervisor as part of this repo.

Configuration

The service reads config.toml by default. You can override the path with the CONFIG_PATH environment variable. The config file supports environment variable interpolation (for example ${DB_HOST}).

Sample config.toml

[server]
bind = "0.0.0.0:8080"

[database]
url = "postgres://${DB_USER}@${DB_HOST}:${DB_PORT}/${DB_NAME}"

[redis]
url = "redis://${REDIS_HOST}:${REDIS_PORT}/${REDIS_DB}"

[rpc]
# Per-chain RPC endpoints. Keys are chain IDs.
[rpc.chains]
"42431" = ["${RPC_URL}"]

[scheduler]
poll_interval_ms = 200
lease_ttl_seconds = 30
max_concurrency = 50
retry_min_ms = 250
retry_max_ms = 900000
expiry_soon_window_seconds = 3600
expiry_soon_retry_max_ms = 5000

[broadcaster]
fanout = 2
timeout_ms = 2000

[watcher]
poll_interval_ms = 1500
use_websocket = true

[api]
max_body_bytes = 1048576

server

• bind: Address and port to listen on (for example 0.0.0.0:8080).

database

• url: PostgreSQL connection string. Environment variables may be interpolated.

redis

• url: Redis connection string. Environment variables may be interpolated.

rpc

• chains: Map of chain IDs to one or more RPC URLs for each chain. Chain IDs are string keys in the TOML file, and each value is an array of URLs. These endpoints are used by the broadcaster and watcher.

scheduler

• poll_interval_ms: How often the scheduler scans for due work.
• lease_ttl_seconds: Duration of DB-backed leases used for multi-replica scheduling.
• max_concurrency: Maximum number of concurrent scheduler tasks.
• retry_min_ms: Minimum delay between retry attempts near eligibility.
• retry_max_ms: Maximum backoff delay between retry attempts.
• expiry_soon_window_seconds: Window before expiry during which retry cadence is adjusted.
• expiry_soon_retry_max_ms: Maximum retry delay when a transaction is nearing expiry.

broadcaster

• fanout: Number of RPC endpoints to broadcast to in parallel.
• timeout_ms: Per-endpoint broadcast timeout in milliseconds.

watcher

• poll_interval_ms: How often the watcher polls for updates when websocket subscriptions are unavailable or disabled.
• use_websocket: Whether to use websocket subscriptions when supported by the RPC endpoint.

api

• max_body_bytes: Maximum request body size accepted by the API.

API Reference

Base path for REST endpoints is /v1. JSON-RPC requests are served on /rpc.

This section consolidates the REST and JSON-RPC APIs described in the README and the specification.

Common Types

Transaction Status

Transactions can have the following statuses:

TxInfo Object

All transaction endpoints return or include TxInfo objects:

If raw transaction data is not stored (for example after canceling a group locally), fields derived from the raw transaction (type, gas, gasPrice, maxFeePerGas, maxPriorityFeePerGas, input, calls) are omitted.

JSON-RPC

Submit Raw Transaction

POST /rpc

Accepts JSON-RPC 2.0 eth_sendRawTransaction requests. The service extracts the chainId from the transaction, validates it against configured chains, and stores it for broadcasting.

Request

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "eth_sendRawTransaction",
  "params": ["0x...signed_tx_hex..."]
}

Response (Success)

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": "0x...tx_hash..."
}

Response (Error)

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": -32602,
    "message": "unsupported chainId 1"
  }
}

Transactions

Submit Transactions (Batch)

POST /v1/transactions

Submit one or more signed transactions for broadcasting.

Request

{
  "chainId": 42431,
  "transactions": ["0x...signed_tx_1...", "0x...signed_tx_2..."]
}

Response

{
  "results": [
    {
      "ok": true,
      "txHash": "0x...",
      "sender": "0x...",
      "nonceKey": "0x...",
      "nonce": 5,
      "groupId": "0x...",
      "eligibleAt": 1700000000,
      "expiresAt": 1700003600,
      "status": "queued",
      "alreadyKnown": false
    }
  ]
}

Behavior

• Hash-based idempotency: (chainId, txHash) is unique, and resubmission returns the existing record.
• Static validation performed at ingest: decoding, signature verification, and not already expired.
• Dynamic validity (nonce, balance) is handled by the scheduler.

Get Transaction

GET /v1/transactions/{txHash}

Retrieve a single transaction by hash.

Path Parameters

Query Parameters

Response

Returns a TxInfo object.

Errors

Cancel Transaction (Mark Stale by Nonce)

DELETE /v1/transactions/{txHash}

Mark a transaction as stale_by_nonce when its nonce has been consumed by another transaction on-chain.

Query Parameters

Response

Returns the updated TxInfo object with status stale_by_nonce.

Errors

Behavior

• Fetches the current nonce for the transaction’s nonce key.
• If the current nonce is higher than the transaction nonce, marks the transaction as stale_by_nonce.
• Otherwise returns 400.

List Transactions

GET /v1/transactions

Query transactions with optional filters.

Query Parameters

Example

GET /v1/transactions?sender=0x1234...&status=queued&status=retry_scheduled&chainId=42431&limit=50

Response

Returns an array of TxInfo objects.

Groups

List Groups

GET /v1/groups

Query parameters:

Response example:

[
  {
    "chainId": 42431,
    "groupId": "0x...",
    "nonceKey": "0x...",
    "nonceKeyInfo": {
      "kind": "0x01",
      "scope": { "encoding": "hex", "value": "0x..." },
      "group": { "encoding": "utf8", "value": "my-group" },
      "memo": { "encoding": "hex", "value": "0x..." }
    },
    "startAt": 1700000000,
    "endAt": 1700086400,
    "nextPaymentAt": 1700043200
  }
]

Response Fields

endAt is the largest eligibleAt for the group. nextPaymentAt is the earliest eligibleAt for non-terminal transactions in the group. active=true returns groups whose endAt is in the future.

Get Group

GET /v1/senders/{sender}/groups/{groupId}

Get detailed information about a transaction group including member transactions and cancel plan.

Path Parameters

Query Parameters

Response

{
  "sender": "0x...",
  "groupId": "0x...",
  "nonceKey": "0x...",
  "nonceKeyInfo": {
    "kind": "0x01",
    "scope": { "encoding": "hex", "value": "0x..." },
    "group": { "encoding": "utf8", "value": "my-group" },
    "memo": { "encoding": "hex", "value": "0x..." }
  },
  "members": [
    {
      "txHash": "0x...",
      "nonceKey": "0x...",
      "nonce": 0,
      "status": "executed"
    },
    {
      "txHash": "0x...",
      "nonceKey": "0x...",
      "nonce": 1,
      "status": "queued"
    }
  ],
  "cancelPlan": {
    "nonceKey": "0x...",
    "nonces": [0, 1, 2],
    "alreadyInvalidated": false
  }
}

Response Fields

To cancel the group, the user must invalidate all nonces for the group’s nonce key. All group members share the same nonce key. Transactions are grouped only when nonceKey matches the explicit grouping format.

Cancel Group (Local)

POST /v1/senders/{sender}/groups/{groupId}/cancel

Cancel a group locally. This marks all group transactions as canceled_locally, clears stored raw_tx data, and removes scheduled retries. This does not affect on-chain state — transactions that have already been broadcast may still be mined.

The specification also notes that local cancel:

• Removes the group from the scheduler.
• Removes the signed transactions from the database while keeping the remaining metadata.

Headers

Response

{
  "canceled": 3,
  "txHashes": ["0x...", "0x...", "0x..."]
}

Errors

Health

GET /health

Returns service status, version, and runtime/config details. If Redis or Postgres are unavailable, the endpoint returns HTTP 503 and status: "degraded".

Response

{
  "status": "ok",
  "service": "tempo-watchtower",
  "version": "0.2.6",
  "build": {
    "gitSha": "abc123",
    "buildTimestamp": "2026-02-03T12:34:56Z"
  },
  "now": 1738612345,
  "startedAt": 1738610000,
  "uptimeSeconds": 2345,
  "chains": [42431],
  "rpcEndpoints": 2,
  "scheduler": {
    "pollIntervalMs": 1000,
    "leaseTtlSeconds": 30,
    "maxConcurrency": 10,
    "retryMinMs": 100,
    "retryMaxMs": 1000,
    "expirySoonWindowSeconds": 3600,
    "expirySoonRetryMaxMs": 5000
  },
  "watcher": {
    "pollIntervalMs": 1000,
    "useWebsocket": false
  },
  "broadcaster": {
    "fanout": 2,
    "timeoutMs": 500
  },
  "api": {
    "maxBodyBytes": 1048576
  },
  "dependencies": {
    "database": { "ok": true },
    "redis": { "ok": true }
  }
}

build.gitSha and build.buildTimestamp are omitted when not provided at build time.

Concepts

Delivery Semantics

• Broadcast starts at valid_after (or immediately if unset).
• Retry continues until:
  • the transaction is mined,
  • the validity window expires,
  • the transaction is provably invalid,
  • or the user cancels locally.

The transaction can fail to be executed for a number of reasons. A non-exhaustive list is:

• Sender out-of-funds
• Fees too low for transaction to be included
• RPC failure

The Watchtower will always retry to send the transaction unless it becomes invalid.

Transaction Format

The service accepts the same payload as Ethereum raw transaction submission:

rawTx = "0x" + hex-encoded signed Tempo tx bytes

This is the output of:

• viem.signTransaction()
• Any Tempo wallet signer

The server performs:

• decoding
• signature validation
• field extraction
• scheduling

Notes:

• Any valid Tempo transaction is accepted, including ones with custom nonce keys.
• The service accepts JSON-RPC eth_sendRawTransaction payloads as-is.

Grouping

Group Identity

Users may run multiple concurrent groups (for example, multiple subscriptions). Group identity is:

GroupKey = (sender_address, group_id)

Each sender can have arbitrarily many groups.

How Grouping Works

Grouping is based solely on the transaction nonce key. Transaction input data and memo contents are ignored.

For every accepted transaction, the group fields are derived from the nonce key:

• group_id = first 16 bytes of keccak256(nonce_key_bytes)

All transactions that share a nonce key are placed in the same group for a given sender.

Transactions are grouped only when nonceKey matches the explicit grouping format described below.

Nonce Key Format

Nonce keys are 32-byte values. Transactions are grouped only when the nonce key matches this explicit format.

Binary Layout (Big-Endian)

0..3   magic     = 0x4E4B4731   // "NKG1"
4      version   = 0x01
5      kind      = enum (purpose)
6..7   flags     = u16
8..15  scope_id  = u64
16..19 group_id  = u32
20..31 memo      = 12 bytes

Flags Encoding (u16)

bits 0..1   scope_id encoding
bits 2..3   group_id encoding
bits 4..5   memo encoding
bits 6..15  reserved (must be 0)

Encoding values (same for scope_id, group_id, memo):

00 = numeric/raw
01 = ASCII
10 = reserved (undefined)
11 = reserved (undefined)

Display / Interpretation Rules

• numeric/raw:
  • scope_id and group_id are interpreted as unsigned integers (big-endian) and displayed in decimal.
  • memo is rendered as hex (full 12 bytes, with 0x prefix).
• ASCII:
  • Bytes must be printable 7-bit ASCII (0x20..0x7E).
  • Trailing 0x00 bytes are trimmed for display.
  • If any non-printable byte is present, render as hex.

Example

magic/version/kind/flags = 4e4b4731 01 02 0001
scope_id (ASCII) = 504159524f4c4c00  // "PAYROLL\0"
group_id (numeric) = 00000f42
memo (ASCII) = 4a414e2d323032360000000000  // "JAN-2026"

Display:

kind=0x02, scope=PAYROLL, group=3906, memo=JAN-2026

State Machine

Transactions follow this lifecycle:

queued → broadcasting ↔ retry_scheduled → terminal

Terminal states:

• executed
• expired
• invalid (provably invalid)
• stale_by_nonce
• canceled_locally

Validity Logic

Time Window

• eligibleAt = valid_after || now
• expiresAt = valid_before || ∞

Provably Invalid (Stop Early)

• malformed tx
• invalid sender signature
• invalid fee payer signature
• expired at ingest
• nonce mismatch (if transaction is replaced on-chain)

Dynamic (Retry Until Expiry)

• insufficient balance
• fee token selection failure
• temporary RPC errors

System Design

Scheduler

Behavior

• Due transactions are pulled by next_action_at.
• Database-backed leasing is used for multi-replica safety.
• Redis ZSET is used as an accelerator only.
• Guaranteed retry continues until expiry.

Retry Strategy

• Near eligibility: 250–500ms attempts.
• Backoff up to 5s max.
• Endpoint rotation.
• Health scoring per RPC endpoint.

Broadcaster

• Fan-out to multiple Tempo RPC endpoints.
• Tracks:
  • accepted
  • rejected (with reason)
  • timeout
• Stops only on terminal conditions.

Chain Watcher

Tracks:

• (sender, nonce_key) → current_nonce
• receipts for known transactions

Transitions:

• mined → terminal
• nonce advanced past tx → stale_by_nonce

Notes:

• The watcher uses websocket subscriptions when available and falls back to polling.

Storage Model

txs Table

Fields:

• chain_id
• tx_hash (unique)
• raw_tx
• sender
• fee_payer
• nonce_key
• nonce
• valid_after
• valid_before
• eligible_at
• expires_at
• status
• group_id (16 bytes nullable)
• next_action_at
• leasing fields
• timestamps

Indexes:

• (chain_id, tx_hash) unique
• (sender, group_id)
• (status, next_action_at)

Redis Acceleration

Keys:

• watchtower:ready:{chain} → ZSET(tx_hash, eligibleAt)
• watchtower:retry:{chain} → ZSET(tx_hash, nextRetryAt)
• Optional inflight/lease keys

Redis is rebuildable from the database.

Redis is used as a scheduling accelerator; the database remains the source of truth.

Observability

Metrics:

• ingest rate
• queue depth
• retry counts
• success/failure rates
• time-to-mined

Tracing:

• full transaction lifecycle

Logs:

• structured per txHash

Security

• request size limits
• strict decoding
• rate limiting
• no signing
• no private key handling