> ## Documentation Index
> Fetch the complete documentation index at: https://docs.boundless.network/llms.txt
> Use this file to discover all available pages before exploring further.
# Broker Configuration & Operation
> Optimizing the Broker for optimal performance on the market.
## Overview
The Broker is a service that runs within the [Bento](/provers/proving-stack#what-is-bento) proving stack. It is responsible for market interactions including bidding on jobs, locking them, issuing job requests to the Bento proving cluster, and submitting proof fulfillments onchain.
## Broker Configuration
Broker will live-reload the `broker.toml` when it changes. In most cases, you will not need to restart the Broker for the configuration to take effect.
Broker configuration is primarily managed through the `broker.toml` file in the Boundless directory. This file is mounted into the Broker container and it is used to configure the Broker daemon.
### Deposit / Balance
The Boundless market requires funds (ZKC) deposited as collateral before a prover can bid on requests.
Brokers must first deposit some ZKC into the market contract to fund their account.
These funds cover collateral during lock-in.
It is recommend that a broker keep a balance on the market >= `max_collateral` (configured via broker.toml).
#### Deposit Collateral to the Market
You will need the Boundless CLI installed to deposit/check your balance.
Please see [Installing the Boundless CLI](/developers/tooling/cli#installation) for instructions.
```bash Terminal theme={null}
export RPC_URL=
export PRIVATE_KEY=
# Example: 'prover deposit-collateral 100'
boundless prover deposit-collateral
```
#### Check Current Collateral Balance
```bash Terminal theme={null}
export RPC_URL=
export PRIVATE_KEY=
boundless prover balance-collateral [wallet_address]
```
You can omit the `PRIVATE_KEY` environment variable here and specify your `wallet_address` as a optional parameter to the `balance` command, i.e., `account balance 0x000....`
### Settings in Broker.toml
Quotation marks matter in TOML so please pay particular attention to the quotation marks for config values.
Below are all `broker.toml` settings organized by section:
#### \[market] Settings
| Setting | Description |
| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| min\_mcycle\_price | The minimum price per mega-cycle (i.e. million RISC-V cycles) for the broker to attempt to lock an order. Accepts `" ETH"` (e.g., `"0.00002 ETH"`) or `" USD"` (e.g., `"0.02 USD"`); USD values are converted to ETH at runtime via the price oracle. Plain numbers without a suffix still work and default to ETH for backward compatibility. This value is used for both primary orders (compared against the order's ETH price per mcycle) and secondary fulfillment orders (converted to ZKC via price oracle for collateral-based comparisons). |
| expected\_probability\_win\_secondary\_fulfillment | Expected probability (as a percentage) of winning a secondary fulfillment proof race. When another prover fails to fulfill an order and is slashed, multiple provers may race to fulfill the order for the collateral reward. This setting scales the expected reward of those lock-expired orders by the configured percentage before comparing against `min_mcycle_price`. Default: `50`. Values below 100 discount the reward (conservative — fewer secondary orders picked up), 100 means no discount, values above 100 boost the reward (aggressive — more secondary orders picked up). |
| priority\_requestor\_addresses | Optional priority requestor addresses that can bypass the mcycle limit and max input size limit. If enabled, the order will be preflighted without constraints. |
| priority\_requestor\_lists | URLs to fetch requestor priority lists from. Requestor priority lists specify requestors that the broker should prioritize for proving. Requestors on these lists are considered more likely to request useful work with profitable pricing. Priority requestors are prioritized when there is a surplus of demand: their requests will be preflighted first, locked first (in situations where multiple orders exceed minimum lock pricing), and configuration for max\_mcycle\_limit and max\_file\_size will be skipped. These lists will be periodically refreshed and merged with priority\_requestor\_addresses. |
| peak\_prove\_khz | Estimated peak performance of the proving cluster, in kHz. Used to estimate proving capacity and accept only as much work as your prover cluster can handle. Estimates can be derived from benchmarking using the Boundless CLI. See [Benchmarking Bento](#benchmarking-bento). |
| max\_mcycle\_limit | Optional max cycles (in mcycles). Orders over this max\_cycles will be skipped after preflight. |
| max\_journal\_bytes | Max journal size in bytes. Orders that produce a journal larger than this size in preflight will be skipped. Since journals must be posted onchain to complete an order, an excessively large journal may prevent completion of a request. |
| min\_deadline | Min seconds left before the deadline to consider bidding on a request. If there is not enough time left before the deadline, the prover may not be able to complete proving of the request and finalize the batch for publishing before expiration. |
| lookback\_blocks | On startup, the number of blocks to look back for possible open orders. |
| max\_collateral | Maximum collateral amount that the broker will use to lock orders. Accepts `" ZKC"` (e.g., `"200 ZKC"`) or `" USD"` (e.g., `"100 USD"`); USD values are converted to ZKC at runtime via the price oracle. Requests that require a higher collateral amount than this will be skipped. |
| max\_file\_size | Max input / image file size allowed for downloading from request URLs. |
| max\_fetch\_retries | Max retries for fetching input / image contents from URLs. |
| assessor\_default\_image\_url | Default URL for assessor image. This URL will be tried first before falling back to the image URL from the boundless market contract. |
| set\_builder\_default\_image\_url | Default URL for set builder image. This URL will be tried first before falling back to the image URL from the set verifier contract. |
| max\_concurrent\_proofs | Maximum number of concurrent proofs that can be processed at once. Used to limit proof tasks spawned to prevent overwhelming the system. |
| max\_concurrent\_preflights | Maximum number of orders to concurrently preflight. Used to limit preflight tasks spawned to prevent overwhelming the system. Recommended default: `8` to be able to follow all market orders. |
| order\_pricing\_priority | Determines how orders are prioritized for pricing. Options: "random" (default, process orders in random order), "observation\_time" (prioritize orders in the order they were observed, FIFO), "shortest\_expiry" (prioritize orders with the earliest deadline first). |
| order\_commitment\_priority | Determines how orders are prioritized when committing to prove them. Options: "cycle\_price" (default, prioritize orders with the highest ETH price per cycle), "random" (process orders in random order), "shortest\_expiry" (prioritize orders with the earliest deadline first), "tightest\_deadline" (prioritize orders whose deadline is closest, giving preference to the most time-urgent work), "price" (prioritize orders with the highest ETH payment regardless of cycle count). |
| max\_critical\_task\_retries | Max critical task retries on recoverable failures. The broker service has a number of subtasks. Some are considered critical. If a task fails, it will be retried, but after this number of retries, the process will exit. |
| allow\_client\_addresses | Optional allow list for customer address. If enabled, all requests from clients not in the allow list are skipped. |
| deny\_requestor\_addresses | Optional deny list for requestor address. If enabled, all requests from clients in the deny list are skipped. |
| lockin\_priority\_gas | Optional additional gas to add to the transaction for lockinRequest, good for increasing the priority if competing with multiple provers during the same block. |
| balance\_warn\_threshold | Optional balance warning threshold (in native token). If the submitter balance drops below this the broker will issue warning logs. |
| balance\_error\_threshold | Optional balance error threshold (in native token). If the submitter balance drops below this the broker will issue error logs. |
| collateral\_balance\_warn\_threshold | Optional collateral balance warning threshold (in collateral tokens). If the collateral balance drops below this the broker will issue warning logs. |
| collateral\_balance\_error\_threshold | Optional collateral balance error threshold (in collateral tokens). If the collateral balance drops below this the broker will issue error logs. |
| cache\_dir | Optional cache directory for storing downloaded images and inputs. If not set, files will be re-downloaded every time. |
| lockin\_gas\_estimate | Gas estimate for lockin call. Used for estimating the gas costs associated with an order during pricing. If not set a conservative default will be used. |
| fulfill\_gas\_estimate | Gas estimate for fulfill call. Used for estimating the gas costs associated with an order during pricing. If not set a conservative default will be used. |
| groth16\_verify\_gas\_estimate | Gas estimate for proof verification using the RiscZeroGroth16Verifier. Used for estimating the gas costs associated with an order during pricing. If not set a conservative default will be used. |
| max\_order\_expiry\_secs | Maximum order expiry duration in seconds (current time to order expiry time). Orders exceeding this duration are skipped. Prevents orders with long deadlines from tying up resources or storing inputs beyond MinIO's TTL. |
| min\_mcycle\_price\_overrides | Optional per-requestor and per-selector overrides for `min_mcycle_price`. Allows fine-grained pricing control so you can set different minimum prices for specific requestors or program selectors. See example below. |
| telemetry\_mode | Controls broker telemetry reporting. `"full"` (default) sends anonymized health and performance metrics to help the Boundless team diagnose issues across the network. No sensitive data (private keys, wallet balances) is collected. Set to `"logsonly"` to disable remote reporting and only emit local debug logs. |
#### \[prover] Settings
| Setting | Description |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| status\_poll\_retry\_count | Number of retries to poll for proving status. Provides a little durability for transient failures. |
| status\_poll\_ms | Polling interval to monitor proving status (in millisecs). |
| req\_retry\_count | Number of retries to query a prover backend for on failures. Used for API requests to a prover backend, creating sessions, preflighting, uploading images, etc. Provides a little durability for transient failures. |
| req\_retry\_sleep\_ms | Number of milliseconds to sleep between retries. |
| proof\_retry\_count | Number of retries for running the entire proof generation process. This is separate from the request retry count, as the proving process is a multi-step process involving multiple API calls to create a proof job and then polling for the proof job to complete. |
| proof\_retry\_sleep\_ms | Number of milliseconds to sleep between proof retries. |
| set\_builder\_guest\_path | Set builder guest program (ELF) path. When using a durable deploy, set this to the published current SOT guest program path on the system. |
| assessor\_set\_guest\_path | Assessor ELF path. |
| reaper\_interval\_secs | Interval for checking expired committed orders (in seconds). This is the interval at which the ReaperTask will check for expired orders and mark them as failed. If not set, it defaults to 60 seconds. |
| reaper\_grace\_period\_secs | Grace period before marking expired orders as failed (in seconds). This provides a buffer time after an order expires before the reaper marks it as failed. This helps prevent race conditions with the aggregator that might be processing the order. If not set, it defaults to 10800 seconds (3 hours). |
#### \[batcher] Settings
| Setting | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| batch\_max\_time | Max batch duration before publishing (in seconds). |
| min\_batch\_size | Batch size (in proofs) before publishing. |
| block\_deadline\_buffer\_secs | Batch blocktime buffer. Number of seconds before the lowest block deadline in the order batch to flush the batch. This should be approximately snark\_proving\_time \* 2. |
| txn\_timeout | Timeout, in seconds for transaction confirmations. |
| single\_txn\_fulfill | Use the single TXN submission that batches submit\_merkle / fulfill\_batch into a single transaction. Requires the `submitRootAndFulfill` method be present on the deployed contract. |
| withdraw | Whether to withdraw from the prover balance when fulfilling. |
| batch\_poll\_time\_ms | Polling time, in milliseconds. The time between polls for new orders to aggregate and how often to check for batch finalize conditions. |
| batch\_max\_journal\_bytes | Max combined journal size (in bytes) that once exceeded will trigger a publish. |
| batch\_max\_fees | Max batch fees (in ETH) before publishing. |
| max\_submission\_attempts | Number of attempts to make to submit a batch before abandoning. |
#### \[price\_oracle] Settings
The price oracle fetches live ETH/USD and ZKC/USD exchange rates, enabling broker config values to be specified in USD. Chainlink (on-chain, ETH/USD only) and CoinGecko (off-chain, both pairs) are enabled by default; CoinMarketCap is optionally available with an API key. Prices from multiple sources are aggregated using the configured `aggregation_mode`.
| Setting | Description |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| eth\_usd | Price source for ETH/USD. `"auto"` (default) uses live oracle prices. Set to a static number (e.g., `"2500.00"`) to fix the price and disable live fetching. |
| zkc\_usd | Price source for ZKC/USD. `"auto"` (default) uses live oracle prices. Set to a static number (e.g., `"1.00"`) to fix the price and disable live fetching. |
| refresh\_interval\_secs | How often to refresh prices from sources, in seconds (default: `60`). |
| aggregation\_mode | How to combine prices from multiple sources: `"median"` (default), `"priority"` (use first available source), or `"average"`. |
| max\_secs\_without\_price\_update | Maximum seconds without a successful price update before the broker logs an error (default: `43200` / 12h). Set to `0` to disable this check. |
**Example configuration with USD-denominated pricing:**
```toml theme={null}
[market]
min_mcycle_price = "0.02 USD"
max_collateral = "100 USD"
expected_probability_win_secondary_fulfillment = 50
[price_oracle]
eth_usd = "auto"
zkc_usd = "auto"
```
**Example per-requestor and per-selector pricing overrides:**
```toml theme={null}
[market]
min_mcycle_price = "0.02 USD"
# Override min_mcycle_price for a specific requestor address
[[market.min_mcycle_price_overrides]]
requestor = "0x1234...abcd"
min_mcycle_price = "0.01 USD"
# Override min_mcycle_price for a specific program selector
[[market.min_mcycle_price_overrides]]
selector = "aabbccdd"
min_mcycle_price = "0.03 USD"
```
### Chain Monitor
v2.0 introduces a rewritten chain-watching layer, `ChainMonitorV2`, that replaces the legacy two-service architecture (`ChainMonitorService` + `MarketMonitor`). Instead of polling `eth_getLogs` for market events on every tick, `ChainMonitorV2` uses `eth_getBlockReceipts` to fetch all receipts per block in a single call, then filters for market events locally. This reduces the number of RPC calls, especially on chains like Base where `eth_getLogs` is expensive or rate-limited.
In steady state, `ChainMonitorV2` requires roughly **2 RPC requests per block**:
* `eth_getBlockByNumber` to follow the chain head and get the base fee
* `eth_getBlockReceipts` to get all receipts and transaction fee data
It also performs local EIP-1559 gas estimation from receipt data, removing the need for separate `eth_feeHistory` calls.
On startup, `ChainMonitorV2` uses adaptive log retrieval with binary-search chunking to auto-discover the maximum block range accepted by the RPC provider, catching up on any missed events for open orders. This means the broker won't miss events if it restarts or experiences downtime.
#### Selecting the chain monitor
Selection is per-chain via the `rpc_mode` field under `[market]` in `broker.toml` (or in `chain-overrides/broker.{chain_id}.toml`):
```toml broker.toml theme={null}
[market]
rpc_mode = "auto" # or "v2" or "legacy"
```
* `"auto"` (default): chain-specific default. Uses `v2` for Base and most chains, `legacy` for Taiko (167000), where the public RPC rate-limits `eth_getBlockReceipts`.
* `"v2"`: pin to `ChainMonitorV2`.
* `"legacy"`: pin to the old `ChainMonitorService` + `MarketMonitor` pair (`eth_getLogs`-based).
You can tune the RPC request timeout (default 15s) to cut off hanging requests so the retry and fallback layers can activate:
```bash Terminal theme={null}
export BROKER_EXTRA_ARGS="--rpc-request-timeout 15"
just prover
```
#### Sequential Fallback Transport
`ChainMonitorV2` uses a `SequentialFallbackTransport` that tries RPC providers in priority order (rather than in parallel), minimizing calls to paid or metered fallback endpoints. When multiple RPC URLs are configured, the transport includes health tracking: it skips providers after consecutive failures and periodically retries them to detect recovery.
The retry and fallback layers are stacked so that on a single RPC failure, the sequential fallback immediately tries the next URL. The outer retry layer only kicks in once all URLs have been exhausted.
#### Analyzing RPC Usage
To analyze RPC call patterns, enable debug logging for the relevant modules and pipe the output to a log file. An [analysis script](https://github.com/boundless-xyz/protocol-experiments/tree/main/broker-rpc) is available to parse these logs:
```bash Terminal theme={null}
RISC0_DEV_MODE=1 \
RUST_LOG=INFO,broker::chain_monitor_v2=debug,broker::rpcmetrics=debug \
BROKER_EXTRA_ARGS="--listen-only" \
just prover 2>&1 | tee rpc_metrics.log
```
Not all RPC providers support the `eth_getBlockReceipts` method that `ChainMonitorV2` relies on. If your provider doesn't support it, set `rpc_mode = "legacy"` for the affected chain. Free public RPCs are unreliable; use a dedicated RPC provider for production.
## Broker Operation
```txt Terminal theme={null}
2024-10-23T14:37:37.364844Z INFO bento_cli: image_id: a0dfc25e54ebde808e4fd8c34b6549bbb91b4928edeea90ceb7d1d8e7e9096c7 | input_id: eccc8f06-488a-426c-ae3d-e5acada9ae22
2024-10-23T14:37:37.368613Z INFO bento_cli: STARK job_id: 0d89e2ca-a1e3-478f-b89d-8ab23b89f51e
2024-10-23T14:37:37.369346Z INFO bento_cli: STARK Job running....
2024-10-23T14:37:39.371331Z INFO bento_cli: STARK Job running....
2024-10-23T14:37:41.373508Z INFO bento_cli: STARK Job running....
2024-10-23T14:37:43.375780Z INFO bento_cli: Job done!
```
### Benchmarking Bento
Start a bento cluster:
```bash Terminal theme={null}
just bento
```
Set the `RPC_URL` environment variable to the network the order is on:
```bash Terminal theme={null}
export RPC_URL=
```
Then, run the benchmark:
```bash Terminal theme={null}
boundless prover benchmark --request-ids
```
where IDS is a comma-separated list of request IDs from the network or order stream configured.
It is recommended to pick a few requests of varying sizes and programs, biased towards larger proofs for a more representative benchmark.
To run programs manually, and for performance optimizations, see [performance optimizations](/provers/performance-optimization).
### Running the Broker service with bento
Running a broker with `just` will also start the Bento cluster through docker compose.
`just` installation instructions can be found [here](https://github.com/casey/just#installation).
```bash Terminal theme={null}
just broker
```
### Make sure Bento is running
A Broker needs a [Bento](/provers/proving-stack#what-is-bento) instance to operate. Please follow the [quick start](/provers/quick-start) guide to get Bento up and running.
To check Bento is running correctly, you can send a sample proof workload:
> Before running this, [install Bento CLI](/provers/quick-start#running-a-test-proof)
```bash Terminal theme={null}
# In the bento directory
RUST_LOG=info bento_cli -c 32
```
### Running a standalone broker
To run broker with an already initialized Bento cluster or with a different prover, you can build and run a broker directly with the following:
```bash Terminal theme={null}
cargo build --bin broker --release
# Run with flags or environment variables based on network/configuration
./target/release/broker
```
### Stopping The Broker Service
```bash Terminal theme={null}
just broker down
```
If running the broker on a network, there may be locked proofs that have not been fulfilled yet. Follow the [Safe Upgrade Steps](#safe-upgrade-steps) to ensure shutdown and/or restart without loss of stake.
## Safe Upgrade Steps
There can be subtle breaking changes between releases that may affect your broker's state. Following these upgrade steps helps minimize issues from state breaking changes.
When upgrading your Boundless broker to a new version, follow these steps to ensure a safe migration:
```bash Terminal theme={null}
just broker clean
# Or stop the broker without clearing volumes
just broker down
```
This will wait for any committed orders to finalize before shutting down. Avoid sending kill signals to the broker process and ensure either through the broker logs or through indexer that your broker does not have any incomplete locked orders before proceeding.
While it is generally not necessary to clear volumes unless specifically noted in release, it is recommended to avoid any potential state breaking changes.
See [releases](https://github.com/boundless-xyz/boundless/releases) for latest tag to use.
```bash Terminal theme={null}
git checkout
# Example: git checkout v0.9.0
```
```bash Terminal theme={null}
just broker
```
## Running Multiple Brokers
You can run multiple broker instances simultaneously to serve different networks at the same time while sharing the same Bento cluster. The Docker compose setup supports this through the `broker2` service example.
### Multi-Broker Configuration
Each broker instance requires:
1. **Separate configuration file**: Create different `broker.toml` files (e.g., `broker.toml`, `broker2.toml`, etc.)
2. **Different RPC URL**: Use different chain endpoints via setting respective `RPC_URL` environment variables, or modifying the compose file manually (`prover-compose.yml` by default, `compose.yml` for the legacy stack).
3. **Optional separate private key**: Use different `PRIVATE_KEY` variables if desired for different accounts on different networks.
### Environment Variables for Multi-Broker Setup
If using the default compose file (`prover-compose.yml`, or `compose.yml` if `PROVER_STACK=legacy` is set) and uncommenting the second broker config:
```bash [.env] theme={null}
# Export environment variables for the first broker
export RPC_URL=
export PRIVATE_KEY=0x...
# Export environment variables for the second broker
export RPC_URL_2=
```
Then, create the new broker config file that the second broker will use:
```bash Terminal theme={null}
# Copy from an existing broker config file
cp broker.toml broker2.toml
# Or creating one from a fresh template
cp broker-template.toml broker2.toml
```
Then, modify configuration values for each network, keeping the following in mind:
* The `peak_prove_khz` setting is shared across all brokers
* For example, if you have [benchmarked](#benchmarking-bento) your broker to be able to prove at 500kHz, the values in each config should not sum up to be more than 500kHz.
* `max_concurrent_preflights` defaults to `8` and should be set to a value that the bento cluster can keep up with
* It is recommended that the max concurrent preflights across all networks is less than the number of exec agents you have specified in your compose file (`prover-compose.yml` by default, `compose.yml` for the legacy stack).
* `max_concurrent_proofs` is a per-broker configuration, and is not shared across brokers
Then, just start the cluster as you normally would with:
```bash Terminal theme={null}
just broker
```
## Broker Optimization
### Increasing Lock-in Rate
Once your broker is running, there are a few methods to optimize the lock-in rate. These methods are aimed at making your broker service more competitive in the market through different means:
1. Decreasing the `min_mcycle_price` would tune your Broker to bid at lower prices for proofs. This value can be specified in ETH (e.g., `"0.00001 ETH"`) or USD (e.g., `"0.02 USD"`).
2. Increasing `lockin_priority_gas` expedites your market operations by consuming more gas which could help outrun other bidders.
### Tuning Service Settings
The `[prover]` settings in `broker.toml` are used to configure the prover service and significantly impact the operation of the service. The most important configuration variable to monitor and iteratively tune is `txn_timeout`. This is the number of seconds to wait for a transaction to be confirmed before timing out. Therefore, if you see timeouts in your logs, `txn_timeout` can be increased to wait longer for transaction confirmations onchain.