Broker Configuration & Operation

Overview

The Broker is a service that runs within the Bento proving stack. It is responsible for market interactions including bidding on jobs, locking them, issuing job requests to the Bento proving cluster, and submitting proof fulfillments onchain.

Broker Configuration

Broker will live-reload the broker.toml when it changes. In most cases, you will not need to restart the Broker for the configuration to take effect.

Broker configuration is primarily managed through the broker.toml file in the Boundless directory. This file is mounted into the Broker container and it is used to configure the Broker daemon.

Deposit / Balance

The Boundless market requires funds (ZKC) deposited as collateral before a prover can bid on requests. Brokers must first deposit some ZKC into the market contract to fund their account. These funds cover collateral during lock-in. It is recommend that a broker keep a balance on the market >= max_collateral (configured via broker.toml).

Deposit Collateral to the Market

You will need the Boundless CLI installed to deposit/check your balance. Please see Installing the Boundless CLI for instructions.

Terminal

export RPC_URL=<TARGET_CHAIN_RPC_URL>
export PRIVATE_KEY=<BROKER_PRIVATE_KEY>

# Example: 'account deposit-collateral 100'
boundless account deposit-collateral <ZKC_TO_DEPOSIT>

Check Current Collateral Balance

Terminal

export RPC_URL=<TARGET_CHAIN_RPC_URL>
export PRIVATE_KEY=<BROKER_PRIVATE_KEY>

boundless account collateral-balance [wallet_address]

You can omit the PRIVATE_KEY environment variable here and specify your wallet_address as a optional parameter to the balance command, i.e., account balance 0x000....

Settings in Broker.toml

Quotation marks matter in TOML so please pay particular attention to the quotation marks for config values.

Below are all broker.toml settings organized by section:

[market] Settings

Setting	Description
min_mcycle_price	The minimum price per mega-cycle (i.e. million RISC-V cycles) for the broker to attempt to lock an order, denominated in the native token (e.g. ETH). Orders are preflighted to determine the number of cycles required to prove the order. Based on this cycle count, and the current price of the order, the broker computes the current price per mega-cycle being offered for the order. This value is compared to the min_mcycle_price to determine if the order should be locked.
min_mcycle_price_collateral_token	The minimum price per mega-cycle for the broker to attempt to fulfill an order in the case that the order was locked by another prover but not fulfilled within the lock timeout. If an order is locked by a prover, but not fulfilled within the lock timeout, that prover is slashed by the amount of collateral specified in the order. A portion of this collateral is then used to incentivize other provers to fulfill the order. This determines the minimum price per mega-cycle in collateral token (ZKC) that must be offered for the broker to attempt to fulfill these orders. Note: attempting to fulfill these orders is a ‘proof race’ where the first prover to fulfill the order is rewarded with the collateral.
priority_requestor_addresses	Optional priority requestor addresses that can bypass the mcycle limit and max input size limit. If enabled, the order will be preflighted without constraints.
priority_requestor_lists	URLs to fetch requestor priority lists from. Requestor priority lists specify requestors that the broker should prioritize for proving. Requestors on these lists are considered more likely to request useful work with profitable pricing. Priority requestors are prioritized when there is a surplus of demand: their requests will be preflighted first, locked first (in situations where multiple orders exceed minimum lock pricing), and configuration for max_mcycle_limit and max_file_size will be skipped. These lists will be periodically refreshed and merged with priority_requestor_addresses.
peak_prove_khz	Estimated peak performance of the proving cluster, in kHz. Used to estimate proving capacity and accept only as much work as your prover cluster can handle. Estimates can be derived from benchmarking using the Boundless CLI. See Benchmarking Bento.
max_mcycle_limit	Optional max cycles (in mcycles). Orders over this max_cycles will be skipped after preflight.
max_journal_bytes	Max journal size in bytes. Orders that produce a journal larger than this size in preflight will be skipped. Since journals must be posted onchain to complete an order, an excessively large journal may prevent completion of a request.
min_deadline	Min seconds left before the deadline to consider bidding on a request. If there is not enough time left before the deadline, the prover may not be able to complete proving of the request and finalize the batch for publishing before expiration.
lookback_blocks	On startup, the number of blocks to look back for possible open orders.
max_collateral	Maximum collateral amount, denominated in the Boundless collateral token, that the broker will use to lock orders. Requests that require a higher collateral amount than this will be skipped.
max_file_size	Max input / image file size allowed for downloading from request URLs.
max_fetch_retries	Max retries for fetching input / image contents from URLs.
assessor_default_image_url	Default URL for assessor image. This URL will be tried first before falling back to the image URL from the boundless market contract.
set_builder_default_image_url	Default URL for set builder image. This URL will be tried first before falling back to the image URL from the set verifier contract.
max_concurrent_proofs	Maximum number of concurrent proofs that can be processed at once. Used to limit proof tasks spawned to prevent overwhelming the system.
max_concurrent_preflights	Maximum number of orders to concurrently preflight. Used to limit preflight tasks spawned to prevent overwhelming the system.
order_pricing_priority	Determines how orders are prioritized for pricing. Options: “random” (process orders in random order), “observation_time” (FIFO), “shortest_expiry” (earliest deadline), “price” (highest ETH payment), “cycle_price” (highest ETH price per cycle, default).
order_commitment_priority	Determines how orders are prioritized when committing to prove them. Options: “random” (process orders in random order, default), “shortest_expiry” (process orders by shortest expiry first).
max_critical_task_retries	Max critical task retries on recoverable failures. The broker service has a number of subtasks. Some are considered critical. If a task fails, it will be retried, but after this number of retries, the process will exit.
allow_client_addresses	Optional allow list for customer address. If enabled, all requests from clients not in the allow list are skipped.
deny_requestor_addresses	Optional deny list for requestor address. If enabled, all requests from clients in the deny list are skipped.
lockin_priority_gas	Optional additional gas to add to the transaction for lockinRequest, good for increasing the priority if competing with multiple provers during the same block.
balance_warn_threshold	Optional balance warning threshold (in native token). If the submitter balance drops below this the broker will issue warning logs.
balance_error_threshold	Optional balance error threshold (in native token). If the submitter balance drops below this the broker will issue error logs.
collateral_balance_warn_threshold	Optional collateral balance warning threshold (in collateral tokens). If the collateral balance drops below this the broker will issue warning logs.
collateral_balance_error_threshold	Optional collateral balance error threshold (in collateral tokens). If the collateral balance drops below this the broker will issue error logs.
cache_dir	Optional cache directory for storing downloaded images and inputs. If not set, files will be re-downloaded every time.
lockin_gas_estimate	Gas estimate for lockin call. Used for estimating the gas costs associated with an order during pricing. If not set a conservative default will be used.
fulfill_gas_estimate	Gas estimate for fulfill call. Used for estimating the gas costs associated with an order during pricing. If not set a conservative default will be used.
groth16_verify_gas_estimate	Gas estimate for proof verification using the RiscZeroGroth16Verifier. Used for estimating the gas costs associated with an order during pricing. If not set a conservative default will be used.

[prover] Settings

Setting	Description
status_poll_retry_count	Number of retries to poll for proving status. Provides a little durability for transient failures.
status_poll_ms	Polling interval to monitor proving status (in millisecs).
req_retry_count	Number of retries to query a prover backend for on failures. Used for API requests to a prover backend, creating sessions, preflighting, uploading images, etc. Provides a little durability for transient failures.
req_retry_sleep_ms	Number of milliseconds to sleep between retries.
proof_retry_count	Number of retries for running the entire proof generation process. This is separate from the request retry count, as the proving process is a multi-step process involving multiple API calls to create a proof job and then polling for the proof job to complete.
proof_retry_sleep_ms	Number of milliseconds to sleep between proof retries.
set_builder_guest_path	Set builder guest program (ELF) path. When using a durable deploy, set this to the published current SOT guest program path on the system.
assessor_set_guest_path	Assessor ELF path.
reaper_interval_secs	Interval for checking expired committed orders (in seconds). This is the interval at which the ReaperTask will check for expired orders and mark them as failed. If not set, it defaults to 60 seconds.
reaper_grace_period_secs	Grace period before marking expired orders as failed (in seconds). This provides a buffer time after an order expires before the reaper marks it as failed. This helps prevent race conditions with the aggregator that might be processing the order. If not set, it defaults to 10800 seconds (3 hours).

[batcher] Settings

Setting	Description
batch_max_time	Max batch duration before publishing (in seconds).
min_batch_size	Batch size (in proofs) before publishing.
block_deadline_buffer_secs	Batch blocktime buffer. Number of seconds before the lowest block deadline in the order batch to flush the batch. This should be approximately snark_proving_time * 2.
txn_timeout	Timeout, in seconds for transaction confirmations.
single_txn_fulfill	Use the single TXN submission that batches submit_merkle / fulfill_batch into a single transaction. Requires the `submitRootAndFulfill` method be present on the deployed contract.
withdraw	Whether to withdraw from the prover balance when fulfilling.
batch_poll_time_ms	Polling time, in milliseconds. The time between polls for new orders to aggregate and how often to check for batch finalize conditions.
batch_max_journal_bytes	Max combined journal size (in bytes) that once exceeded will trigger a publish.
batch_max_fees	Max batch fees (in ETH) before publishing.
max_submission_attempts	Number of attempts to make to submit a batch before abandoning.

Broker Operation

Terminal

2024-10-23T14:37:37.364844Z  INFO bento_cli: image_id: a0dfc25e54ebde808e4fd8c34b6549bbb91b4928edeea90ceb7d1d8e7e9096c7 | input_id: eccc8f06-488a-426c-ae3d-e5acada9ae22
2024-10-23T14:37:37.368613Z  INFO bento_cli: STARK job_id: 0d89e2ca-a1e3-478f-b89d-8ab23b89f51e
2024-10-23T14:37:37.369346Z  INFO bento_cli: STARK Job running....
2024-10-23T14:37:39.371331Z  INFO bento_cli: STARK Job running....
2024-10-23T14:37:41.373508Z  INFO bento_cli: STARK Job running....
2024-10-23T14:37:43.375780Z  INFO bento_cli: Job done!

Benchmarking Bento

Start a bento cluster:

Terminal

just bento

Set the RPC_URL environment variable to the network the order is on:

Terminal

export RPC_URL=<TARGET_CHAIN_RPC_URL>

Then, run the benchmark:

Terminal

boundless proving benchmark --request-ids <IDS>

where IDS is a comma-separated list of request IDs from the network or order stream configured. It is recommended to pick a few requests of varying sizes and programs, biased towards larger proofs for a more representative benchmark. To run programs manually, and for performance optimizations, see performance optimizations.

Running the Broker service with bento

Running a broker with just will also start the Bento cluster through docker compose.

just installation instructions can be found here.

Terminal

just broker

Make sure Bento is running

A Broker needs a Bento instance to operate. Please follow the quick start guide to get Bento up and running.

To check Bento is running correctly, you can send a sample proof workload:

Before running this, install Bento CLI

Terminal

# In the bento directory
RUST_LOG=info bento_cli -c 32

Running a standalone broker

To run broker with an already initialized Bento cluster or with a different prover, you can build and run a broker directly with the following:

Terminal

cargo build --bin broker --release
# Run with flags or environment variables based on network/configuration
./target/release/broker

Stopping The Broker Service

Terminal

just broker down

If running the broker on a network, there may be locked proofs that have not been fulfilled yet. Follow the Safe Upgrade Steps to ensure shutdown and/or restart without loss of stake.

Safe Upgrade Steps

There can be subtle breaking changes between releases that may affect your broker’s state. Following these upgrade steps helps minimize issues from state breaking changes.

When upgrading your Boundless broker to a new version, follow these steps to ensure a safe migration:

Stop the broker and optionally clean the database

Terminal

just broker clean

# Or stop the broker without clearing volumes
just broker down

This will wait for any committed orders to finalize before shutting down. Avoid sending kill signals to the broker process and ensure either through the broker logs or through indexer that your broker does not have any incomplete locked orders before proceeding.

While it is generally not necessary to clear volumes unless specifically noted in release, it is recommended to avoid any potential state breaking changes.

Update to the new version

See releases for latest tag to use.

Terminal

git checkout <new_version_tag>
# Example: git checkout v0.9.0

Start the broker with the new version

Terminal

just broker

Running Multiple Brokers

You can run multiple broker instances simultaneously to serve different networks at the same time while sharing the same Bento cluster. The Docker compose setup supports this through the broker2 service example.

Multi-Broker Configuration

Each broker instance requires:

Separate configuration file: Create different broker.toml files (e.g., broker.toml, broker2.toml, etc.)
Different RPC URL: Use different chain endpoints via setting respective RPC_URL environment variables, or modifying the compose.yml manually.
Optional separate private key: Use different PRIVATE_KEY variables if desired for different accounts on different networks.

Environment Variables for Multi-Broker Setup

If using the default compose.yml file and uncommenting the second broker config:

[.env]

# Export environment variables for the first broker
export RPC_URL=<URL for network 1>
export PRIVATE_KEY=0x...

# Export environment variables for the second broker
export RPC_URL_2=<URL for network 2>

Then, create the new broker config file that the second broker will use:

Terminal

# Copy from an existing broker config file
cp broker.toml broker2.toml

# Or creating one from a fresh template
cp broker-template.toml broker2.toml

Then, modify configuration values for each network, keeping the following in mind:

The peak_prove_khz setting is shared across all brokers
- For example, if you have benchmarked your broker to be able to prove at 500kHz, the values in each config should not sum up to be more than 500kHz.
max_concurrent_preflights is set to a value that the bento cluster can keep up with
- It is recommended that the max concurrent preflights across all networks is less than the number of exec agents you have specified in your compose.yml.
max_concurrent_proofs is a per-broker configuration, and is not shared across brokers

Then, just start the cluster as you normally would with:

Terminal

just broker

Broker Optimization

Increasing Lock-in Rate

Once your broker is running, there are a few methods to optimize the lock-in rate. These methods are aimed at making your broker service more competitive in the market through different means:

Decreasing the min_mcycle_price would tune your Broker to bid at lower prices for proofs.
Increasing lockin_priority_gas expedites your market operations by consuming more gas which could help outrun other bidders.

Tuning Service Settings

The [prover] settings in broker.toml are used to configure the prover service and significantly impact the operation of the service. The most important configuration variable to monitor and iteratively tune is txn_timeout. This is the number of seconds to wait for a transaction to be confirmed before timing out. Therefore, if you see timeouts in your logs, txn_timeout can be increased to wait longer for transaction confirmations onchain.

Core Concepts

$ZKC Overview

ZK Mining

Broker Configuration & Operation

Overview