Broker Configuration & Operation
Overview
The Broker is a service that runs within the Bento proving stack. It is responsible for market interactions including bidding on jobs, locking them, issuing job requests to the Bento proving cluster, and submitting proof fulfillments onchain.
Broker Configuration
Broker configuration is primarily managed through the broker.toml file in the Boundless directory. This file is mounted into the Broker container and it is used to configure the Broker daemon.
Deposit / Balance
The Boundless market requires funds (ZKC) deposited as collateral before a prover can bid on requests.
Brokers must first deposit some ZKC into the market contract to fund their account.
These funds cover collateral during lock-in.
It is recommend that a broker keep a balance on the market >= max_collateral (configured via broker.toml).
Deposit Collateral to the Market
export RPC_URL=<TARGET_CHAIN_RPC_URL>
export PRIVATE_KEY=<BROKER_PRIVATE_KEY>
# Example: 'account deposit-collateral 100'
boundless account deposit-collateral <ZKC_TO_DEPOSIT>Check Current Collateral Balance
export RPC_URL=<TARGET_CHAIN_RPC_URL>
export PRIVATE_KEY=<BROKER_PRIVATE_KEY>
boundless account collateral-balance [wallet_address]You can omit the PRIVATE_KEY environment variable here and specify your wallet_address as a optional parameter to the balance command, i.e., account balance 0x000....
Settings in Broker.toml
Below are all broker.toml settings organized by section:
[market] Settings
| Setting | Description |
|---|---|
| min_mcycle_price | The minimum price per mega-cycle (i.e. million RISC-V cycles) for the broker to attempt to lock an order, denominated in the native token (e.g. ETH). Orders are preflighted to determine the number of cycles required to prove the order. Based on this cycle count, and the current price of the order, the broker computes the current price per mega-cycle being offered for the order. This value is compared to the min_mcycle_price to determine if the order should be locked. |
| min_mcycle_price_collateral_token | The minimum price per mega-cycle for the broker to attempt to fulfill an order in the case that the order was locked by another prover but not fulfilled within the lock timeout. If an order is locked by a prover, but not fulfilled within the lock timeout, that prover is slashed by the amount of collateral specified in the order. A portion of this collateral is then used to incentivize other provers to fulfill the order. This determines the minimum price per mega-cycle in collateral token (ZKC) that must be offered for the broker to attempt to fulfill these orders. Note: attempting to fulfill these orders is a 'proof race' where the first prover to fulfill the order is rewarded with the collateral. |
| priority_requestor_addresses | Optional priority requestor addresses that can bypass the mcycle limit and max input size limit. If enabled, the order will be preflighted without constraints. |
| priority_requestor_lists | URLs to fetch requestor priority lists from. Requestor priority lists specify requestors that the broker should prioritize for proving. Requestors on these lists are considered more likely to request useful work with profitable pricing. Priority requestors are prioritized when there is a surplus of demand: their requests will be preflighted first, locked first (in situations where multiple orders exceed minimum lock pricing), and configuration for max_mcycle_limit and max_file_size will be skipped. These lists will be periodically refreshed and merged with priority_requestor_addresses. |
| peak_prove_khz | Estimated peak performance of the proving cluster, in kHz. Used to estimate proving capacity and accept only as much work as your prover cluster can handle. Estimates can be derived from benchmarking using the Boundless CLI. See Benchmarking Bento. |
| max_mcycle_limit | Optional max cycles (in mcycles). Orders over this max_cycles will be skipped after preflight. |
| max_journal_bytes | Max journal size in bytes. Orders that produce a journal larger than this size in preflight will be skipped. Since journals must be posted onchain to complete an order, an excessively large journal may prevent completion of a request. |
| min_deadline | Min seconds left before the deadline to consider bidding on a request. If there is not enough time left before the deadline, the prover may not be able to complete proving of the request and finalize the batch for publishing before expiration. |
| lookback_blocks | On startup, the number of blocks to look back for possible open orders. |
| max_collateral | Maximum collateral amount, denominated in the Boundless collateral token, that the broker will use to lock orders. Requests that require a higher collateral amount than this will be skipped. |
| max_file_size | Max input / image file size allowed for downloading from request URLs. |
| max_fetch_retries | Max retries for fetching input / image contents from URLs. |
| assessor_default_image_url | Default URL for assessor image. This URL will be tried first before falling back to the image URL from the boundless market contract. |
| set_builder_default_image_url | Default URL for set builder image. This URL will be tried first before falling back to the image URL from the set verifier contract. |
| max_concurrent_proofs | Maximum number of concurrent proofs that can be processed at once. Used to limit proof tasks spawned to prevent overwhelming the system. |
| max_concurrent_preflights | Maximum number of orders to concurrently preflight. Used to limit preflight tasks spawned to prevent overwhelming the system. |
| order_pricing_priority | Determines how orders are prioritized for pricing. Options: "random" (process orders in random order), "observation_time" (FIFO), "shortest_expiry" (earliest deadline), "price" (highest ETH payment), "cycle_price" (highest ETH price per cycle, default). |
| order_commitment_priority | Determines how orders are prioritized when committing to prove them. Options: "random" (process orders in random order, default), "shortest_expiry" (process orders by shortest expiry first). |
| max_critical_task_retries | Max critical task retries on recoverable failures. The broker service has a number of subtasks. Some are considered critical. If a task fails, it will be retried, but after this number of retries, the process will exit. |
| allow_client_addresses | Optional allow list for customer address. If enabled, all requests from clients not in the allow list are skipped. |
| deny_requestor_addresses | Optional deny list for requestor address. If enabled, all requests from clients in the deny list are skipped. |
| lockin_priority_gas | Optional additional gas to add to the transaction for lockinRequest, good for increasing the priority if competing with multiple provers during the same block. |
| balance_warn_threshold | Optional balance warning threshold (in native token). If the submitter balance drops below this the broker will issue warning logs. |
| balance_error_threshold | Optional balance error threshold (in native token). If the submitter balance drops below this the broker will issue error logs. |
| collateral_balance_warn_threshold | Optional collateral balance warning threshold (in collateral tokens). If the collateral balance drops below this the broker will issue warning logs. |
| collateral_balance_error_threshold | Optional collateral balance error threshold (in collateral tokens). If the collateral balance drops below this the broker will issue error logs. |
| cache_dir | Optional cache directory for storing downloaded images and inputs. If not set, files will be re-downloaded every time. |
| lockin_gas_estimate | Gas estimate for lockin call. Used for estimating the gas costs associated with an order during pricing. If not set a conservative default will be used. |
| fulfill_gas_estimate | Gas estimate for fulfill call. Used for estimating the gas costs associated with an order during pricing. If not set a conservative default will be used. |
| groth16_verify_gas_estimate | Gas estimate for proof verification using the RiscZeroGroth16Verifier. Used for estimating the gas costs associated with an order during pricing. If not set a conservative default will be used. |
[prover] Settings
| Setting | Description |
|---|---|
| status_poll_retry_count | Number of retries to poll for proving status. Provides a little durability for transient failures. |
| status_poll_ms | Polling interval to monitor proving status (in millisecs). |
| req_retry_count | Number of retries to query a prover backend for on failures. Used for API requests to a prover backend, creating sessions, preflighting, uploading images, etc. Provides a little durability for transient failures. |
| req_retry_sleep_ms | Number of milliseconds to sleep between retries. |
| proof_retry_count | Number of retries for running the entire proof generation process. This is separate from the request retry count, as the proving process is a multi-step process involving multiple API calls to create a proof job and then polling for the proof job to complete. |
| proof_retry_sleep_ms | Number of milliseconds to sleep between proof retries. |
| set_builder_guest_path | Set builder guest program (ELF) path. When using a durable deploy, set this to the published current SOT guest program path on the system. |
| assessor_set_guest_path | Assessor ELF path. |
| reaper_interval_secs | Interval for checking expired committed orders (in seconds). This is the interval at which the ReaperTask will check for expired orders and mark them as failed. If not set, it defaults to 60 seconds. |
| reaper_grace_period_secs | Grace period before marking expired orders as failed (in seconds). This provides a buffer time after an order expires before the reaper marks it as failed. This helps prevent race conditions with the aggregator that might be processing the order. If not set, it defaults to 10800 seconds (3 hours). |
[batcher] Settings
| Setting | Description |
|---|---|
| batch_max_time | Max batch duration before publishing (in seconds). |
| min_batch_size | Batch size (in proofs) before publishing. |
| block_deadline_buffer_secs | Batch blocktime buffer. Number of seconds before the lowest block deadline in the order batch to flush the batch. This should be approximately snark_proving_time * 2. |
| txn_timeout | Timeout, in seconds for transaction confirmations. |
| single_txn_fulfill | Use the single TXN submission that batches submit_merkle / fulfill_batch into a single transaction. Requires the submitRootAndFulfill method be present on the deployed contract. |
| withdraw | Whether to withdraw from the prover balance when fulfilling. |
| batch_poll_time_ms | Polling time, in milliseconds. The time between polls for new orders to aggregate and how often to check for batch finalize conditions. |
| batch_max_journal_bytes | Max combined journal size (in bytes) that once exceeded will trigger a publish. |
| batch_max_fees | Max batch fees (in ETH) before publishing. |
| max_submission_attempts | Number of attempts to make to submit a batch before abandoning. |
Broker Operation
2024-10-23T14:37:37.364844Z INFO bento_cli: image_id: a0dfc25e54ebde808e4fd8c34b6549bbb91b4928edeea90ceb7d1d8e7e9096c7 | input_id: eccc8f06-488a-426c-ae3d-e5acada9ae22
2024-10-23T14:37:37.368613Z INFO bento_cli: STARK job_id: 0d89e2ca-a1e3-478f-b89d-8ab23b89f51e
2024-10-23T14:37:37.369346Z INFO bento_cli: STARK Job running....
2024-10-23T14:37:39.371331Z INFO bento_cli: STARK Job running....
2024-10-23T14:37:41.373508Z INFO bento_cli: STARK Job running....
2024-10-23T14:37:43.375780Z INFO bento_cli: Job done!Benchmarking Bento
Start a bento cluster:
just bentoSet the RPC_URL environment variable to the network the order is on:
export RPC_URL=<TARGET_CHAIN_RPC_URL>Then, run the benchmark:
boundless proving benchmark --request-ids <IDS>where IDS is a comma-separated list of request IDs from the network or order stream configured.
It is recommended to pick a few requests of varying sizes and programs, biased towards larger proofs for a more representative benchmark.
To run programs manually, and for performance optimizations, see performance optimizations.
Running the Broker service with bento
Running a broker with just will also start the Bento cluster through docker compose.
just brokerMake sure Bento is running
To check Bento is running correctly, you can send a sample proof workload:
Before running this, install Bento CLI
# In the bento directory
RUST_LOG=info bento_cli -c 32Running a standalone broker
To run broker with an already initialized Bento cluster or with a different prover, you can build and run a broker directly with the following:
cargo build --bin broker --release
# Run with flags or environment variables based on network/configuration
./target/release/brokerStopping The Broker Service
just broker downSafe Upgrade Steps
When upgrading your Boundless broker to a new version, follow these steps to ensure a safe migration:
Stop the broker and optionally clean the database
just broker clean
# Or stop the broker without clearing volumes
just broker downThis will wait for any committed orders to finalize before shutting down. Avoid sending kill signals to the broker process and ensure either through the broker logs or through indexer that your broker does not have any incomplete locked orders before proceeding.
Update to the new version
See releases for latest tag to use.
git checkout <new_version_tag>
# Example: git checkout v0.9.0Start the broker with the new version
just brokerRunning Multiple Brokers
You can run multiple broker instances simultaneously to serve different networks at the same time while sharing the same Bento cluster. The Docker compose setup supports this through the broker2 service example.
Multi-Broker Configuration
Each broker instance requires:
- Separate configuration file: Create different
broker.tomlfiles (e.g.,broker.toml,broker2.toml, etc.) - Different RPC URL: Use different chain endpoints via setting respective
RPC_URLenvironment variables, or modifying thecompose.ymlmanually. - Optional separate private key: Use different
PRIVATE_KEYvariables if desired for different accounts on different networks.
Environment Variables for Multi-Broker Setup
If using the default compose.yml file and uncommenting the second broker config:
# Export environment variables for the first broker
export RPC_URL=<URL for network 1>
export PRIVATE_KEY=0x...
# Export environment variables for the second broker
export RPC_URL_2=<URL for network 2>Then, create the new broker config file that the second broker will use:
# Copy from an existing broker config file
cp broker.toml broker2.toml
# Or creating one from a fresh template
cp broker-template.toml broker2.tomlThen, modify configuration values for each network, keeping the following in mind:
- The
peak_prove_khzsetting is shared across all brokers- For example, if you have benchmarked your broker to be able to prove at 500kHz, the values in each config should not sum up to be more than 500kHz.
max_concurrent_preflightsis set to a value that the bento cluster can keep up with- It is recommended that the max concurrent preflights across all networks is less than the number of exec agents you have specified in your
compose.yml.
- It is recommended that the max concurrent preflights across all networks is less than the number of exec agents you have specified in your
max_concurrent_proofsis a per-broker configuration, and is not shared across brokers
Then, just start the cluster as you normally would with:
just brokerBroker Optimization
Increasing Lock-in Rate
Once your broker is running, there are a few methods to optimize the lock-in rate. These methods are aimed at making your broker service more competitive in the market through different means:
- Decreasing the
min_mcycle_pricewould tune your Broker to bid at lower prices for proofs. - Increasing
lockin_priority_gasexpedites your market operations by consuming more gas which could help outrun other bidders.
Tuning Service Settings
The [prover] settings in broker.toml are used to configure the prover service and significantly impact the operation of the service. The most important configuration variable to monitor and iteratively tune is txn_timeout. This is the number of seconds to wait for a transaction to be confirmed before timing out. Therefore, if you see timeouts in your logs, txn_timeout can be increased to wait longer for transaction confirmations onchain.