Enhance Loadtest Observability.

core/tests/loadnext/README.md

@@ -1,125 +1,112 @@
-# Loadnext: the next generation loadtest for zkSync
+# Loadnext: loadtest for zkSync
 
 Loadnext is an utility for random stress-testing the zkSync server. It is capable of simulating the behavior of many
 independent users of zkSync network, who are sending quasi-random requests to the server.
 
+The general flow is as follows:
+
+- The master account performs an initial deposit to L2
+- Paymaster on L2 is funded if necessary
+- The L2 master account distributes funds to the participating accounts (`accounts_amount` configuration option)
+- Each account continiously sends L2 transactions as configured in `contract_execution_params` configuration option. At
+  any given time there are no more than `max_inflight_txs` transactions in flight for each account.
+- Once each account is done with the initial deposit, the test is run for `duration_sec` seconds.
+- After the test is finished, the master account withdraws all the remaining funds from L2.
+- The average TPS is reported.
+
+## Features
+
 It:
 
-- doesn't care whether the server is alive or not. At worst, it will just consider the test failed. No panics, no
-  mindless unwraps, yay.
+- doesn't care whether the server is alive or not. At worst, it will just consider the test failed.
 - does a unique set of operations for each participating account.
 - sends transactions and priority operations.
 - sends incorrect transactions as well as correct ones and compares the outcome to the expected one.
 - has an easy-to-extend command system that allows adding new types of actions to the flow.
 - has an easy-to-extend report analysis system.
 
-Flaws:
-
-- It does not send API requests other than required to execute transactions.
-- So far it has pretty primitive report system.
+## Transactions Parameters
 
-## Launch
+The smart contract that is used for every l2 transaction can be found here:
+`etc/contracts-test-data/contracts/loadnext/loadnext_contract.sol`.
 
-In order to launch the test in the development scenario, you must first run server and prover (it is recommended to use
-dummy prover), and then launch the test itself.
+The `execute` function of the contract has the following parameters:
 
-```sh
-# First terminal
-zk server
-# Second terminal
-RUST_BACKTRACE=1 RUST_LOG=info,jsonrpsee_ws_client=error cargo run --bin loadnext
+```
+    function execute(uint reads, uint writes, uint hashes, uint events, uint max_recursion, uint deploys) external returns(uint) {
 ```
 
-Without any configuration supplied, the test will fallback to the dev defaults:
-
-- Use one of the "rich" accounts in the private local Ethereum chain.
-- Use a random ERC-20 token from `etc/tokens/localhost.json`.
-- Connect to the localhost zkSync node and use localhost web3 API.
-
-**Note:** when running the loadtest in the localhost scenario, you **must** adjust the supported block chunks sizes.
-Edit the `etc/env/dev/chain.toml` and set `block_chunk_sizes` to `[10,32,72,156,322,654]` and `aggregated_proof_sizes`
-to `[1,4,8,18]`. Do not forget to re-compile configs after that.
+which correspond to the following configuration options:
 
-This is required because the loadtest relies on batches, which will not fit into smaller block sizes.
+```
+pub struct LoadnextContractExecutionParams {
+    pub reads: usize,
+    pub writes: usize,
+    pub events: usize,
+    pub hashes: usize,
+    pub recursive_calls: usize,
+    pub deploys: usize,
+}
+```
 
-## Configuration
+For example, to simulate an average transaction on mainnet, one could do:
 
-For cases when loadtest is launched outside of the localhost environment, configuration is provided via environment
-variables.
-
-The following variables are required:
-
-```sh
-# Address of the Ethereum web3 API.
-L1_RPC_ADDRESS
-# Ethereum private key of the wallet that has funds to perform a test (without `0x` prefix).
-MASTER_WALLET_PK
-# Amount of accounts to be used in test.
-# This option configures the "width" of the test:
-# how many concurrent operation flows will be executed.
-ACCOUNTS_AMOUNT
-# All of test accounts get split into groups that share the
-# deployed contract address. This helps to emulate the behavior of
-# sending `Execute` to the same contract and reading its events by
-# single a group. This value should be less than or equal to `ACCOUNTS_AMOUNT`.
-ACCOUNTS_GROUP_SIZE
-# Amount of operations per account.
-# This option configures the "length" of the test:
-# how many individual operations each account of the test will execute.
-OPERATIONS_PER_ACCOUNT
-# Address of the ERC-20 token to be used in test.
-#
-# Token must satisfy two criteria:
-# - Be supported by zkSync.
-# - Have `mint` operation.
-#
-# Note that we use ERC-20 token since we can't easily mint a lot of ETH on
-# Rinkeby or Ropsten without caring about collecting it back.
-MAIN_TOKEN
-# Path to test contracts bytecode and ABI required for sending
-# deploy and execute L2 transactions. Each folder in the path is expected
-# to have the following structure:
-# .
-# ├── bytecode
-# └── abi.json
-# Contract folder names names are not restricted.
-# An example:
-# .
-# ├── erc-20
-# │   ├── bytecode
-# │   └── abi.json
-# └── simple-contract
-#     ├── bytecode
-#     └── abi.json
-TEST_CONTRACTS_PATH
-# Limits the number of simultaneous API requests being performed at any moment of time.
-#
-# Setting it to:
-# - 0 turns off API requests.
-# - `ACCOUNTS_AMOUNT` relieves the limit.
-SYNC_API_REQUESTS_LIMIT
-# zkSync Chain ID.
-L2_CHAIN_ID
-# Address of the zkSync web3 API.
-L2_RPC_ADDRESS
+```
+CONTRACT_EXECUTION_PARAMS_WRITES=2
+CONTRACT_EXECUTION_PARAMS_READS=6
+CONTRACT_EXECUTION_PARAMS_EVENTS=2
+CONTRACT_EXECUTION_PARAMS_HASHES=10
+CONTRACT_EXECUTION_PARAMS_RECURSIVE_CALLS=0
+CONTRACT_EXECUTION_PARAMS_DEPLOYS=0
 ```
 
-Optional parameters:
+Similarly, to simulate a lightweight transaction:
 
-```sh
-# Optional seed to be used in the test: normally you don't need to set the seed,
-# but you can re-use seed from previous run to reproduce the sequence of operations locally.
-# Seed must be represented as a hexadecimal string.
-SEED
+```
+CONTRACT_EXECUTION_PARAMS_WRITES=0
+CONTRACT_EXECUTION_PARAMS_READS=0
+CONTRACT_EXECUTION_PARAMS_EVENTS=0
+CONTRACT_EXECUTION_PARAMS_HASHES=0
+CONTRACT_EXECUTION_PARAMS_RECURSIVE_CALLS=0
+CONTRACT_EXECUTION_PARAMS_DEPLOYS=0
 ```
 
-## Infrastructure relationship
+## Configuration
+
+For the full list of configuration options, see `loadnext/src/config.rs`.
 
-This crate is meant to be independent of the existing zkSync infrastructure. It is not integrated in `zk` and does not
-rely on `zksync_config` crate, and is not meant to be.
+Example invocation:
 
-The reason is that this application is meant to be used in CI, where any kind of configuration pre-requisites makes the
-tool harder to use:
+- transactions similar to mainnet
+- 300 accounts - should be enough to put full load to the sequencer
+- 20 transactions in flight - corresponds to the current limits on the mainnet and testnet
+- 20 minutes of testing - should be enough to properly estimate the TPS
+- As `L2_RPC_ADDRESS`, `L2_WS_RPC_ADDRESS`, `L1_RPC_ADDRESS` and `L1_RPC_ADDRESS` is not set, the test will run against
+  the local environment.
+- `MASTER_WALLET_PK` needs to be set to the private key of the master account.
+- `MAIN_TOKEN` needs to be set to the address of the token to be used for the loadtest.
 
-- Additional tools (e.g. `zk`) must be shipped together with the application, or included into the docker container.
-- Configuration that lies in files is harder to use in CI, due some sensitive data being stored in GITHUB_SECRETS.
+```
+cargo build
+
+CONTRACT_EXECUTION_PARAMS_WRITES=2 \
+CONTRACT_EXECUTION_PARAMS_READS=6 \
+CONTRACT_EXECUTION_PARAMS_EVENTS=2 \
+CONTRACT_EXECUTION_PARAMS_HASHES=10 \
+CONTRACT_EXECUTION_PARAMS_RECURSIVE_CALLS=0 \
+CONTRACT_EXECUTION_PARAMS_DEPLOYS=0 \
+ACCOUNTS_AMOUNT=300 \
+ACCOUNTS_GROUP_SIZE=300 \
+MAX_INFLIGHT_TXS=20 \
+RUST_LOG="info,loadnext=debug" \
+SYNC_API_REQUESTS_LIMIT=0 \
+SYNC_PUBSUB_SUBSCRIPTIONS_LIMIT=0 \
+TRANSACTION_WEIGHTS_DEPOSIT=0 \
+TRANSACTION_WEIGHTS_WITHDRAWAL=0 \
+TRANSACTION_WEIGHTS_L1_TRANSACTIONS=0 \
+TRANSACTION_WEIGHTS_L2_TRANSACTIONS=1 \
+DURATION_SEC=1200 \
+MASTER_WALLET_PK="..." \
+MAIN_TOKEN="..." \
+cargo run --bin loadnext
+```

core/tests/loadnext/src/account/mod.rs

@@ -15,6 +15,7 @@ use zksync_types::{
 };
 use zksync_web3_decl::jsonrpsee;
 
+use crate::utils::format_gwei;
 use crate::{
     account::{explorer_api_executor::ExplorerApiClient, tx_command_executor::SubmitResult},
     account_pool::{AddressPool, TestWallet},
@@ -173,7 +174,7 @@ impl AccountLifespan {
         // Due to natural sleep for sending tx, usually more than 1 tx can be already
         // processed and have a receipt
         let start = Instant::now();
-        vlog::debug!(
+        vlog::trace!(
             "Account {:?}: check_inflight_txs len {:?}",
             self.wallet.wallet.address(),
             self.inflight_txs.len()
@@ -186,17 +187,24 @@ impl AccountLifespan {
                         &transaction_receipt,
                         &tx.command.modifier.expected_outcome(),
                     );
-                    vlog::trace!(
-                        "Account {:?}: check_inflight_txs tx is included after {:?} attempt {:?}",
+                    let gas_used = transaction_receipt.gas_used.unwrap_or(U256::zero());
+                    let effective_gas_price = transaction_receipt
+                        .effective_gas_price
+                        .unwrap_or(U256::zero());
+                    vlog::debug!(
+                        "Account {:?}: tx included. Total fee: {}, gas used: {}gas, gas price: {} WEI. Latency {:?} at attempt {:?}",
                         self.wallet.wallet.address(),
+                        format_gwei(gas_used * effective_gas_price),
+                        gas_used,
+                        effective_gas_price,
                         tx.start.elapsed(),
-                        tx.attempt
+                        tx.attempt,
                     );
                     self.report(label, tx.start.elapsed(), tx.attempt, tx.command)
                         .await;
                 }
                 other => {
-                    vlog::debug!(
+                    vlog::trace!(
                         "Account {:?}: check_inflight_txs tx not yet included: {:?}",
                         self.wallet.wallet.address(),
                         other
@@ -206,7 +214,7 @@ impl AccountLifespan {
                 }
             }
         }
-        vlog::debug!(
+        vlog::trace!(
             "Account {:?}: check_inflight_txs complete {:?}",
             self.wallet.wallet.address(),
             start.elapsed()

core/tests/loadnext/src/account/tx_command_executor.rs

@@ -17,6 +17,7 @@ use zksync_types::{
 };
 
 use crate::account::ExecutionType;
+use crate::utils::format_gwei;
 use crate::{
     account::AccountLifespan,
     command::{IncorrectnessModifier, TxCommand, TxType},
@@ -403,6 +404,14 @@ impl AccountLifespan {
                 self.main_l2_token,
             )))
             .await?;
+        vlog::trace!(
+            "Account {:?}: fee estimated. Max total fee: {}, gas limit: {}gas; Max gas price: {}WEI, Gas per pubdata: {:?}gas",
+            self.wallet.wallet.address(),
+            format_gwei(fee.max_total_fee()),
+            fee.gas_limit,
+            fee.max_fee_per_gas,
+            fee.gas_per_pubdata_limit
+        );
         builder = builder.fee(fee.clone());
 
         let paymaster_params = get_approval_based_paymaster_input(

core/tests/loadnext/src/config.rs

@@ -30,10 +30,12 @@ pub struct LoadtestConfig {
     /// Amount of accounts to be used in test.
     /// This option configures the "width" of the test:
     /// how many concurrent operation flows will be executed.
+    /// The higher the value is, the more load will be put on the node.
+    /// If testing the sequencer throughput, this number must be sufficiently high.
     #[serde(default = "default_accounts_amount")]
     pub accounts_amount: usize,
 
-    /// Duration of the test.
+    /// Duration of the test. For proper results, this value should be at least 10 minutes.
     #[serde(default = "default_duration_sec")]
     pub duration_sec: u64,
 
@@ -44,7 +46,7 @@ pub struct LoadtestConfig {
     /// - Have `mint` operation.
     ///
     /// Note that we use ERC-20 token since we can't easily mint a lot of ETH on
-    /// Rinkeby or Ropsten without caring about collecting it back.
+    /// Testnets without caring about collecting it back.
     #[serde(default = "default_main_token")]
     pub main_token: Address,
 
@@ -78,7 +80,7 @@ pub struct LoadtestConfig {
     #[serde(default = "default_sync_api_requests_limit")]
     pub sync_api_requests_limit: usize,
 
-    /// Limits the number of simultaneous active PubSub subscriptions at any moment of time.
+    /// Limits the number of simultaneously active PubSub subscriptions at any moment of time.
     ///
     /// Setting it to:
     /// - 0 turns off PubSub subscriptions.
@@ -114,7 +116,8 @@ pub struct LoadtestConfig {
     #[serde(default = "default_l2_explorer_api_address")]
     pub l2_explorer_api_address: String,
 
-    /// The maximum number of transactions per account that can be sent without waiting for confirmation
+    /// The maximum number of transactions per account that can be sent without waiting for confirmation.
+    /// Should not exceed the corresponding value in the L2 node configuration.
     #[serde(default = "default_max_inflight_txs")]
     pub max_inflight_txs: usize,
 
@@ -127,6 +130,7 @@ pub struct LoadtestConfig {
 
     /// The expected number of the processed transactions during loadtest
     /// that should be compared to the actual result.
+    /// If the value is `None`, the comparison is not performed.
     #[serde(default = "default_expected_tx_count")]
     pub expected_tx_count: Option<usize>,
 
@@ -142,7 +146,6 @@ fn default_max_inflight_txs() -> usize {
 }
 
 fn default_l1_rpc_address() -> String {
-    // https://rinkeby.infura.io/v3/8934c959275444d480834ba1587c095f for rinkeby
     let result = "http://127.0.0.1:8545".to_string();
     vlog::info!("Using default L1_RPC_ADDRESS: {}", result);
     result

core/tests/loadnext/src/constants.rs

@@ -21,3 +21,18 @@ pub const MAX_OUTSTANDING_NONCE: usize = 20;
 /// Each account continuously sends API requests in addition to transactions. Such requests are considered failed
 /// after this amount of time elapsed without any server response.
 pub const API_REQUEST_TIMEOUT: Duration = Duration::from_secs(5);
+
+/// Max number of L1 transactions that can be sent in parallel.
+pub const MAX_L1_TRANSACTIONS: u64 = 10;
+
+/// Minimum amount of funds that should be present on the master account.
+/// Loadtest won't start if master account balance is less than this value.
+pub const MIN_MASTER_ACCOUNT_BALANCE: u64 = 2u64.pow(17);
+
+/// Minimum amount of funds that should be present on the paymaster account.
+/// Loadtest will deposit funds to the paymaster account if its balance is less than this value.
+pub const MIN_PAYMASTER_BALANCE: u64 = 10u64.pow(18) * 10;
+
+/// If the paymaster balance is lower than MIN_PAYMASTER_BALANCE,
+/// loadtest will deposit funds to the paymaster account so that its balance reaches this value
+pub const TARGET_PAYMASTER_BALANCE: u128 = 10u128.pow(18) * 20;