From 702520d22f1ce09841712a672f0e8260565afa1b Mon Sep 17 00:00:00 2001 From: "Dmitry K." Date: Thu, 27 Jun 2024 11:21:58 -0700 Subject: [PATCH 1/9] --wip-- [skip ci] --- .../docs/operate/testing/load_testing.md | 165 ++++++++---------- load-testing/README.md | 0 load-testing/loadtest_manifest_localnet.yaml | 9 +- 3 files changed, 83 insertions(+), 91 deletions(-) delete mode 100644 load-testing/README.md diff --git a/docusaurus/docs/operate/testing/load_testing.md b/docusaurus/docs/operate/testing/load_testing.md index c30604278..d3c9c601f 100644 --- a/docusaurus/docs/operate/testing/load_testing.md +++ b/docusaurus/docs/operate/testing/load_testing.md @@ -9,129 +9,114 @@ Poktroll load-testing suite. - [Overview](#overview) - [Dependencies](#dependencies) +- [Load Test manifests](#load-test-manifests) +- [Test Features](#test-features) - [How to run tests](#how-to-run-tests) - - [Common k6 CLI flags](#common-k6-cli-flags) -- [Understanding Output](#understanding-output) - - [CLI Output](#cli-output) - - [HTML Report](#html-report) - - [Adding a new load test](#adding-a-new-load-test) - - [Sophisticated Reporting](#sophisticated-reporting) -- [Troubleshooting](#troubleshooting) -- [File structure](#file-structure) + - [LocalNet](#localnet) + - [Reading the results](#reading-the-results) + - [Non-ephemeral networks (TestNets, MainNet, etc)](#non-ephemeral-networks-testnets-mainnet-etc) + - [Prerequisites](#prerequisites) + - [Modify the load test manifest](#modify-the-load-test-manifest) + - [Run the test](#run-the-test) + - [Reading the results](#reading-the-results-1) +- [How to write your own tests](#how-to-write-your-own-tests) ## Overview -We use [k6](https://k6.io/) for load testing. For detailed information about k6 internals and best practices, please refer to the [k6 documentation](https://grafana.com/docs/k6/latest/). +We built a load-testing suite on top of [Gherkin](https://cucumber.io/docs/gherkin/) which allows to write simple and human readable tests. ## Dependencies -- [k6](https://grafana.com/docs/k6/latest/get-started/installation/) - (For local suite execution) [LocalNet](../infrastructure/localnet.md) +- [Golang](https://go.dev/dl/) -## How to run tests +## Load Test manifests -Tests are stored in the [load-testing/tests](https://github.com/pokt-network/poktroll/tree/main/load-testing/tests) folder, covering various use cases. +Load test manifests are YAML files that describe the environment of the network the test can be run against. Properties +such as what blockchain address to use to fund and stake applications, what suppliers and gateways to use are +all covered in the manifest YAML file. [The LocalNet's manifest](https://github.com/pokt-network/poktroll/blob/main/load-testing/loadtest_manifest_localnet.yaml) +can be used as an example - it includes comments for each property in the manefest. -For instance, here's a basic load test sending requests through `AppGateServer`, which proxies them to `RelayMiner`: +## Test Features -```bash -k6 run load-testing/tests/appGateServerEtherium.js +Test features are stored in the [load-testing/tests](https://github.com/pokt-network/poktroll/tree/main/load-testing/tests) directory, +covering various use cases. + +As the load-testing suite is built on top of Gherkin, the features files contain human-readable load tests. +For example, here is a simple feature that checks if the `anvil` node can handle the maximum number of concurrent users: + +``` +Feature: Loading anvil node only + Scenario Outline: Anvil can handle the maximum number of concurrent users + Given anvil is running + And load of concurrent requests for the "eth_blockNumber" JSON-RPC method + Then load is handled within seconds + + Examples: + | num_requests | timeout | + | 10 | 1 | + | 100 | 1 | + | 1000 | 5 | + | 10000 | 10 | ``` -### Common k6 CLI flags +The natural human-readable text is parsed by [gocuke](https://github.com/regen-network/gocuke). -The `k6` load testing tool provides various command-line flags to customize and control your load tests. Below are some of the key flags that you can use: +## How to run tests -- `--vus`: Specifies the number of virtual users (VUs) to simulate. This flag allows you to set the concurrency level of your load test. +### LocalNet - - Example: `k6 run script.js --vus=50` runs the test with 50 virtual users. +We have a handy make target to run the tests on the LocalNet. -- `--duration`: Defines the duration for which the test should run. You can specify the time in seconds (s), minutes (m), or hours (h). +1. Make sure [LocalNet](../infrastructure/localnet.md) is up and running; +2. In your `localnet_config.yaml` file, make sure to set `gateways.count` and `relayminers.count` to `3`; +3. Run `make acc_initialize_pubkeys` to initialize the public keys in the blockchain state; +4. Run `make test_load_relays_stress_localnet` to run all tests on LocalNet. - - Example: `k6 run script.js --duration=1m` runs the test for 1 minute. +#### Reading the results -- `--http-debug`: Enables HTTP debugging. This flag can be set to `full` for detailed logging of all HTTP requests and responses. Useful for troubleshooting and debugging your tests. - - Use `--http-debug` for a summary of HTTP requests. - - Use `--http-debug=full` for detailed request and response logging. - - Example: `k6 run script.js --http-debug=full` provides a detailed log of HTTP transactions. +- The CLI output shows standard Go test output. If there are no issues during the test execution, you'll see `PASS`, otherwise the test will show `FAIL` and the error that caused the test to fail will be shown. +- As the test progresses, the obserability stack continously gathers the metric data from off-chain actors. On LocalNet, [Grafana can be accessed on 3003 port](http://localhost:3003/?orgId=1). `Stress test` and `Load Testing` dashboards can be helpful to understand current state of the system. -The default configurations for VUs and duration are in the [config file](https://github.com/pokt-network/poktroll/tree/main/load-testing/config/index.js). Override these defaults by passing the appropriate flags to the `k6` command. For example: +### Non-ephemeral networks (TestNets, MainNet, etc) -```bash -k6 run load-testing/tests/appGateServerEtherium.js --vus=300 --duration=30s -``` +Such networks have been generated with random addresses, so we need to modify the load test manifest to reflect +the accounts from that network. -For a comprehensive list of available flags and their usage, refer to the [k6 documentation](https://grafana.com/docs/k6/latest/). +::: info +Such networks usually have other participants and load testing can be performed against the off-chain actors deployed by +other people. As a result of running the test against the software you don't control and can't observe - you won't +get the metrics and logs. If you wish to gather metrics, logs and look at behavior of the off-chain actors, you can +create a new service and deploy your own gateways and suppliers, and run the test against that new service. As a result +you'll get full observability information from the software you deployed. +::: -## Understanding Output +#### Prerequisites -### CLI Output +- An address with tokens that will be used to fund and stake applications. It must be available on the local keychain (e.g. `poktrolld keys list`) +- A list of gateways to issue requests to. They could be gateways hosted by other people, or they can be your own gateways. +- If you are running a test on a custom service, then make sure the suppliers are set up and ready to accept requests. -Post-test, basic performance metrics are available: +#### Modify the load test manifest -``` - scenarios: (100.00%) 1 scenario, 100 max VUs, 1m30s max duration (incl. graceful stop): - * default: 100 looping VUs for 1m0s (gracefulStop: 30s) - -INFO[0061] [k6-reporter v2.3.0] Generating HTML summary report source=console - ✓ is status 200 - ✓ is successful JSON-RPC response - - checks.........................: 100.00% ✓ 12000 ✗ 0 - data_received..................: 1.6 MB 27 kB/s - data_sent......................: 1.2 MB 19 kB/s - http_req_blocked...............: avg=52.52µs min=0s med=2µs max=7.08ms p(90)=4µs p(95)=6µs - http_req_connecting............: avg=37.73µs min=0s med=0s max=5.48ms p(90)=0s p(95)=0s - http_req_duration..............: avg=8.51ms min=1.58ms med=5.98ms max=81.98ms p(90)=15.45ms p(95)=19.97ms - { expected_response:true }...: avg=8.51ms min=1.58ms med=5.98ms max=81.98ms p(90)=15.45ms p(95)=19.97ms - http_req_failed................: 0.00% ✓ 0 ✗ 6000 - http_req_receiving.............: avg=42.18µs min=5µs med=25µs max=5.71ms p(90)=65µs p(95)=125µs - http_req_sending...............: avg=26.5µs min=2µs med=9µs max=7.2ms p(90)=21µs p(95)=42µs - http_req_tls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s - http_req_waiting...............: avg=8.44ms min=1.55ms med=5.93ms max=81.28ms p(90)=15.38ms p(95)=19.82ms - http_reqs......................: 6000 98.915624/s - iteration_duration.............: avg=1s min=1s med=1s max=1.08s p(90)=1.01s p(95)=1.02s - iterations.....................: 6000 98.915624/s - vus............................: 100 min=100 max=100 - -running (1m00.7s), 000/100 VUs, 6000 complete and 0 interrupted iterations -default ✓ [======================================] 100 VUs 1m0s -``` +Using [loadtest_manifest_example.yaml](https://github.com/pokt-network/poktroll/blob/main/load-testing/loadtest_manifest_example.yaml) +as a reference, modify it to reflect the values for your test. -### HTML Report +#### Run the test -An HTML report is generated in the execution directory. Open it in your default browser: +We have a handy makefile target to run the `relays_stress.feature` with the modified manifest (`loadtest_manifest_example.yaml`): ```bash -open summary.html +make test_load_relays_stress_example ``` -### Adding a new load test - -TODO_DOCUMENT(@okdas): Add link to PR next time a new type of load test is added. - -### Sophisticated Reporting - -We're developing advanced reporting that integrates additional tags set in the code. This will require time-series databases and is planned for DevNets. +#### Reading the results -## Troubleshooting +If the test was ran against the suppliers and gateways that are not hosted by you, but rather the community members, you can +only look at the transactions on blockchain and the test output. If you deployed your own service, then you will have full observability +and you should be able to see all the metrics, logs and behavior of the system under load in Grafana or any other monitoring tool that is set up for your service. -To debug, activate the logging feature: +## How to write your own tests -`--http-debug` or `--http-debug=full` - -## File structure - -``` -load-testing -├── README.md -├── config -│ └── index.js -├── modules # reusable code for scenarios and tests -│ └── etheriumRequests.js -├── scenarios # different scenarios for tests -│ └── requestBlockNumberEtherium.js -└── tests # test scripts - ├── anvilDirectEtherium.js - └── appGateServerEtherium.js -``` +Please refer to the [gocuke documentation](https://github.com/regen-network/gocuke?tab=readme-ov-file#quick-start). You +can also use a simple [anvil test](https://github.com/pokt-network/poktroll/blob/main/load-testing/tests/anvil_test.go) as a reference. \ No newline at end of file diff --git a/load-testing/README.md b/load-testing/README.md deleted file mode 100644 index e69de29bb..000000000 diff --git a/load-testing/loadtest_manifest_localnet.yaml b/load-testing/loadtest_manifest_localnet.yaml index ff952d390..e75e55bf0 100644 --- a/load-testing/loadtest_manifest_localnet.yaml +++ b/load-testing/loadtest_manifest_localnet.yaml @@ -1,6 +1,7 @@ # NB: The number of pre-provisioned **LocalNet** actors are managed in # 'localnet_config.yaml' by the respective actors `count` property. -is_ephemeral_chain: true + +is_ephemeral_chain: true # This should be `true` for LocalNet as it is an ephemeral network # The service ID to use for the load test. service_id: anvil @@ -12,6 +13,9 @@ funding_account_address: pokt1eeeksh2tvkh7wzmfrljnhw4wrhs55lcuvmekkw # address f # List of pre-provisioned suppliers used for load testing. # These suppliers will be progressively staked during the load test, according # to the test plan that is being executed. +# +# When running on **LocalNet**, the number of the suppliers defined below should match +# the `relayminers.count` in your `localnet_config.yaml`. suppliers: # The supplier address that is available in the load test's environment keyring, # used to identify the supplier and sign relays and transactions with. @@ -33,6 +37,9 @@ suppliers: # List of pre-provisioned gateways used for load testing. # These gateways will be progressively staked and delegated to during the load test. +# +# When running on LocalNet, the number of the gateways defined below should match +# the `gateways.count` in your `localnet_config.yaml`. gateways: # The gateway address that is available in the load test's environment keyring, # used to identify the gateway and sign relays and transactions with. From 98fbc2782a371e77602a2226d6eae4a3937eb063 Mon Sep 17 00:00:00 2001 From: "Dmitry K." Date: Thu, 27 Jun 2024 18:03:21 -0700 Subject: [PATCH 2/9] add DevNet load testing docs --- .gitignore | 3 + .../docs/operate/infrastructure/devnet.md | 120 +++++++++++++++++- .../docs/operate/testing/load_testing.md | 109 +++++++--------- .../operate/testing/load_testing_devnet.md | 117 +++++++++++++++++ .../operate/testing/load_testing_plan_1.md | 26 ++-- load-testing/loadtest_manifest_example.yaml | 19 ++- 6 files changed, 308 insertions(+), 86 deletions(-) create mode 100644 docusaurus/docs/operate/testing/load_testing_devnet.md diff --git a/.gitignore b/.gitignore index 5c06b0162..cd0856e5c 100644 --- a/.gitignore +++ b/.gitignore @@ -95,3 +95,6 @@ gentx-*.json # Binaries VSCode builds for debugging cmd/poktrolld/*debug_bin* + +# The temporary file created by e2e scripts +job.yaml \ No newline at end of file diff --git a/docusaurus/docs/operate/infrastructure/devnet.md b/docusaurus/docs/operate/infrastructure/devnet.md index 228789f79..d45c3a493 100644 --- a/docusaurus/docs/operate/infrastructure/devnet.md +++ b/docusaurus/docs/operate/infrastructure/devnet.md @@ -1,9 +1,9 @@ --- sidebar_position: 3 -title: DevNet +title: DevNets --- -# DevNet +# DevNets :::note @@ -20,6 +20,15 @@ to set up your access to GCP. - [Grafana logs](#grafana-logs) - [Infrastructure Provisioning](#infrastructure-provisioning) - [Configuration](#configuration) +- [Interaction with DevNet](#interaction-with-devnet) +- [DevNet types](#devnet-types) +- [Manually provisioned DevNets](#manually-provisioned-devnets) + - [How to create](#how-to-create) + - [How to delete](#how-to-delete) + - [Configure and update the version of the software](#configure-and-update-the-version-of-the-software) + - [Scale actors up and down](#scale-actors-up-and-down) + - [Run e2e tests on a manually provisioned DevNet](#run-e2e-tests-on-a-manually-provisioned-devnet) + - [Stake actors](#stake-actors) ## GCP Console @@ -54,3 +63,110 @@ Each DevNet ArgoCD App (following the App of Apps pattern) provisions a Helm cha Each `full-network` includes other ArgoCD applications that deploy Validators and off-chain actors. Each Helm chart receives a list of configuration files. For example, see the [relayminer configuration](https://github.com/pokt-network/protocol-infra/blob/main/charts/full-network/templates/Application-Relayminer.yaml#L37). All possible values can be found in the `values.yaml` of the Helm chart, such as the [relayminer Helm chart](https://github.com/pokt-network/helm-charts/blob/main/charts/relayminer/values.yaml). + +## Interaction with DevNet + +:::note +Devnets are provisioned with the same mnemonc phrases as LocalNet, so it is possible to reuse the same keys +from the keybase - the user just need to change the `--node=` argument to point to the DevNet RPC endpoint. +::: + +The DevNet RPC endpoint is exposed on `https://devnet-**NETWORK_NAME**-validator-rpc.poktroll.com`. + +It is possible to reuse the LocalNet make targets with DevNet endpoints. For example + +```bash +POCKET_NODE=https://devnet-issue-420-validator-rpc.poktroll.com make supplier_list +``` + +## DevNet types + +We have two types of DevNets: +1. Automatically provisioned by `devnet-test-e2e` GitHub label. + - Their main purpose is to automatically run e2e tests for each PR this label has been assigned to. + - They are not easily modified and configured. E.g. we automatically provision single instance of each actor only. +2. Manually provisioned by creating a file in [protocol-infra repo under `devnet-configs` directory](https://github.com/pokt-network/protocol-infra/tree/main/devnets-configs). + - Their main purpose is for testing new features or configurations that require more complex setups than the first type can provide. + - They are easily modified and configured, allowing you to add multiple instances of each actor. + +## Manually provisioned DevNets + +### How to create + +Create a new YAML file in [devnets-configs](https://github.com/pokt-network/protocol-infra/tree/main/devnets-configs). +Use [template](https://github.com/pokt-network/protocol-infra/blob/main/devnets-configs/_TEMPLATE_YAML_) as a reference. + +### How to delete + +Remove the devnet config file. + +### Configure and update the version of the software + +We create a new image on all commits created in `main` branch and non-main branches if the PR has a `push-image` label. +Images named in the following format: + +``` +ghcr.io/pokt-network/poktrolld:sha-7042be3 +ghcr.io/pokt-network/poktrolld:sha-7042be3922245fb4313ee90e1f28d0f402b700a0 +``` + +You can update the version of DevNet by changing this parameter in the devnet config file: + +```yaml +image: + tag: sha-7042be3 +``` + +As all parameters in this config actually passed to the [downstream helm chart](https://github.com/pokt-network/protocol-infra/tree/main/charts/full-network) via an [ArgoCD Application](https://github.com/pokt-network/protocol-infra/blob/main/clusters/protocol-us-central1/devnets-persistent.yaml), +it is possible to modify some other parameters besides the image tag. Here is a [list of all options](https://github.com/pokt-network/protocol-infra/blob/main/charts/full-network/values.yaml). + +### Scale actors up and down + +You can modify the number of each actor by changing the devnet config file: + +:::info +We use the same ignite `config.yaml` to provision a genesis as we use on devnets. That means we should not provision more +actors than we configure in ignite's `config.yaml`. Good rule of thumb: don't go over `3`. +::: + + +```yaml +appgateservers: + count: 1 +gateways: + count: 1 +relayminers: + count: 1 +``` + + +### Run e2e tests on a manually provisioned DevNet + +1. The image tag must match the tag of the image from the devnet config YAML file. +2. The name of the devnet in the environment variables must be specified. +3. The kubernetes context must be changed to the protocol cluster (`kubectl config set-context gke_protocol-us-central1-d505_us-central1_protocol-us-central1`) +4. Command should be executed from the root of the poktroll repo. + +The command: +```bash +IMAGE_TAG=**IMAGE TAG NAME FROM DEVNET CONFIG** NAMESPACE=devnet-**NETWORK NAME** JOB_NAME=e2e-test-**GITSHA FROM IMAGE TAG** POCKET_NODE=tcp://devnet-**NETWORK NAME**-validator-poktrolld:26657 bash .github/workflows-helpers/run-e2e-test.sh +``` + +For example: +```bash +IMAGE_TAG=sha-7042be3 NAMESPACE=devnet-dimatests JOB_NAME=e2e-test-7042be3 POCKET_NODE=tcp://devnet-dimatests-validator-poktrolld:26657 bash .github/workflows-helpers/run-e2e-test.sh +``` + +### Stake actors + +Since the keys are the same as LocalNet, we can use the same commands for DevNet to stake the actors. Also, DevNets +match the hostnames with LocalNet, which makes it possible for the same stake configs to work on different networks. +For example, this command stakes supplier2 on DevNet `devnet-sophon`: + +```bash +POCKET_NODE=https://devnet-sophon-validator-rpc.poktroll.com make supplier2_stake +``` + +:::note +Only manually provisioned DevNets have more than one actor to stake. +::: \ No newline at end of file diff --git a/docusaurus/docs/operate/testing/load_testing.md b/docusaurus/docs/operate/testing/load_testing.md index d3c9c601f..f7efa696f 100644 --- a/docusaurus/docs/operate/testing/load_testing.md +++ b/docusaurus/docs/operate/testing/load_testing.md @@ -3,49 +3,45 @@ sidebar_position: 1 title: Load Testing --- -# Load Testing +# Load Testing Poktroll load-testing suite. -- [Overview](#overview) -- [Dependencies](#dependencies) -- [Load Test manifests](#load-test-manifests) -- [Test Features](#test-features) -- [How to run tests](#how-to-run-tests) - - [LocalNet](#localnet) - - [Reading the results](#reading-the-results) - - [Non-ephemeral networks (TestNets, MainNet, etc)](#non-ephemeral-networks-testnets-mainnet-etc) - - [Prerequisites](#prerequisites) - - [Modify the load test manifest](#modify-the-load-test-manifest) - - [Run the test](#run-the-test) - - [Reading the results](#reading-the-results-1) -- [How to write your own tests](#how-to-write-your-own-tests) +- [Load Testing](#load-testing) + - [Overview](#overview) + - [Dependencies](#dependencies) + - [Load Test Manifests](#load-test-manifests) + - [Test Features](#test-features) + - [Executing Tests](#executing-tests) + - [LocalNet Environment](#localnet-environment) + - [Interpreting Results](#interpreting-results) + - [Non-Ephemeral Networks (TestNets, MainNet, etc)](#non-ephemeral-networks-testnets-mainnet-etc) + - [Prerequisites](#prerequisites) + - [Manifest Modification](#manifest-modification) + - [Test Execution](#test-execution) + - [Result Analysis](#result-analysis) + - [Developing Custom Tests](#developing-custom-tests) ## Overview -We built a load-testing suite on top of [Gherkin](https://cucumber.io/docs/gherkin/) which allows to write simple and human readable tests. +The load-testing suite is built on [Gherkin](https://cucumber.io/docs/gherkin/), enabling the creation of simple and human-readable tests. ## Dependencies -- (For local suite execution) [LocalNet](../infrastructure/localnet.md) +- [LocalNet](../infrastructure/localnet.md) (for local suite execution) - [Golang](https://go.dev/dl/) -## Load Test manifests +## Load Test Manifests -Load test manifests are YAML files that describe the environment of the network the test can be run against. Properties -such as what blockchain address to use to fund and stake applications, what suppliers and gateways to use are -all covered in the manifest YAML file. [The LocalNet's manifest](https://github.com/pokt-network/poktroll/blob/main/load-testing/loadtest_manifest_localnet.yaml) -can be used as an example - it includes comments for each property in the manefest. +Load test manifests are YAML files that define the network environment for test execution. These files specify properties such as blockchain addresses for funding and staking applications, and the suppliers and gateways to utilize. The [LocalNet manifest](https://github.com/pokt-network/poktroll/blob/main/load-testing/loadtest_manifest_localnet.yaml) serves as a comprehensive example, including detailed comments for each manifest property. ## Test Features -Test features are stored in the [load-testing/tests](https://github.com/pokt-network/poktroll/tree/main/load-testing/tests) directory, -covering various use cases. +Test features are located in the [load-testing/tests](https://github.com/pokt-network/poktroll/tree/main/load-testing/tests) directory, encompassing various use cases. -As the load-testing suite is built on top of Gherkin, the features files contain human-readable load tests. -For example, here is a simple feature that checks if the `anvil` node can handle the maximum number of concurrent users: +Leveraging Gherkin, the feature files contain human-readable load tests. For instance: -``` +```gherkin Feature: Loading anvil node only Scenario Outline: Anvil can handle the maximum number of concurrent users Given anvil is running @@ -60,63 +56,54 @@ Feature: Loading anvil node only | 10000 | 10 | ``` -The natural human-readable text is parsed by [gocuke](https://github.com/regen-network/gocuke). +This natural language syntax is parsed by [gocuke](https://github.com/regen-network/gocuke). -## How to run tests +## Executing Tests -### LocalNet +### LocalNet Environment -We have a handy make target to run the tests on the LocalNet. +To execute tests on LocalNet: -1. Make sure [LocalNet](../infrastructure/localnet.md) is up and running; -2. In your `localnet_config.yaml` file, make sure to set `gateways.count` and `relayminers.count` to `3`; -3. Run `make acc_initialize_pubkeys` to initialize the public keys in the blockchain state; -4. Run `make test_load_relays_stress_localnet` to run all tests on LocalNet. +1. Ensure [LocalNet](../infrastructure/localnet.md) is operational. +2. In the `localnet_config.yaml` file, set `gateways.count` and `relayminers.count` to `3`. +3. Execute `make acc_initialize_pubkeys` to initialize blockchain state public keys. +4. Run `make test_load_relays_stress_localnet` to run the LocalNet stress-test. -#### Reading the results +#### Interpreting Results -- The CLI output shows standard Go test output. If there are no issues during the test execution, you'll see `PASS`, otherwise the test will show `FAIL` and the error that caused the test to fail will be shown. -- As the test progresses, the obserability stack continously gathers the metric data from off-chain actors. On LocalNet, [Grafana can be accessed on 3003 port](http://localhost:3003/?orgId=1). `Stress test` and `Load Testing` dashboards can be helpful to understand current state of the system. +- The CLI output displays standard Go test results. Successful tests are indicated by `PASS`, while failures are denoted by `FAIL` with accompanying error messages. +- During test execution, the observability stack continuously collects metric data from off-chain actors. On LocalNet, [Grafana is accessible on port 3003](http://localhost:3003/?orgId=1). The "Stress test" and "Load Testing" dashboards provide valuable insights into system status. -### Non-ephemeral networks (TestNets, MainNet, etc) +### Non-Ephemeral Networks (TestNets, MainNet, etc) -Such networks have been generated with random addresses, so we need to modify the load test manifest to reflect -the accounts from that network. +These networks are generated with random addresses, necessitating modifications to the load test manifest to reflect network-specific accounts. -::: info -Such networks usually have other participants and load testing can be performed against the off-chain actors deployed by -other people. As a result of running the test against the software you don't control and can't observe - you won't -get the metrics and logs. If you wish to gather metrics, logs and look at behavior of the off-chain actors, you can -create a new service and deploy your own gateways and suppliers, and run the test against that new service. As a result -you'll get full observability information from the software you deployed. +:::info +Note: Such networks typically involve other participants, allowing load testing against off-chain actors deployed by third parties. Consequently, metrics and logs may not be available when testing against uncontrolled software. For comprehensive observability, consider creating a new service with custom gateways and suppliers, and conduct tests against this controlled environment. ::: #### Prerequisites -- An address with tokens that will be used to fund and stake applications. It must be available on the local keychain (e.g. `poktrolld keys list`) -- A list of gateways to issue requests to. They could be gateways hosted by other people, or they can be your own gateways. -- If you are running a test on a custom service, then make sure the suppliers are set up and ready to accept requests. +- An address with sufficient tokens for application funding and staking, accessible on the local keychain (e.g., `poktrolld keys list`) +- A list of target gateways for request issuance +- For custom service testing, ensure suppliers are configured and ready to process requests -#### Modify the load test manifest +#### Manifest Modification -Using [loadtest_manifest_example.yaml](https://github.com/pokt-network/poktroll/blob/main/load-testing/loadtest_manifest_example.yaml) -as a reference, modify it to reflect the values for your test. +Using [loadtest_manifest_example.yaml](https://github.com/pokt-network/poktroll/blob/main/load-testing/loadtest_manifest_example.yaml) as a template, modify the values to align with the test requirements. -#### Run the test +#### Test Execution -We have a handy makefile target to run the `relays_stress.feature` with the modified manifest (`loadtest_manifest_example.yaml`): +Utilize the provided makefile target to run the `relays_stress.feature` with the modified manifest: ```bash make test_load_relays_stress_example ``` -#### Reading the results +#### Result Analysis -If the test was ran against the suppliers and gateways that are not hosted by you, but rather the community members, you can -only look at the transactions on blockchain and the test output. If you deployed your own service, then you will have full observability -and you should be able to see all the metrics, logs and behavior of the system under load in Grafana or any other monitoring tool that is set up for your service. +For tests conducted against community-hosted suppliers and gateways, analysis is limited to blockchain transactions and test output. When testing against a custom-deployed service, comprehensive observability is available, including metrics, logs, and system behavior under load, accessible through Grafana or other configured monitoring tools. -## How to write your own tests +## Developing Custom Tests -Please refer to the [gocuke documentation](https://github.com/regen-network/gocuke?tab=readme-ov-file#quick-start). You -can also use a simple [anvil test](https://github.com/pokt-network/poktroll/blob/main/load-testing/tests/anvil_test.go) as a reference. \ No newline at end of file +For custom test development, refer to the [gocuke documentation](https://github.com/regen-network/gocuke?tab=readme-ov-file#quick-start). The [anvil test](https://github.com/pokt-network/poktroll/blob/main/load-testing/tests/anvil_test.go) provides a practical reference implementation. \ No newline at end of file diff --git a/docusaurus/docs/operate/testing/load_testing_devnet.md b/docusaurus/docs/operate/testing/load_testing_devnet.md new file mode 100644 index 000000000..a7eadc79b --- /dev/null +++ b/docusaurus/docs/operate/testing/load_testing_devnet.md @@ -0,0 +1,117 @@ +--- +sidebar_position: 2 +title: Load Testing - DevNet +--- + +# Load Testing on DevNets + +A guide on how to perform load testing on DevNets. + +- [Load Testing on DevNets](#load-testing-on-devnets) + - [Overview](#overview) + - [Prerequisites](#prerequisites) + - [1. Create and configure the TestNet](#1-create-and-configure-the-testnet) + - [2. Stake the necessary actors](#2-stake-the-necessary-actors) + - [3. Configure the load testing manifest](#3-configure-the-load-testing-manifest) + - [Full example](#full-example) + + +## Overview + +We can create DevNets that are suitable for running load-tests. + +:::warning +DevNets created with GitHub PRs using `devnet-test-e2e` tags are not suitable for load testing, as they only provision a +single instance of each off-chain actor. We can create custom DevNets with multiple instances of each off-chain actor for load testing purposes. +::: + +## Prerequisites + +### 1. Create and configure the TestNet + +Please refer to DevNet creation guide [here](../infrastructure/devnet.md#how-to-create) + +### 2. Stake the necessary actors + +- Depending on your LoadTesting requirement, you may need to stake one or more `gateways` and `suppliers`; +- [DevNet documentation](../infrastructure/devnet.md#stake-actors) provides more details about staking actors in DevNets; +- Note to save the addresses of + +### 3. Configure the load testing manifest + +[Load Testing documentation](./load_testing.md#manifest-modification) provides information on how the load test manifest +can be modified to run against DevNets. DevNets are not much different from TestNets, but they do not have randomly +generated addresses. Instead, the addresses are shared with LocalNet for convinience. + +## Full example + +1. DevNet `sophon` can be created with the following devnet-config file: + +```yaml +networkName: "sophon" + +image: + tag: sha-7042be3 + +appgateservers: + count: 1 +gateways: + count: 3 +relayminers: + count: 3 +``` + +2. Gateways and suppliers can be staked using the following commands: + +::info +`supplier1` and `gateway1` are pre-staked as a part of genesis generation process. +::: + +```bash +POCKET_NODE=https://devnet-sophon-validator-rpc.poktroll.com make gateway2_stake +POCKET_NODE=https://devnet-sophon-validator-rpc.poktroll.com make gateway3_stake +POCKET_NODE=https://devnet-sophon-validator-rpc.poktroll.com make supplier2_stake +POCKET_NODE=https://devnet-sophon-validator-rpc.poktroll.com make supplier3_stake +``` + +3. Update manifest. The content of +[loadtest_manifest_example.yaml](https://github.com/pokt-network/poktroll/blob/main/load-testing/loadtest_manifest_example.yaml) +can be modified as follows: + +```yaml +# This file is used to configure the load test for non-ephemeral chains. +# It is intended to target a remote environment, such as a devnet or testnet. +is_ephemeral_chain: false + +# testnet_node is the URL of the node that the load test will use to query the +# chain and submit transactions. +testnet_node: https://devnet-sophon-validator-rpc.poktroll.com + +# The service ID to request relays from. +service_id: "anvil" + +# The address of the account that will be used to fund the the application accounts +# so that they can stake on the network. +funding_account_address: pokt1eeeksh2tvkh7wzmfrljnhw4wrhs55lcuvmekkw # address for pnf account + +# In non-ephemeral chains, the gateways are identified by their address. +gateways: + - address: pokt15vzxjqklzjtlz7lahe8z2dfe9nm5vxwwmscne4 + exposed_url: https://devnet-sophon-gateway-1.poktroll.com + - address: pokt15w3fhfyc0lttv7r585e2ncpf6t2kl9uh8rsnyz + exposed_url: https://devnet-sophon-gateway-2.poktroll.com + - address: pokt1zhmkkd0rh788mc9prfq0m2h88t9ge0j83gnxya + exposed_url: https://devnet-sophon-gateway-3.poktroll.com +``` + +4. Run the test + +```bash +make test_load_relays_stress_example +``` + +5. Observe the results + +You can see the performance of the requests on [Grafana dashboards](https://grafana.poktroll.com/d/nginx/nginx-ingress-controller). +The DevNets have LoadBalancers which allows to get more metrics about network load and latency. When looking +at the `NGINX Ingress controller` Dashboard, make sure to change the namespace to match the DevNet name. diff --git a/docusaurus/docs/operate/testing/load_testing_plan_1.md b/docusaurus/docs/operate/testing/load_testing_plan_1.md index 601cb77c0..33f717bea 100644 --- a/docusaurus/docs/operate/testing/load_testing_plan_1.md +++ b/docusaurus/docs/operate/testing/load_testing_plan_1.md @@ -1,5 +1,5 @@ --- -sidebar_position: 2 +sidebar_position: 5 title: Load Testing - Plan #1 --- @@ -126,10 +126,10 @@ pie showData | | RAM | CPU | Network | Disk | Time | | ----------------------- | --- | --- | ------- | ---- | ---- | -| Proof Validation | ❓ | ❓ | | | ❓ | -| Block Generation | ❓ | | | | | -| Block Publishing | | | ❓ | | | -| Data Availability State | | | | ❓ | | +| Proof Validation | ❓ | ❓ | | | ❓ | +| Block Generation | ❓ | | | | | +| Block Publishing | | | ❓ | | | +| Data Availability State | | | | ❓ | | #### 3. AppGate Server (Application, Gateway, etc…) @@ -144,10 +144,10 @@ pie showData | | RAM | CPU | Network | Disk | Time | | ------------------ | --- | --- | ------- | ---- | ---- | -| Relay Proxies | | | ❓ | | ❓ | -| Caches & State | ❓ | ❓ | | | | -| Request Processing | | | | | ❓ | -| ??? | | | | ❓ | | +| Relay Proxies | | | ❓ | | ❓ | +| Caches & State | ❓ | ❓ | | | | +| Request Processing | | | | | ❓ | +| ??? | | | | ❓ | | #### 4. RelayMiner (Supplier, SMT, etc..) @@ -162,10 +162,10 @@ pie showData | | RAM | CPU | Network | Disk | Time | | ------------------- | --- | --- | ------- | ---- | ---- | -| SMT | ❓ | ❓ | | ❓ | ❓ | -| Caches & State | ❓ | | | | | -| Request Processing | | | | | ❓ | -| Response Generation | | ❓ | ❓ | | ❓ | +| SMT | ❓ | ❓ | | ❓ | ❓ | +| Caches & State | ❓ | | | | | +| Request Processing | | | | | ❓ | +| Response Generation | | ❓ | ❓ | | ❓ | ### Out-of-scope diff --git a/load-testing/loadtest_manifest_example.yaml b/load-testing/loadtest_manifest_example.yaml index 33a2fc6e0..4c90d367a 100644 --- a/load-testing/loadtest_manifest_example.yaml +++ b/load-testing/loadtest_manifest_example.yaml @@ -1,24 +1,23 @@ -# The load test uses the provided gateways and the staked suppliers to send relays. - # This file is used to configure the load test for non-ephemeral chains. # It is intended to target a remote environment, such as a devnet or testnet. is_ephemeral_chain: false # testnet_node is the URL of the node that the load test will use to query the # chain and submit transactions. -testnet_node: https://testnet-validated-validator-rpc.poktroll.com +testnet_node: https://devnet-dimatests-validator-rpc.poktroll.com # The service ID to request relays from. -service_id: "0021" +service_id: "anvil" # The address of the account that will be used to fund the the application accounts # so that they can stake on the network. -funding_account_address: pokt14eh973xt99s7edugnyvd5d5u50d6j0hysw2vsm # address for pnf account +funding_account_address: pokt1eeeksh2tvkh7wzmfrljnhw4wrhs55lcuvmekkw # address for pnf account # In non-ephemeral chains, the gateways are identified by their address. gateways: - # address is the bech32 pokt address of the gateway. - - address: pokt16sty9mjdh4u2fwgj8ptufg42cysvh6gsyx6wfp - # The url used to send relays to the gateway on. - # TODO_INVESTIGATE(@okdas): Why is this port number so low? What should it be? - exposed_url: http://localhost:84 + - address: pokt15vzxjqklzjtlz7lahe8z2dfe9nm5vxwwmscne4 + exposed_url: https://devnet-dimatests-gateway-1.poktroll.com + - address: pokt15w3fhfyc0lttv7r585e2ncpf6t2kl9uh8rsnyz + exposed_url: https://devnet-dimatests-gateway-2.poktroll.com + - address: pokt1zhmkkd0rh788mc9prfq0m2h88t9ge0j83gnxya + exposed_url: https://devnet-dimatests-gateway-3.poktroll.com From e79cd81c1a15cff6bf1f9f350c2f72910f38301b Mon Sep 17 00:00:00 2001 From: "Dmitry K." Date: Thu, 27 Jun 2024 18:06:50 -0700 Subject: [PATCH 3/9] resolve a docusarus warning --- docusaurus/docs/operate/testing/load_testing_devnet.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docusaurus/docs/operate/testing/load_testing_devnet.md b/docusaurus/docs/operate/testing/load_testing_devnet.md index a7eadc79b..1ea619855 100644 --- a/docusaurus/docs/operate/testing/load_testing_devnet.md +++ b/docusaurus/docs/operate/testing/load_testing_devnet.md @@ -63,7 +63,7 @@ relayminers: 2. Gateways and suppliers can be staked using the following commands: -::info +:::info `supplier1` and `gateway1` are pre-staked as a part of genesis generation process. ::: From 1eac4ddbaec39e07e5ff10c58d9bef5a8736eafb Mon Sep 17 00:00:00 2001 From: "Dmitry K." Date: Thu, 27 Jun 2024 18:27:55 -0700 Subject: [PATCH 4/9] some proofreading --- .../operate/testing/load_testing_devnet.md | 30 ++++++++-------- .../operate/testing/load_testing_testnet.md | 36 +++++++++++++++++++ 2 files changed, 50 insertions(+), 16 deletions(-) create mode 100644 docusaurus/docs/operate/testing/load_testing_testnet.md diff --git a/docusaurus/docs/operate/testing/load_testing_devnet.md b/docusaurus/docs/operate/testing/load_testing_devnet.md index 1ea619855..2ee9e8bb3 100644 --- a/docusaurus/docs/operate/testing/load_testing_devnet.md +++ b/docusaurus/docs/operate/testing/load_testing_devnet.md @@ -10,15 +10,14 @@ A guide on how to perform load testing on DevNets. - [Load Testing on DevNets](#load-testing-on-devnets) - [Overview](#overview) - [Prerequisites](#prerequisites) - - [1. Create and configure the TestNet](#1-create-and-configure-the-testnet) + - [1. Create and configure the DevNet](#1-create-and-configure-the-devnet) - [2. Stake the necessary actors](#2-stake-the-necessary-actors) - [3. Configure the load testing manifest](#3-configure-the-load-testing-manifest) - [Full example](#full-example) - ## Overview -We can create DevNets that are suitable for running load-tests. +We can create DevNets that are suitable for running load tests. :::warning DevNets created with GitHub PRs using `devnet-test-e2e` tags are not suitable for load testing, as they only provision a @@ -27,21 +26,20 @@ single instance of each off-chain actor. We can create custom DevNets with multi ## Prerequisites -### 1. Create and configure the TestNet +### 1. Create and configure the DevNet -Please refer to DevNet creation guide [here](../infrastructure/devnet.md#how-to-create) +Please refer to the DevNet creation guide [here](../infrastructure/devnet.md#how-to-create). ### 2. Stake the necessary actors -- Depending on your LoadTesting requirement, you may need to stake one or more `gateways` and `suppliers`; -- [DevNet documentation](../infrastructure/devnet.md#stake-actors) provides more details about staking actors in DevNets; -- Note to save the addresses of +- Depending on your load testing requirements, you may need to stake one or more `gateways` and `suppliers`. +- [DevNet documentation](../infrastructure/devnet.md#stake-actors) provides more details about staking actors in DevNets. ### 3. Configure the load testing manifest [Load Testing documentation](./load_testing.md#manifest-modification) provides information on how the load test manifest can be modified to run against DevNets. DevNets are not much different from TestNets, but they do not have randomly -generated addresses. Instead, the addresses are shared with LocalNet for convinience. +generated addresses. Instead, the addresses are shared with LocalNet for convenience. ## Full example @@ -64,7 +62,7 @@ relayminers: 2. Gateways and suppliers can be staked using the following commands: :::info -`supplier1` and `gateway1` are pre-staked as a part of genesis generation process. +`supplier1` and `gateway1` are pre-staked as part of the genesis generation process. ::: ```bash @@ -74,7 +72,7 @@ POCKET_NODE=https://devnet-sophon-validator-rpc.poktroll.com make supplier2_stak POCKET_NODE=https://devnet-sophon-validator-rpc.poktroll.com make supplier3_stake ``` -3. Update manifest. The content of +3. Update the manifest. The content of [loadtest_manifest_example.yaml](https://github.com/pokt-network/poktroll/blob/main/load-testing/loadtest_manifest_example.yaml) can be modified as follows: @@ -90,7 +88,7 @@ testnet_node: https://devnet-sophon-validator-rpc.poktroll.com # The service ID to request relays from. service_id: "anvil" -# The address of the account that will be used to fund the the application accounts +# The address of the account that will be used to fund the application accounts # so that they can stake on the network. funding_account_address: pokt1eeeksh2tvkh7wzmfrljnhw4wrhs55lcuvmekkw # address for pnf account @@ -104,14 +102,14 @@ gateways: exposed_url: https://devnet-sophon-gateway-3.poktroll.com ``` -4. Run the test +4. Run the test: ```bash make test_load_relays_stress_example ``` -5. Observe the results +5. Observe the results: You can see the performance of the requests on [Grafana dashboards](https://grafana.poktroll.com/d/nginx/nginx-ingress-controller). -The DevNets have LoadBalancers which allows to get more metrics about network load and latency. When looking -at the `NGINX Ingress controller` Dashboard, make sure to change the namespace to match the DevNet name. +The DevNets have LoadBalancers which allow for more metrics about network load and latency. When looking +at the `NGINX Ingress controller` Dashboard, make sure to change the namespace to match the DevNet name. \ No newline at end of file diff --git a/docusaurus/docs/operate/testing/load_testing_testnet.md b/docusaurus/docs/operate/testing/load_testing_testnet.md new file mode 100644 index 000000000..d521f4b83 --- /dev/null +++ b/docusaurus/docs/operate/testing/load_testing_testnet.md @@ -0,0 +1,36 @@ +--- +sidebar_position: 2 +title: Load Testing - TestNet +--- + +# Load Testing on TestNets + +A guide on how to perform load testing on TestNets. + +- [Load Testing on TestNets](#load-testing-on-testnets) + - [Overview](#overview) + - [Load Testing Steps](#load-testing-steps) + - [Examples](#examples) + +## Overview + +Load Testing on TestNets is very similar to testing on DevNets. The main difference is that the test might run against software you don't control. Gateways and Suppliers can be hosted by other teams or organizations, so you need to ensure that your tests do not adversely affect their operations. As another side effect, you won't be able to collect metrics and logs on the software you don't run. + +## Load Testing Steps + +Please refer to the generic [load testing documentation](./load_testing.md#non-ephemeral-networks-testnets-mainnet-etc) for non-ephemeral networks (TestNets, MainNet, etc.). The steps are very similar to those for DevNets, which are thoroughly documented on the dedicated [Load Testing on DevNets](./load_testing_devnet.md) page. + +Key points to consider: +1. Identify the TestNet endpoints and actors (gateways, suppliers) you'll be testing against. +2. Ensure you have the necessary permissions and have communicated your intent to run load tests. +3. Configure your load testing manifest to target the TestNet resources. +4. Start with lower load and gradually increase to avoid overwhelming the network. +5. Monitor the TestNet's performance metrics during the test. + +## Examples + +:::info +TODO_DOCUMENT(@okdas): add a few examples on how to perform load testing on TestNets: +- How to run the test against gateways and suppliers that are not under your control. +- How to run the test against gateways and suppliers deployed in a separate service for isolation. +::: \ No newline at end of file From 9537e37f0a46d0c75e188791e2db35d98daf8564 Mon Sep 17 00:00:00 2001 From: "Dmitry K." Date: Thu, 27 Jun 2024 18:30:53 -0700 Subject: [PATCH 5/9] omit in toc --- .../docs/operate/testing/load_testing.md | 29 +++++++++---------- .../operate/testing/load_testing_devnet.md | 15 +++++----- .../operate/testing/load_testing_testnet.md | 9 +++--- 3 files changed, 25 insertions(+), 28 deletions(-) diff --git a/docusaurus/docs/operate/testing/load_testing.md b/docusaurus/docs/operate/testing/load_testing.md index f7efa696f..986193dd6 100644 --- a/docusaurus/docs/operate/testing/load_testing.md +++ b/docusaurus/docs/operate/testing/load_testing.md @@ -3,24 +3,23 @@ sidebar_position: 1 title: Load Testing --- -# Load Testing +# Load Testing Poktroll load-testing suite. -- [Load Testing](#load-testing) - - [Overview](#overview) - - [Dependencies](#dependencies) - - [Load Test Manifests](#load-test-manifests) - - [Test Features](#test-features) - - [Executing Tests](#executing-tests) - - [LocalNet Environment](#localnet-environment) - - [Interpreting Results](#interpreting-results) - - [Non-Ephemeral Networks (TestNets, MainNet, etc)](#non-ephemeral-networks-testnets-mainnet-etc) - - [Prerequisites](#prerequisites) - - [Manifest Modification](#manifest-modification) - - [Test Execution](#test-execution) - - [Result Analysis](#result-analysis) - - [Developing Custom Tests](#developing-custom-tests) +- [Overview](#overview) +- [Dependencies](#dependencies) +- [Load Test Manifests](#load-test-manifests) +- [Test Features](#test-features) +- [Executing Tests](#executing-tests) + - [LocalNet Environment](#localnet-environment) + - [Interpreting Results](#interpreting-results) + - [Non-Ephemeral Networks (TestNets, MainNet, etc)](#non-ephemeral-networks-testnets-mainnet-etc) + - [Prerequisites](#prerequisites) + - [Manifest Modification](#manifest-modification) + - [Test Execution](#test-execution) + - [Result Analysis](#result-analysis) +- [Developing Custom Tests](#developing-custom-tests) ## Overview diff --git a/docusaurus/docs/operate/testing/load_testing_devnet.md b/docusaurus/docs/operate/testing/load_testing_devnet.md index 2ee9e8bb3..c7fc22662 100644 --- a/docusaurus/docs/operate/testing/load_testing_devnet.md +++ b/docusaurus/docs/operate/testing/load_testing_devnet.md @@ -3,17 +3,16 @@ sidebar_position: 2 title: Load Testing - DevNet --- -# Load Testing on DevNets +# Load Testing on DevNets A guide on how to perform load testing on DevNets. -- [Load Testing on DevNets](#load-testing-on-devnets) - - [Overview](#overview) - - [Prerequisites](#prerequisites) - - [1. Create and configure the DevNet](#1-create-and-configure-the-devnet) - - [2. Stake the necessary actors](#2-stake-the-necessary-actors) - - [3. Configure the load testing manifest](#3-configure-the-load-testing-manifest) - - [Full example](#full-example) +- [Overview](#overview) +- [Prerequisites](#prerequisites) + - [1. Create and configure the DevNet](#1-create-and-configure-the-devnet) + - [2. Stake the necessary actors](#2-stake-the-necessary-actors) + - [3. Configure the load testing manifest](#3-configure-the-load-testing-manifest) +- [Full example](#full-example) ## Overview diff --git a/docusaurus/docs/operate/testing/load_testing_testnet.md b/docusaurus/docs/operate/testing/load_testing_testnet.md index d521f4b83..f40b3faf1 100644 --- a/docusaurus/docs/operate/testing/load_testing_testnet.md +++ b/docusaurus/docs/operate/testing/load_testing_testnet.md @@ -3,14 +3,13 @@ sidebar_position: 2 title: Load Testing - TestNet --- -# Load Testing on TestNets +# Load Testing on TestNets A guide on how to perform load testing on TestNets. -- [Load Testing on TestNets](#load-testing-on-testnets) - - [Overview](#overview) - - [Load Testing Steps](#load-testing-steps) - - [Examples](#examples) +- [Overview](#overview) +- [Load Testing Steps](#load-testing-steps) +- [Examples](#examples) ## Overview From 8bbdcc3db967a22e6bd5571a842445cc87d5f724 Mon Sep 17 00:00:00 2001 From: "Dmitry K." Date: Thu, 27 Jun 2024 18:31:54 -0700 Subject: [PATCH 6/9] rename the devnet --- docusaurus/docs/operate/infrastructure/devnet.md | 2 +- load-testing/loadtest_manifest_example.yaml | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docusaurus/docs/operate/infrastructure/devnet.md b/docusaurus/docs/operate/infrastructure/devnet.md index d45c3a493..2b67ed429 100644 --- a/docusaurus/docs/operate/infrastructure/devnet.md +++ b/docusaurus/docs/operate/infrastructure/devnet.md @@ -154,7 +154,7 @@ IMAGE_TAG=**IMAGE TAG NAME FROM DEVNET CONFIG** NAMESPACE=devnet-**NETWORK NAME* For example: ```bash -IMAGE_TAG=sha-7042be3 NAMESPACE=devnet-dimatests JOB_NAME=e2e-test-7042be3 POCKET_NODE=tcp://devnet-dimatests-validator-poktrolld:26657 bash .github/workflows-helpers/run-e2e-test.sh +IMAGE_TAG=sha-7042be3 NAMESPACE=devnet-sophon JOB_NAME=e2e-test-7042be3 POCKET_NODE=tcp://devnet-sophon-validator-poktrolld:26657 bash .github/workflows-helpers/run-e2e-test.sh ``` ### Stake actors diff --git a/load-testing/loadtest_manifest_example.yaml b/load-testing/loadtest_manifest_example.yaml index 4c90d367a..2e3612388 100644 --- a/load-testing/loadtest_manifest_example.yaml +++ b/load-testing/loadtest_manifest_example.yaml @@ -4,7 +4,7 @@ is_ephemeral_chain: false # testnet_node is the URL of the node that the load test will use to query the # chain and submit transactions. -testnet_node: https://devnet-dimatests-validator-rpc.poktroll.com +testnet_node: https://devnet-sophon-validator-rpc.poktroll.com # The service ID to request relays from. service_id: "anvil" @@ -16,8 +16,8 @@ funding_account_address: pokt1eeeksh2tvkh7wzmfrljnhw4wrhs55lcuvmekkw # address f # In non-ephemeral chains, the gateways are identified by their address. gateways: - address: pokt15vzxjqklzjtlz7lahe8z2dfe9nm5vxwwmscne4 - exposed_url: https://devnet-dimatests-gateway-1.poktroll.com + exposed_url: https://devnet-sophon-gateway-1.poktroll.com - address: pokt15w3fhfyc0lttv7r585e2ncpf6t2kl9uh8rsnyz - exposed_url: https://devnet-dimatests-gateway-2.poktroll.com + exposed_url: https://devnet-sophon-gateway-2.poktroll.com - address: pokt1zhmkkd0rh788mc9prfq0m2h88t9ge0j83gnxya - exposed_url: https://devnet-dimatests-gateway-3.poktroll.com + exposed_url: https://devnet-sophon-gateway-3.poktroll.com From e899039cabc7140749d678040e110da50be773c5 Mon Sep 17 00:00:00 2001 From: Dima K Date: Mon, 8 Jul 2024 11:17:42 -0700 Subject: [PATCH 7/9] Apply suggestions from code review Co-authored-by: Bryan White --- .../docs/operate/infrastructure/devnet.md | 26 +++++++++---------- .../docs/operate/testing/load_testing.md | 18 ++++++------- .../operate/testing/load_testing_devnet.md | 9 ++++--- .../operate/testing/load_testing_testnet.md | 2 +- 4 files changed, 27 insertions(+), 28 deletions(-) diff --git a/docusaurus/docs/operate/infrastructure/devnet.md b/docusaurus/docs/operate/infrastructure/devnet.md index 2b67ed429..3c316a35d 100644 --- a/docusaurus/docs/operate/infrastructure/devnet.md +++ b/docusaurus/docs/operate/infrastructure/devnet.md @@ -68,7 +68,7 @@ Each Helm chart receives a list of configuration files. For example, see the [re :::note Devnets are provisioned with the same mnemonc phrases as LocalNet, so it is possible to reuse the same keys -from the keybase - the user just need to change the `--node=` argument to point to the DevNet RPC endpoint. +from the keybase - the user just needs to change the `--node=` flag to point to the DevNet RPC endpoint when using the `poktrolld` CLI. ::: The DevNet RPC endpoint is exposed on `https://devnet-**NETWORK_NAME**-validator-rpc.poktroll.com`. @@ -94,16 +94,16 @@ We have two types of DevNets: ### How to create Create a new YAML file in [devnets-configs](https://github.com/pokt-network/protocol-infra/tree/main/devnets-configs). -Use [template](https://github.com/pokt-network/protocol-infra/blob/main/devnets-configs/_TEMPLATE_YAML_) as a reference. +Use [this template](https://github.com/pokt-network/protocol-infra/blob/main/devnets-configs/_TEMPLATE_YAML_) as a reference. ### How to delete Remove the devnet config file. -### Configure and update the version of the software +### Configuring and updating DevNets -We create a new image on all commits created in `main` branch and non-main branches if the PR has a `push-image` label. -Images named in the following format: +New container images are automatically built on each merge to the `main` branch and each push to non-main branches for which there exists a PR with the `push-image` label. +Images named according to the following format: ``` ghcr.io/pokt-network/poktrolld:sha-7042be3 @@ -117,16 +117,15 @@ image: tag: sha-7042be3 ``` -As all parameters in this config actually passed to the [downstream helm chart](https://github.com/pokt-network/protocol-infra/tree/main/charts/full-network) via an [ArgoCD Application](https://github.com/pokt-network/protocol-infra/blob/main/clusters/protocol-us-central1/devnets-persistent.yaml), -it is possible to modify some other parameters besides the image tag. Here is a [list of all options](https://github.com/pokt-network/protocol-infra/blob/main/charts/full-network/values.yaml). +As all parameters in this config are passed to the [downstream helm chart](https://github.com/pokt-network/protocol-infra/tree/main/charts/full-network) via an [ArgoCD Application](https://github.com/pokt-network/protocol-infra/blob/main/clusters/protocol-us-central1/devnets-persistent.yaml). +Here is a [list of all options](https://github.com/pokt-network/protocol-infra/blob/main/charts/full-network/values.yaml) which can be modified, in addition to the image tag. -### Scale actors up and down +### Scaling actors up and down You can modify the number of each actor by changing the devnet config file: :::info -We use the same ignite `config.yaml` to provision a genesis as we use on devnets. That means we should not provision more -actors than we configure in ignite's `config.yaml`. Good rule of thumb: don't go over `3`. +We use the same ignite `config.yaml` to provision genesis in devnet as is used in localnet. Because localnet supports a max of 3 of each actor type, any devnet deployment with more actors would also require additional corresponding genesis state to be included in `config.yaml`, or on-demand account funding and staking. General rule of thumb: don't go over `3`. ::: @@ -145,9 +144,8 @@ relayminers: 1. The image tag must match the tag of the image from the devnet config YAML file. 2. The name of the devnet in the environment variables must be specified. 3. The kubernetes context must be changed to the protocol cluster (`kubectl config set-context gke_protocol-us-central1-d505_us-central1_protocol-us-central1`) -4. Command should be executed from the root of the poktroll repo. +4. Run the following command from the root of the poktroll repo: -The command: ```bash IMAGE_TAG=**IMAGE TAG NAME FROM DEVNET CONFIG** NAMESPACE=devnet-**NETWORK NAME** JOB_NAME=e2e-test-**GITSHA FROM IMAGE TAG** POCKET_NODE=tcp://devnet-**NETWORK NAME**-validator-poktrolld:26657 bash .github/workflows-helpers/run-e2e-test.sh ``` @@ -157,9 +155,9 @@ For example: IMAGE_TAG=sha-7042be3 NAMESPACE=devnet-sophon JOB_NAME=e2e-test-7042be3 POCKET_NODE=tcp://devnet-sophon-validator-poktrolld:26657 bash .github/workflows-helpers/run-e2e-test.sh ``` -### Stake actors +### Staking actors -Since the keys are the same as LocalNet, we can use the same commands for DevNet to stake the actors. Also, DevNets +Since the genesis actor keys are the same as LocalNet, we can use the same commands (and keyring) to stake actors on DevNet. Additionally, DevNets match the hostnames with LocalNet, which makes it possible for the same stake configs to work on different networks. For example, this command stakes supplier2 on DevNet `devnet-sophon`: diff --git a/docusaurus/docs/operate/testing/load_testing.md b/docusaurus/docs/operate/testing/load_testing.md index 986193dd6..8f9af7f14 100644 --- a/docusaurus/docs/operate/testing/load_testing.md +++ b/docusaurus/docs/operate/testing/load_testing.md @@ -32,13 +32,13 @@ The load-testing suite is built on [Gherkin](https://cucumber.io/docs/gherkin/), ## Load Test Manifests -Load test manifests are YAML files that define the network environment for test execution. These files specify properties such as blockchain addresses for funding and staking applications, and the suppliers and gateways to utilize. The [LocalNet manifest](https://github.com/pokt-network/poktroll/blob/main/load-testing/loadtest_manifest_localnet.yaml) serves as a comprehensive example, including detailed comments for each manifest property. +Load test manifests are YAML files that define the network environment for test execution. These files specify properties such as blockchain account addresses for funding and staking applications, and for utilization by suppliers and gateways. The [LocalNet manifest](https://github.com/pokt-network/poktroll/blob/main/load-testing/loadtest_manifest_localnet.yaml) serves as a comprehensive example, including detailed comments for each manifest property. ## Test Features -Test features are located in the [load-testing/tests](https://github.com/pokt-network/poktroll/tree/main/load-testing/tests) directory, encompassing various use cases. +Test features are located in the [load-testing/tests](https://github.com/pokt-network/poktroll/tree/main/load-testing/tests) directory, encompassing various scenarios. -Leveraging Gherkin, the feature files contain human-readable load tests. For instance: +Feature files are composed of one or more [`Scenario`](https://cucumber.io/docs/gherkin/reference/?sbsearch=Scenarios)s (or [`Scenario Outline`](https://cucumber.io/docs/gherkin/reference/?sbsearch=Scenarios#scenario-outline)s), each of which is composed of one or more [steps](https://cucumber.io/docs/gherkin/reference/#steps). For example: ```gherkin Feature: Loading anvil node only @@ -55,7 +55,7 @@ Feature: Loading anvil node only | 10000 | 10 | ``` -This natural language syntax is parsed by [gocuke](https://github.com/regen-network/gocuke). +This natural language syntax is parsed and used to match and execute the corresponding [step definitions](https://cucumber.io/docs/cucumber/step-definitions/?lang=javascript) using [gocuke](https://github.com/regen-network/gocuke). ## Executing Tests @@ -65,7 +65,7 @@ To execute tests on LocalNet: 1. Ensure [LocalNet](../infrastructure/localnet.md) is operational. 2. In the `localnet_config.yaml` file, set `gateways.count` and `relayminers.count` to `3`. -3. Execute `make acc_initialize_pubkeys` to initialize blockchain state public keys. +3. Run `make acc_initialize_pubkeys` to initialize blockchain state public keys. 4. Run `make test_load_relays_stress_localnet` to run the LocalNet stress-test. #### Interpreting Results @@ -83,9 +83,9 @@ Note: Such networks typically involve other participants, allowing load testing #### Prerequisites -- An address with sufficient tokens for application funding and staking, accessible on the local keychain (e.g., `poktrolld keys list`) -- A list of target gateways for request issuance -- For custom service testing, ensure suppliers are configured and ready to process requests +- A account with sufficient tokens for application funding and staking, accessible on the local keychain (e.g., `poktrolld keys list`). +- A list of target gateways to which relay requests will be sent throughout the course of the test. +- For custom service testing, ensure supplier(s') corresponding relayminer(s) and custom service process are properly configured, running, and ready to process requests. #### Manifest Modification @@ -105,4 +105,4 @@ For tests conducted against community-hosted suppliers and gateways, analysis is ## Developing Custom Tests -For custom test development, refer to the [gocuke documentation](https://github.com/regen-network/gocuke?tab=readme-ov-file#quick-start). The [anvil test](https://github.com/pokt-network/poktroll/blob/main/load-testing/tests/anvil_test.go) provides a practical reference implementation. \ No newline at end of file +For custom test development, refer to the [gocuke documentation](https://github.com/regen-network/gocuke?tab=readme-ov-file#quick-start). The [anvil test](https://github.com/pokt-network/poktroll/blob/main/load-testing/tests/anvil_test.go) provides a small but practical reference implementation. \ No newline at end of file diff --git a/docusaurus/docs/operate/testing/load_testing_devnet.md b/docusaurus/docs/operate/testing/load_testing_devnet.md index c7fc22662..faf2429a0 100644 --- a/docusaurus/docs/operate/testing/load_testing_devnet.md +++ b/docusaurus/docs/operate/testing/load_testing_devnet.md @@ -34,11 +34,11 @@ Please refer to the DevNet creation guide [here](../infrastructure/devnet.md#how - Depending on your load testing requirements, you may need to stake one or more `gateways` and `suppliers`. - [DevNet documentation](../infrastructure/devnet.md#stake-actors) provides more details about staking actors in DevNets. -### 3. Configure the load testing manifest +### 3. Configure the load test manifest [Load Testing documentation](./load_testing.md#manifest-modification) provides information on how the load test manifest -can be modified to run against DevNets. DevNets are not much different from TestNets, but they do not have randomly -generated addresses. Instead, the addresses are shared with LocalNet for convenience. +can be modified to run against DevNets. DevNets are not much different from TestNets, but they **do not** have randomly +generated accounts. Instead, the accounts are derived from the ignite `config.yaml` (as in LocalNet) for convenience. ## Full example @@ -89,9 +89,10 @@ service_id: "anvil" # The address of the account that will be used to fund the application accounts # so that they can stake on the network. +# TODO_TECHDEBT(@bryanchriswhite, #512): Replace with faucet address. funding_account_address: pokt1eeeksh2tvkh7wzmfrljnhw4wrhs55lcuvmekkw # address for pnf account -# In non-ephemeral chains, the gateways are identified by their address. +# In non-ephemeral chains, the gateways are identified by their addresses. gateways: - address: pokt15vzxjqklzjtlz7lahe8z2dfe9nm5vxwwmscne4 exposed_url: https://devnet-sophon-gateway-1.poktroll.com diff --git a/docusaurus/docs/operate/testing/load_testing_testnet.md b/docusaurus/docs/operate/testing/load_testing_testnet.md index f40b3faf1..60878eb41 100644 --- a/docusaurus/docs/operate/testing/load_testing_testnet.md +++ b/docusaurus/docs/operate/testing/load_testing_testnet.md @@ -22,7 +22,7 @@ Please refer to the generic [load testing documentation](./load_testing.md#non-e Key points to consider: 1. Identify the TestNet endpoints and actors (gateways, suppliers) you'll be testing against. 2. Ensure you have the necessary permissions and have communicated your intent to run load tests. -3. Configure your load testing manifest to target the TestNet resources. +3. Configure your load test manifest to target the TestNet resources. 4. Start with lower load and gradually increase to avoid overwhelming the network. 5. Monitor the TestNet's performance metrics during the test. From fb520a57563e2f45441c15c1092e5cc59c3c2c4f Mon Sep 17 00:00:00 2001 From: "Dmitry K." Date: Mon, 8 Jul 2024 17:51:32 -0700 Subject: [PATCH 8/9] requested changes --- .../docs/operate/infrastructure/devnet.md | 43 +++++++++++++------ 1 file changed, 31 insertions(+), 12 deletions(-) diff --git a/docusaurus/docs/operate/infrastructure/devnet.md b/docusaurus/docs/operate/infrastructure/devnet.md index 3c316a35d..e346cf112 100644 --- a/docusaurus/docs/operate/infrastructure/devnet.md +++ b/docusaurus/docs/operate/infrastructure/devnet.md @@ -25,10 +25,10 @@ to set up your access to GCP. - [Manually provisioned DevNets](#manually-provisioned-devnets) - [How to create](#how-to-create) - [How to delete](#how-to-delete) - - [Configure and update the version of the software](#configure-and-update-the-version-of-the-software) - - [Scale actors up and down](#scale-actors-up-and-down) + - [Configuring and updating DevNets](#configuring-and-updating-devnets) + - [Scaling actors up and down](#scaling-actors-up-and-down) - [Run e2e tests on a manually provisioned DevNet](#run-e2e-tests-on-a-manually-provisioned-devnet) - - [Stake actors](#stake-actors) + - [Staking actors](#staking-actors) ## GCP Console @@ -93,12 +93,17 @@ We have two types of DevNets: ### How to create -Create a new YAML file in [devnets-configs](https://github.com/pokt-network/protocol-infra/tree/main/devnets-configs). -Use [this template](https://github.com/pokt-network/protocol-infra/blob/main/devnets-configs/_TEMPLATE_YAML_) as a reference. +Commit a new YAML file in the [devnets-configs](https://github.com/pokt-network/protocol-infra/tree/main/devnets-configs) +directory on the `main` branch. Use [this template](https://github.com/pokt-network/protocol-infra/blob/main/devnets-configs/_TEMPLATE_YAML_) +as a reference. If you are not a member of the protocol team, open a pull request (PR) to add the new YAML file to the +`main` branch. Protocol team members can bypass the `main` branch protection to commit directly without going through +the PR flow. ### How to delete -Remove the devnet config file. +Remove the devnet config file by committing the deletion to the `main` branch. If you are not a member of the protocol +team, open a pull request (PR) to delete the YAML file from the `main` branch. Protocol team members can bypass the +`main` branch protection to commit the deletion directly without going through the PR flow. ### Configuring and updating DevNets @@ -138,23 +143,37 @@ relayminers: count: 1 ``` - ### Run e2e tests on a manually provisioned DevNet -1. The image tag must match the tag of the image from the devnet config YAML file. -2. The name of the devnet in the environment variables must be specified. -3. The kubernetes context must be changed to the protocol cluster (`kubectl config set-context gke_protocol-us-central1-d505_us-central1_protocol-us-central1`) -4. Run the following command from the root of the poktroll repo: +To run e2e tests on a manually provisioned DevNet, use the following command from the root of the poktroll repo: ```bash IMAGE_TAG=**IMAGE TAG NAME FROM DEVNET CONFIG** NAMESPACE=devnet-**NETWORK NAME** JOB_NAME=e2e-test-**GITSHA FROM IMAGE TAG** POCKET_NODE=tcp://devnet-**NETWORK NAME**-validator-poktrolld:26657 bash .github/workflows-helpers/run-e2e-test.sh ``` -For example: +**Environment Variables**: + +- **IMAGE_TAG**: The tag of the image from the devnet config YAML file. This tag must match the tag of the image used in the DevNet configuration. +- **NAMESPACE**: The name of the devnet. This should be specified in the environment variables and follow the format devnet-**NETWORK NAME**. +- **JOB_NAME**: A unique identifier for the e2e test job. It follows the format e2e-test-**GITSHA FROM IMAGE TAG**. +- **POCKET_NODE**: The address of the pocket node, following the format tcp://devnet-**NETWORK NAME**-validator-poktrolld:26657. + + +**Example** + ```bash IMAGE_TAG=sha-7042be3 NAMESPACE=devnet-sophon JOB_NAME=e2e-test-7042be3 POCKET_NODE=tcp://devnet-sophon-validator-poktrolld:26657 bash .github/workflows-helpers/run-e2e-test.sh ``` +:::info +- The `IMAGE_TAG` must match the tag of the image from the devnet config YAML file. +- The name of the devnet in the environment variables must be specified. +- The Kubernetes context must be pointed to the protocol cluster. This command can be used to change context: + ```bash + kubectl config set-context gke_protocol-us-central1-d505_us-central1_protocol-us-central1 + ``` +::: + ### Staking actors Since the genesis actor keys are the same as LocalNet, we can use the same commands (and keyring) to stake actors on DevNet. Additionally, DevNets From 91dd241b798e260d5e3b4fa3401bd10517f88f40 Mon Sep 17 00:00:00 2001 From: "Dmitry K." Date: Tue, 9 Jul 2024 17:12:47 -0700 Subject: [PATCH 9/9] requested changes --- Makefile | 7 ++++--- .../docs/operate/infrastructure/devnet.md | 6 +++--- docusaurus/docs/operate/testing/load_testing.md | 17 ++++++++++++++--- 3 files changed, 21 insertions(+), 9 deletions(-) diff --git a/Makefile b/Makefile index 9e2295d54..e040fd122 100644 --- a/Makefile +++ b/Makefile @@ -16,6 +16,7 @@ TESTNET_RPC ?= https://testnet-validated-validator-rpc.poktroll.com/ # TestNet R APPGATE_SERVER ?= http://localhost:42069 GATEWAY_URL ?= http://localhost:42079 POCKET_ADDR_PREFIX = pokt +LOAD_TEST_CUSTOM_MANIFEST ?= loadtest_manifest_example.yaml # The domain ending in ".town" is staging, ".city" is production GROVE_GATEWAY_STAGING_ETH_MAINNET = https://eth-mainnet.rpc.grove.town @@ -403,11 +404,11 @@ test_e2e_settlement: test_e2e_env ## Run only the E2E suite that exercises the s test_e2e_params: test_e2e_env ## Run only the E2E suite that exercises parameter updates for all modules go test -v ./e2e/tests/... -tags=e2e,test --features-path=update_params.feature -.PHONY: test_load_relays_stress_example -test_load_relays_stress_example: ## Run the stress test for E2E relays on a persistent (non-ephemeral) remote (devnet) chain. Note that this is just an example. +.PHONY: test_load_relays_stress_custom +test_load_relays_stress_custom: ## Run the stress test for E2E relays using custom manifest. "loadtest_manifest_example.yaml" manifest is used by default. Set `LOAD_TEST_CUSTOM_MANIFEST` environment variable to use the different manifest. go test -v -count=1 ./load-testing/tests/... \ -tags=load,test -run LoadRelays --log-level=debug --timeout=30m \ - --manifest ./load-testing/loadtest_manifest_example.yaml + --manifest ./load-testing/$(LOAD_TEST_CUSTOM_MANIFEST) .PHONY: test_load_relays_stress_localnet test_load_relays_stress_localnet: test_e2e_env warn_message_local_stress_test ## Run the stress test for E2E relays on LocalNet. diff --git a/docusaurus/docs/operate/infrastructure/devnet.md b/docusaurus/docs/operate/infrastructure/devnet.md index e346cf112..05ce3b679 100644 --- a/docusaurus/docs/operate/infrastructure/devnet.md +++ b/docusaurus/docs/operate/infrastructure/devnet.md @@ -122,7 +122,7 @@ image: tag: sha-7042be3 ``` -As all parameters in this config are passed to the [downstream helm chart](https://github.com/pokt-network/protocol-infra/tree/main/charts/full-network) via an [ArgoCD Application](https://github.com/pokt-network/protocol-infra/blob/main/clusters/protocol-us-central1/devnets-persistent.yaml). +All parameters in this config are passed to the [downstream helm chart](https://github.com/pokt-network/protocol-infra/tree/main/charts/full-network) via an [ArgoCD Application](https://github.com/pokt-network/protocol-infra/blob/main/clusters/protocol-us-central1/devnets-persistent.yaml). Here is a [list of all options](https://github.com/pokt-network/protocol-infra/blob/main/charts/full-network/values.yaml) which can be modified, in addition to the image tag. ### Scaling actors up and down @@ -176,8 +176,8 @@ IMAGE_TAG=sha-7042be3 NAMESPACE=devnet-sophon JOB_NAME=e2e-test-7042be3 POCKET_N ### Staking actors -Since the genesis actor keys are the same as LocalNet, we can use the same commands (and keyring) to stake actors on DevNet. Additionally, DevNets -match the hostnames with LocalNet, which makes it possible for the same stake configs to work on different networks. +Since the genesis actor keys are the same as LocalNet, we can use the same commands (and keyring) to stake actors on +DevNet. Additionally, DevNets match the hostnames with LocalNet, making it possible for the same stake configs to work on different networks. For example, this command stakes supplier2 on DevNet `devnet-sophon`: ```bash diff --git a/docusaurus/docs/operate/testing/load_testing.md b/docusaurus/docs/operate/testing/load_testing.md index 8f9af7f14..f1b802c8c 100644 --- a/docusaurus/docs/operate/testing/load_testing.md +++ b/docusaurus/docs/operate/testing/load_testing.md @@ -71,7 +71,10 @@ To execute tests on LocalNet: #### Interpreting Results - The CLI output displays standard Go test results. Successful tests are indicated by `PASS`, while failures are denoted by `FAIL` with accompanying error messages. -- During test execution, the observability stack continuously collects metric data from off-chain actors. On LocalNet, [Grafana is accessible on port 3003](http://localhost:3003/?orgId=1). The "Stress test" and "Load Testing" dashboards provide valuable insights into system status. +- During test execution, the observability stack continuously collects metric data from off-chain actors. On LocalNet, [Grafana is accessible on port 3003](http://localhost:3003/?orgId=1). The + [Stress test](http://localhost:3003/d/ddkakqetrti4gb/protocol-stress-test?orgId=1&refresh=5s) + and [Load Testing](http://localhost:3003/d/fdjwb9u9t9ts0e/protocol-load-testing?orgId=1) dashboards provide valuable + insights into system status. ### Non-Ephemeral Networks (TestNets, MainNet, etc) @@ -93,10 +96,18 @@ Using [loadtest_manifest_example.yaml](https://github.com/pokt-network/poktroll/ #### Test Execution -Utilize the provided makefile target to run the `relays_stress.feature` with the modified manifest: +Utilize the provided makefile target to run the `relays_stress.feature` with the modified manifest. By default, the example manifest file is used. You can specify a different manifest file by setting the `LOAD_TEST_CUSTOM_MANIFEST` environment variable. + +To run the stress test using the default manifest: + +```bash +make test_load_relays_stress_custom +``` + +To run the stress test with a custom manifest: ```bash -make test_load_relays_stress_example +LOAD_TEST_CUSTOM_MANIFEST=your_new_manifest.yaml make test_load_relays_stress_custom ``` #### Result Analysis