[Documentation] Load Testing on DevNets and TestNets

.gitignore

@@ -95,3 +95,6 @@ gentx-*.json
 
 # Binaries VSCode builds for debugging
 cmd/poktrolld/*debug_bin*
+
+# The temporary file created by e2e scripts
+job.yaml

Makefile

@@ -16,6 +16,7 @@ TESTNET_RPC ?= https://testnet-validated-validator-rpc.poktroll.com/ # TestNet R
 APPGATE_SERVER ?= http://localhost:42069
 GATEWAY_URL ?= http://localhost:42079
 POCKET_ADDR_PREFIX = pokt
+LOAD_TEST_CUSTOM_MANIFEST ?= loadtest_manifest_example.yaml
 
 # The domain ending in ".town" is staging, ".city" is production
 GROVE_GATEWAY_STAGING_ETH_MAINNET = https://eth-mainnet.rpc.grove.town
@@ -403,11 +404,11 @@ test_e2e_settlement: test_e2e_env ## Run only the E2E suite that exercises the s
 test_e2e_params: test_e2e_env ## Run only the E2E suite that exercises parameter updates for all modules
 	go test -v ./e2e/tests/... -tags=e2e,test --features-path=update_params.feature
 
-.PHONY: test_load_relays_stress_example
-test_load_relays_stress_example: ## Run the stress test for E2E relays on a persistent (non-ephemeral) remote (devnet) chain. Note that this is just an example.
+.PHONY: test_load_relays_stress_custom
+test_load_relays_stress_custom: ## Run the stress test for E2E relays using custom manifest. "loadtest_manifest_example.yaml" manifest is used by default. Set `LOAD_TEST_CUSTOM_MANIFEST` environment variable to use the different manifest.
 	go test -v -count=1 ./load-testing/tests/... \
 	-tags=load,test -run LoadRelays --log-level=debug --timeout=30m \
-	--manifest ./load-testing/loadtest_manifest_example.yaml
+	--manifest ./load-testing/$(LOAD_TEST_CUSTOM_MANIFEST)
 
 .PHONY: test_load_relays_stress_localnet
 test_load_relays_stress_localnet: test_e2e_env warn_message_local_stress_test ## Run the stress test for E2E relays on LocalNet.

docusaurus/docs/operate/infrastructure/devnet.md

@@ -1,9 +1,9 @@
 ---
 sidebar_position: 3
-title: DevNet
+title: DevNets
 ---
 
-# DevNet <!-- omit in toc -->
+# DevNets <!-- omit in toc -->
 
 :::note
 
@@ -20,6 +20,15 @@ to set up your access to GCP.
 - [Grafana logs](#grafana-logs)
 - [Infrastructure Provisioning](#infrastructure-provisioning)
 - [Configuration](#configuration)
+- [Interaction with DevNet](#interaction-with-devnet)
+- [DevNet types](#devnet-types)
+- [Manually provisioned DevNets](#manually-provisioned-devnets)
+  - [How to create](#how-to-create)
+  - [How to delete](#how-to-delete)
+  - [Configuring and updating DevNets](#configuring-and-updating-devnets)
+  - [Scaling actors up and down](#scaling-actors-up-and-down)
+  - [Run e2e tests on a manually provisioned DevNet](#run-e2e-tests-on-a-manually-provisioned-devnet)
+  - [Staking actors](#staking-actors)
 
 ## GCP Console
 
@@ -54,3 +63,127 @@ Each DevNet ArgoCD App (following the App of Apps pattern) provisions a Helm cha
 Each `full-network` includes other ArgoCD applications that deploy Validators and off-chain actors.
 
 Each Helm chart receives a list of configuration files. For example, see the [relayminer configuration](https://github.com/pokt-network/protocol-infra/blob/main/charts/full-network/templates/Application-Relayminer.yaml#L37). All possible values can be found in the `values.yaml` of the Helm chart, such as the [relayminer Helm chart](https://github.com/pokt-network/helm-charts/blob/main/charts/relayminer/values.yaml).
+
+## Interaction with DevNet
+
+:::note
+Devnets are provisioned with the same mnemonc phrases as LocalNet, so it is possible to reuse the same keys
+from the keybase - the user just needs to change the `--node=` flag to point to the DevNet RPC endpoint when using the `poktrolld` CLI.
+:::
+
+The DevNet RPC endpoint is exposed on `https://devnet-**NETWORK_NAME**-validator-rpc.poktroll.com`.
+
+It is possible to reuse the LocalNet make targets with DevNet endpoints. For example
+
+```bash
+POCKET_NODE=https://devnet-issue-420-validator-rpc.poktroll.com make supplier_list
+```
+
+## DevNet types
+
+We have two types of DevNets:
+1. Automatically provisioned by `devnet-test-e2e` GitHub label.
+  - Their main purpose is to automatically run e2e tests for each PR this label has been assigned to.
+  - They are not easily modified and configured. E.g. we automatically provision single instance of each actor only.
+2. Manually provisioned by creating a file in [protocol-infra repo under `devnet-configs` directory](https://github.com/pokt-network/protocol-infra/tree/main/devnets-configs).
+  - Their main purpose is for testing new features or configurations that require more complex setups than the first type can provide.
+  - They are easily modified and configured, allowing you to add multiple instances of each actor.
+
+## Manually provisioned DevNets
+
+### How to create
+
+Commit a new YAML file in the [devnets-configs](https://github.com/pokt-network/protocol-infra/tree/main/devnets-configs)
+directory on the `main` branch. Use [this template](https://github.com/pokt-network/protocol-infra/blob/main/devnets-configs/_TEMPLATE_YAML_)
+as a reference. If you are not a member of the protocol team, open a pull request (PR) to add the new YAML file to the
+`main` branch. Protocol team members can bypass the `main` branch protection to commit directly without going through
+the PR flow.
+
+### How to delete
+
+Remove the devnet config file by committing the deletion to the `main` branch. If you are not a member of the protocol
+team, open a pull request (PR) to delete the YAML file from the `main` branch. Protocol team members can bypass the
+`main` branch protection to commit the deletion directly without going through the PR flow.
+
+### Configuring and updating DevNets
+
+New container images are automatically built on each merge to the `main` branch and each push to non-main branches for which there exists a PR with the `push-image` label.
+Images named according to the following format:
+
+```
+ghcr.io/pokt-network/poktrolld:sha-7042be3
+ghcr.io/pokt-network/poktrolld:sha-7042be3922245fb4313ee90e1f28d0f402b700a0
+```
+
+You can update the version of DevNet by changing this parameter in the devnet config file:
+
+```yaml
+image:
+  tag: sha-7042be3
+```
+
+All parameters in this config are passed to the [downstream helm chart](https://github.com/pokt-network/protocol-infra/tree/main/charts/full-network) via an [ArgoCD Application](https://github.com/pokt-network/protocol-infra/blob/main/clusters/protocol-us-central1/devnets-persistent.yaml).
+Here is a [list of all options](https://github.com/pokt-network/protocol-infra/blob/main/charts/full-network/values.yaml) which can be modified, in addition to the image tag.
+
+### Scaling actors up and down
+
+You can modify the number of each actor by changing the devnet config file:
+
+:::info
+We use the same ignite `config.yaml` to provision genesis in devnet as is used in localnet. Because localnet supports a max of 3 of each actor type, any devnet deployment with more actors would also require additional corresponding genesis state to be included in `config.yaml`, or on-demand account funding and staking. General rule of thumb: don't go over `3`.
+:::
+
+
+```yaml
+appgateservers:
+  count: 1
+gateways:
+  count: 1
+relayminers:
+  count: 1
+```
+
+### Run e2e tests on a manually provisioned DevNet
+
+To run e2e tests on a manually provisioned DevNet, use the following command from the root of the poktroll repo:
+
+```bash
+IMAGE_TAG=**IMAGE TAG NAME FROM DEVNET CONFIG** NAMESPACE=devnet-**NETWORK NAME** JOB_NAME=e2e-test-**GITSHA FROM IMAGE TAG** POCKET_NODE=tcp://devnet-**NETWORK NAME**-validator-poktrolld:26657 bash .github/workflows-helpers/run-e2e-test.sh
+```
+
+**Environment Variables**:
+
+- **IMAGE_TAG**: The tag of the image from the devnet config YAML file. This tag must match the tag of the image used in the DevNet configuration.
+- **NAMESPACE**: The name of the devnet. This should be specified in the environment variables and follow the format devnet-**NETWORK NAME**.
+- **JOB_NAME**: A unique identifier for the e2e test job. It follows the format e2e-test-**GITSHA FROM IMAGE TAG**.
+- **POCKET_NODE**: The address of the pocket node, following the format tcp://devnet-**NETWORK NAME**-validator-poktrolld:26657.
+
+
+**Example**
+
+```bash
+IMAGE_TAG=sha-7042be3 NAMESPACE=devnet-sophon JOB_NAME=e2e-test-7042be3 POCKET_NODE=tcp://devnet-sophon-validator-poktrolld:26657 bash .github/workflows-helpers/run-e2e-test.sh
+```
+
+:::info
+- The `IMAGE_TAG` must match the tag of the image from the devnet config YAML file.
+- The name of the devnet in the environment variables must be specified.
+- The Kubernetes context must be pointed to the protocol cluster. This command can be used to change context:
+  ```bash
+  kubectl config set-context gke_protocol-us-central1-d505_us-central1_protocol-us-central1
+  ```
+:::
+
+### Staking actors
+
+Since the genesis actor keys are the same as LocalNet, we can use the same commands (and keyring) to stake actors on
+DevNet. Additionally, DevNets match the hostnames with LocalNet, making it possible for the same stake configs to work on different networks.
+For example, this command stakes supplier2 on DevNet `devnet-sophon`:
+
+```bash
+POCKET_NODE=https://devnet-sophon-validator-rpc.poktroll.com make supplier2_stake
+```
+
+:::note
+Only manually provisioned DevNets have more than one actor to stake. 
+:::