Orion integration tests proposal

[Lezek123]

The problem

We can test the Query Node mappings and GraphQL queries using integration tests in the the Joystream monorepo.
The usual flow of those tests is:

1. We build and run joystream node (docker image)
2. We start the query node, which processes the local chain
3. We send some extrinsics to joystream node started in step 1.
4. The query node processes those extrinsics
5. We issue GraphQL queries to verify that the query node has correctly processed the blockchain events, updated the database and provides valid state in the response

It is simple to run such integration tests in the monorepo on each pull request, because the monorepo includes:

• the source code of the Joystream node and Joystream runtime,
• the source code of the Query Node,
• docker setup for running all the required services,
• source code of the integration tests.

However, with Orion living in a separate repository, the execution of similar integration tests for Orion would be a little bit more involved.

Possible solutions

There are approaches to this problem:

1. The approach taken by @ignazio-bovo in which we try to replicate (at least partially) the integration tests setup from the monorepo in Orion repository. The benefit of this approach is that running the integration tests as part of CI checks in Orion repo becomes very simple and we can run the checks on each pull request the same way we do in the monorepo. The drawback is that we then need to maintain the integration tests setup/framework together with all the reusable fixtures etc. across 2 different repositories, which would be quite difficult to manage.

2. An alternative approach, described in this proposal, is to keep all integration tests in one place (ie. the Joystream monorepo). In this case we can add Orion-specific checks to individual fixtures, just like we do with query node checks currently (via Fixture.runQueryNodeChecks). Both the query node checks and orion checks can be conditionally skipped via respective SKIP_QUERY_NODE_CHECKS and SKIP_ORION_CHECKS env variables. In Orion repository we can then add a GitHub workflow triggered by pull_request event that will:

   • Build Orion docker image
   • Fetch a target Joystream node docker image (artifact) from the monorepo
   • Checkout a commit/branch in the monorepo (with the right integration tests implemented)
   • Run the integration tests using the Orion and Joystream docker images. The Query node checks can be skipped using SKIP_QUERY_NODE_CHECKS=true env.

   The drawback of this solution is that it's a little bit complex to setup the Github workflow this way (as you will notice reading the Proof of Concept below) and the fact that the integration tests run will need to be approved manually by a reviewer with sufficient permissions on the "Orion" repo (for security reasons, as will be further described below). The main benefit is that we can avoid the dual-maintenance issue when it comes to integration tests setup/framework and reusable fixtures (they will all live in the monorepo and will only need to be maintained there).

Proposal

In this proposal I'll describe how we can adjust the integration tests framework in the monorepo and sync the GitHub workflows between Joystream monorepo and Orion monorepo to be able to trigger the Orion integration tests (implemented in monorepo) as part of PR/push CI checks in Orion repository.

The Proof of Concept implementation of this idea can be found in the following draft PRs:
Orion side: #145
Monorepo (integration tests) side: Joystream/joystream#4790

Adjusting integration tests framework

Here some work has been already done by @ignazio-bovo as part of implementing a solution described in point 1. under Possible solutions, related branch: #143, for example, the OrionApi service was introduced (similar to pre-existing QueryNodeApi) which provides a great interface for executing queries against a running Orion instance.

On my PoC branch I have shown how I imagine separating the Orion checks from other checks (like Query node checks etc., which are already separated from the main logic of each Fixture), so that it's easy to coditionally enable / disable them depending on whether we care about testing Orion while running a given test scenario.

Basically, each fixture can optionally implement runOrionChecks method, similar to the exising runQueryNodeChecks.
The runOrionChecks method of a fixture will only be called if:

• SKIP_ORION_CHECKS=true env is not set
• FixtureRunner(fixture).runWithQueryNodeChecks() is called (which will run both Query Node and/or Orion checks, depending on env settings). The default FixtureRunner(fixture).run() will skip both the Query Node and Orion checks and should be used only if we're running the fixture are part of some integration tests flow just for the purpose of making some initial setup (for example, creating memberships that will be required later in the flow), but the flow itself is not intended to test the behavior of that specific fixture.

Github workflow

The PoC GitHub workflow that can be added to Orion repository can be found here: https://github.com/Lezek123/orion/blob/orion-integration-tests/.github/workflows/trigger-tests.yml

It can be seen in action in my example PR in Lezek123/orion fork: Lezek123#5

It runs on pull-request-target and push (to master) events and consists of a single job with the following steps:

1. First it waits for build-docker-image.yml workflow in the Orion repo to finish building the Orion docker image and uploading it as artifact
2. It then fetches and loads the Orion docker image (joystream/orion:latest) uploaded by build-docker-image.yml workflow
3. If it was triggered by pull-request-target event, it looks for /set-integration-tests-branch command in the PR comments and updates the INTEGRATION_TESTS_BRANCH env accordingly if a comment with such command is found. The idea behind this is descibed in the /set-integration-tests-branch command section.
4. It checks out a branch in the Joystream monorepo (vars.INTEGRATION_TESTS_REPO) based on the value of INTEGRATION_TESTS_BRANCH env.
5. It downloads a Joystream node docker image (joystream/node:{runtime_hash}) built on the checked out branch by the run-network-tests.yml workflow in the monorepo (as it's exposed as an artifact) and loads it.
6. It then runs the integration tests scenario using the loaded Orion and Joystream node docker images, just the way it can be run from within the run-network-tests.yml workflow in the monorepo. The scenario name is currently hardcoded in the workflow and is called orion. The SKIP_QUERY_NODE_CHECKS env is also set to true, so that only the Orion checks are ran and any QN failures will not affect the test results.

Required GitHub variables, secrets and environments

This section describes the required setup in https://github.com/Joystream/orion and https://github.com/Joystream/joystream repos in order to enable the workflow described in this proposal.

GitHub PAT (monorepo)

Repository: Monorepo

A fine-grained GitHub PAT needs to be created, allowing access to read artifacts from the monorepo:

GitHub environments

Repository: Orion

One GitHub environment needs to be created for each Orion repository branch on which the integration tests are supposed to run:

The environment has to be configured with:

• Required reviewers - ie. people who can trigger (deploy) the integration tests check
• INTEGRATION_TESTS_TOKEN secret - a fine-grained GitHub PAT with permissions to read artifacts from the Joystream monorepo (created accordingly to instructions provided in the section above)
• INTEGRATION_TESTS_BRANCH variable - specifies the default branch in the Joystream monorepo that will be used to execute the integration tests in this GitHub environment (ie. on this Orion branch)

Repository variables

Repository: Orion

In the "global context" of Orion repository, the INTEGRATION_TESTS_REPO variable should be set to point to the Joystream monorepo (Joystream/joystream)

/set-integration-tests-branch command

Oftentimes it will probably be the case that the PR to introduce changes in Orion will co-exist with a monorepo PR to introduce the related integration tests. In this case we may want to run integration tests from the not-yet-mergered monorepo PR commit as part of the Orion PR CI checks.

To facilitate this use-case, I introduced /set-integration-tests-branch command functionality (as part of the PoC) to allow dynamically specifying the branch/commit in the monorepo that will be used when running the integration tests in a given Orion PR.

The command is intended to be provided in a GitHub comment to a PR on which we want to change the target monorepo integration tests branch. The comment will only affect the workflow if it is posted by someone who is a MEMBER of the Joystream orgnaization on GitHub (see: https://github.com/orgs/Joystream/people).

Example usage

To ilustrate the usage of /set-integration-tests-branch command I created an example PR in my monorepo fork: Lezek123/substrate-runtime-joystream#15. The only change introduced in this PR is an additional console log statement inside the orion integration tests scenario:

On the PR page I checked the hash of the commit in which I introduced this change:

The full hash of this commit is 6a6feac57cc31f7fbe779d2cf10db25f53fd5d57

I then added a /set-integration-tests-branch comment in the Orion PR and provided this hash as the target: Lezek123#5 (comment)

If you check the output of the integration tests check in the Orion PR (https://github.com/Lezek123/orion/actions/runs/5258188430/jobs/9502311512), you will notice that it includes the console.log added in commit 6a6feac57cc31f7fbe779d2cf10db25f53fd5d57 (from PR Lezek123/substrate-runtime-joystream#15):

[ignazio-bovo]

I don't have any improvement to add to the current proposal, I think it's valid. however:

1. If we are not adding any new features to the QN, it means that the integration tests for the query node will not be touched
2. I was under the idea that we will we drop the QN completely in favour of full backend in Orion. In that case would make sense to drop the integration tests also in the monorepo.

[Lezek123]

1. It's possibly true that we won't be adding any new features to QN, but I wouldn't rule out this possibility. Also I don't think not adding any new features to QN automatically implies the QN integration tests will not be touched. For example, we may discover new issues with the QN, missing test coverage etc. Also integration tests in the monorepo are testsing other services as well, like the Argus, Colossus, CLI etc. and the test coverage of those is quite low and would ideally be expanded in the future, so I imagine there may still be quite a few new changes to be introduced there.

2. Orion on its own will cannot completely replace QN. For example, it doesn't make sense to have Subsquid mappings related to council elections or proposals in Orion. They would need to be implemented as part of a separate backend service that would power Pioneer. Also Storage Node and Distributor Node will probably not use Orion as well, but instead have their own Subsquid mappings to only process the events they care about. So as it stands, it's very difficult to get rid of QN. It would take at least a few months to migrate all services to Subsquid.

   And even if you get rid of QN completely, the point of integration tests is not just to test the Query Node, but the entire Joystream infrastructure (ie. how all the services work together). So: Joystream node, Colossus nodes, Argus nodes etc.