Skip to content
This repository has been archived by the owner on Mar 4, 2024. It is now read-only.

Latest commit

 

History

History
38 lines (27 loc) · 2.12 KB

operations.md

File metadata and controls

38 lines (27 loc) · 2.12 KB

Indexer Operations

This document is intended to help engineers maintain the indexer and solve related on-call issues.

Expected issues

Web3 provider rate limits

The indexer pulls blockchain data from a full node aka web3 provider. We have seen the indexer exceed rate limits on our web3 provider in the past which brought the service to a halt, affecting analytics (no valora user transfer events going to BQ, Mixpanel, or Statsig). Retries along were not sufficient for the issue to go away without intervention.

Symptoms

You should see logs here related to rate limiting.

In case the link doesn't work, you can open Logs Explorer on the GCP console and paste the following query:

resource.type="gae_app"
resource.labels.module_id="indexer"
textPayload=~"rate limit"

Troubleshooting

We may choose to use Forno as a web3 provider to save on cost. That being said, the indexer works well using QuickNode as a web3 provider, even when coming back from a several day pause. If rate limit issues are seen with Forno, consider changing the web3 provider to QuickNode.

Note that QuickNode includes an API key in the web3 provider URL, so it is stored as a secret.

To change the web3 provider to QuickNode:

  • delete WEB3_PROVIDER_URL from app.mainnet.yaml if it is set there
  • find the web3 provider url from the QuickNode dashboard. At time of writing the endpoint is located here, see "HTTP Provider"
    • if you cannot access the QuickNode dashboard, ask an admin to add you. At time of writing, admins included Satish, Kathy, Silas, Charlie, Joe, and Jean.
  • add WEB3_PROVIDER_URL to indexer-secrets
  • redeploy the indexer for the changes to take effect