Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing data #198

Open
fewensa opened this issue Sep 13, 2024 · 3 comments
Open

Missing data #198

fewensa opened this issue Sep 13, 2024 · 3 comments

Comments

@fewensa
Copy link

fewensa commented Sep 13, 2024

Describe the bug

https://envio.dev/app/darwinia-network/dip7index/1c7b270/playground

This playground,

First query chain metadata

  chain_metadata {
    start_block
    block_height
    end_block
    is_hyper_sync
    chain_id
  }

Result is

      {
        "start_block": 1075463,
        "block_height": 1121690,
        "end_block": null,
        "is_hyper_sync": false,
        "chain_id": 701
      }

Then query addCollator

  CollatorStakingHub_AddCollator(
    where: {cur: {_eq: "0x86c3e255128782947ca8f21b7ed1166c9241f560"}}
  ) {
    cur
    votes
    prev
    blockNumber
  }

Result is

   "CollatorStakingHub_AddCollator": [
      {
        "cur": "0x86c3e255128782947ca8f21b7ed1166c9241f560",
        "votes": "0",
        "prev": "0x0f14341a7f464320319025540e8fe48ad0fe5aec",
        "blockNumber": "1116809"
      },
      {
        "cur": "0x86c3e255128782947ca8f21b7ed1166c9241f560",
        "votes": "0",
        "prev": "0x0f14341a7f464320319025540e8fe48ad0fe5aec",
        "blockNumber": "1116811"
      },
      {
        "cur": "0x86c3e255128782947ca8f21b7ed1166c9241f560",
        "votes": "359",
        "prev": "0x95ec69dd90e4b120c413b209b3a31a5c74a58b76",
        "blockNumber": "1116815"
      }
    ],

this block events is missing
https://koi-scan.darwinia.network/tx/0x11a1bf08a4d7bf52f8bb428bbffece62e21cef340e63a1c08b18e63241a47b52?tab=logs

1117132

To Reproduce
Steps to reproduce the behavior:

E.g.

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Local (please complete the following information):

  • Envio version 2.2.0
  • Node version
  • pnpm version
  • Docker version (if running locally)

Hosted Service (please complete the following information):

Additional context
Add any other context about the problem here.

@JonoPrest
Copy link
Collaborator

Hey @fewensa, could share a way to reproduce your indexer?

Does this happen deterministically each time you run your indexer?

Do you have rollback_on_reorg enabled?

@fewensa
Copy link
Author

fewensa commented Sep 13, 2024

  1. reproduce
    Deploy https://github.com/darwinia-network/dip7index to hosted service, or run local npm run dev
  2. Yes
  3. Yes, enabled, and set confirmed_block_threshold: 8 https://github.com/darwinia-network/dip7index/blob/1c7b270ac7f758db47d2940fc6fb75cc22e5560f/config.yaml#L32

@JonoPrest
Copy link
Collaborator

Hey @fewensa,

I've done some analysis on your case here. It is an interesting one because I didn't expect local/hosted environments to have such different data.

I ran your indexer locally and compared all your entities to find that there were 3647 mismatch where an entity appeared either locally or in the hosted indexer but not the other. They were all on the koi network. I took the block numbers from all of these queried the koi-scan API to find that the majority of these blocks were reorg blocks. OR they had neighbouring blocks which were reorg blocks. A small amount of them had a reorg block present within 10 blocks.

So unfortunately the data inaccuracy is currently due to reorgs. And there is a known edge case with our reorg detection implementation RPC in that it is impossible to atomically run eth_getLogs AND eth_getBlock. Since we are checking blockhashes on the toBlock of a getLogs query its possible for Reorgs to slip by undetected. Which appears to be the case here.

I'm going to do further investigation on an reorg detection with RPC to see if we can improve it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants