Add rate limited notifications queue

[davecreaser]

Description

• Since this is really a block ingestor change, the description is here: Add rate limited notifications queue block-ingestor#320

Testing

• ⚠️ You'll need to restart your dev environment and make sure you run npm run dev --notifications to enable notifications locally.
• Once it's up and running, stop the block ingestor in docker. Open .env of the CDapp and copy the auto generated MAGICBELL_DEV_KEY, and add it to the .env of your block ingestor. Now run the block ingestor locally with npm run dev.
• Now you need a way to trigger more than 60 notifications inside a minute. The easiest way is probably to open src/handlers/actions/mintTokens.ts in the block ingestor, and go to line 35 where it calls sendPermissionsActionNotifications. Just add a for loop around this call to fire it like 80 times when a mint token action happens.

for (let index = 0; index < 80; index++) {
      sendPermissionsActionNotifications({
        creator: initiatorAddress,
        colonyAddress,
        transactionHash,
        notificationCategory: NotificationCategory.Payment,
      });
    }

• Open localhost, and mint some tokens. You should see 30 notifications come in pretty quickly, and after waiting a minute, more should come through.
• Check the block ingestor logs, there should be no errors about API rate limits being hit.

Diffs

Changes 🏗

• Block ingestor changes: Add rate limited notifications queue block-ingestor#320

[mmioana]

Really nice work on this one @davecreaser 🙌

Applied the changes in the local block-ingestor and created a mint tokens action

There were no errors in the console

Added an extra log to check the queue size and it seems the extra 20 notifications compared to the queue cap were added

However @davecreaser, after creating many more Mint tokens actions, I did manage to still spot the API limit reached error. Not sure if I did something wrong?

[davecreaser]

Thanks @mmioana, turns out they had more hidden rate limits 🙄 I think this one happened because the front end was also sending a fetch request each time a new notification came in (to get that notification) and these were all combining to then hit the rate limit again.

I've reduced the block ingestor rate to only send 30 per minute, which will keep us nicely within the limits, and leave a big buffer for the front end to make requests.

Also added a retry mechanism with priority to the notifications queue, so that notifications which still hit the rate limit (unlikely hopefully) will get retried.

[bassgeta]

Nice one 😎 And a really nice choice of package, so we don't need to run our own heavy duty queue instances for simple stuff like this 👌


Waited for a minute and posted some dank memes into our chat, and voila, here is the rest!

[mmioana]

Thanks for your really cool work on this issue @davecreaser 🥇

Applied the local changes to the block ingestor and created a Mint tokens action
Immediately I got 30 notifications

No errors on the block ingestor side

After 1 minute I got 30 more new notifications

And after another minute, all notifications were there

Awesome work! 🎉

[rdig]

Rate limiting working as expected, nothing triggered in the block ingestor logs, while notifications showing up in the frontend end.

Nice job fixing this