Merge pull request #340 from dzhelezov/develop

Develop

docs/cloud/resources/best-practices.md → docs/cloud/best-practices.md

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 97
+sidebar_position: 50
 title: Best practices
 description: Checklist for going to production
 ---

docs/cloud/overview.md

@@ -5,15 +5,15 @@ description: Quickstart on how to deploy a squid
 
 # Quickstart
 
-This section goes through deploying a squid to [Subsquid Cloud](https://app.subsquid.io).
+This section goes through serverless deployment of an indexing project (*squid*) to [Subsquid Cloud](https://app.subsquid.io).
 The deployment is managed by the file `squid.yaml` in the root folder of the squid and defines:
 
 - [squid name and version](/cloud/reference/manifest/#header)
 - [which services should be deployed](/cloud/reference/manifest/#deploy) for the squid (e.g. postgres, processor, api, RPC proxy)
 - [what resources should be allocated](/cloud/reference/scale) by Cloud for each squid service (only configurable if you deploy to a [Professional organization](/cloud/resources/organizations/#professional-organizations))
 
 :::tip
-Make sure to check our [best practices guide](/cloud/resources/best-practices) before deploying to production!
+Make sure to check our [best practices guide](/cloud/best-practices) before deploying to production!
 :::
 
 :::warning
@@ -62,5 +62,5 @@ You can also read logs on the squid page in [Cloud](https://app.subsquid.io/squi
 - See how to [update](/squid-cli/deploy) and [kill](/squid-cli/rm) the deployed squid versions.
 - See [environment variables](/cloud/resources/env-variables) to add secrets to a squid deployment.
 - See the supported options for [`sqd logs`](/squid-cli/logs) such as filtering and log following.
-- Learn about [organizations](/cloud/resources/organizations) and [billing](/cloud/reference/pricing).
+- Learn about [organizations](/cloud/resources/organizations) and [billing](/cloud/pricing).
 - Learn how to [scale](/cloud/reference/scale) squids by requesting more resources.

docs/cloud/reference/pricing.md → docs/cloud/pricing.md

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 95
+sidebar_position: 20
 title: Pricing
 description: Subscription types and pay-as-you-go prices
 ---

docs/cloud/reference/rpc-proxy.md

@@ -117,4 +117,4 @@ processor.setDataSource({
 
 ## Pricing
 
-RPC proxy requests are priced [at a flat rate](/cloud/reference/pricing/#rpc-requests), with substantial packages included for free for all [organization types](/cloud/resources/organizations). Pricing does not depend on the call method.
+RPC proxy requests are priced [at a flat rate](/cloud/pricing/#rpc-requests), with substantial packages included for free for all [organization types](/cloud/resources/organizations). Pricing does not depend on the call method.

docs/cloud/reference/scale.md

@@ -17,7 +17,7 @@ Default: `dedicated: false`.
 By default, the service deployments are _collocated_ - that is, share resources with other deployments - and so the maximal allowed resource allocation is not guaranteed. With the `dedicated: true` option, the resources are reserved in advance. See the profile specifications below for details.
 We recommend setting `dedicated: true` for all squids running in production.
 
-Squid deployment must be dedicated to be a subject of [Subsquid Cloud SLA](../pricing/#slas).
+Squid deployment must be dedicated to be a subject of [Subsquid Cloud SLA](/cloud/pricing).
 
 ## `addons:`
 

docs/cloud/resources/organizations.md

@@ -33,7 +33,7 @@ Adding a valid payment method promotes an organization to Professional status. V
 
 ![Upgrading an organization](./billing-setup.png)
 
-Once an organization is upgraded you can deploy as many squids as you requre. [`scale:` section](/cloud/reference/scale) is now unlocked: use it to request any resources suitable for your use case. Your organization will be billed according to our [pricing schedule](/cloud/reference/pricing).
+Once an organization is upgraded you can deploy as many squids as you requre. [`scale:` section](/cloud/reference/scale) is now unlocked: use it to request any resources suitable for your use case. Your organization will be billed according to our [pricing schedule](/cloud/pricing).
 
 ## Working with organizations
 

docs/cloud/troubleshooting.md

@@ -4,7 +4,7 @@ sidebar_position: 60
 
 # Troubleshooting
 
-### `Secrets outdated. Please restart the squid` notification in Aquarium
+### "Secrets outdated. Please restart the squid" warning
 
 This occurs when you have a squid deployed, then create, remove or change some [environment variables](/squid-cli/secrets) of [relevance](/cloud/resources/organizations). Squids must be restarted manually for such changes to have effect. Navigate to the squid version page (e.g. by clicking on the warning sign) and click restart. The restart will not touch the database, so unless your new secret values cause the squid to crash this procedure should be quick and easy.
 
@@ -31,8 +31,8 @@ Since both the squid and version name become part of the squid API endpoint URL,
 
 ### My squid ran out of disk space
 
-Get in contact with the [Squid Squad](https://t.me/SquidDevs) and request extra resources. 
+Edit the [postgres addon](/cloud/reference/pg) section of `squid.yaml` and request more space for the database. 
 
-### I have exceeded the limit of squid versions
+### My squid is behind the chain, but is shows that it is in sync
 
-Get in contact with the [Squid Squad](https://t.me/SquidDevs) to get a Premium tier.
+Check that your processor uses both `archive:` and `chain:` data sources. 

docs/overview.mdx

@@ -69,7 +69,7 @@ A Platform-as-a-Service for deploying Squid SDK indexers, featuring
 - [zero downtime migrations](/cloud/resources/production-alias) between indexer versions
 - provisioning of [high-performance RPC endpoints](/cloud/reference/rpc-proxy) for real-time applications
 - intuitive deployment management through a [Web application](https://app.subsquid.io) or [CLI](/squid-cli)
-- Google Cloud-level [SLA](/cloud/reference/pricing/#slas)
+- Google Cloud-level [SLA](/cloud/pricing/#slas)
 
 ## What's next?
 
@@ -100,7 +100,7 @@ A Platform-as-a-Service for deploying Squid SDK indexers, featuring
 - Learn how to [deploy your squid](/cloud/overview)
 - Learn about [organizations](/cloud/resources/organizations) and how they are billed
 - Use your free [playground](/cloud/resources/organizations/#playgrounds) to develop a prototype of your indexer
-- View our [pricing schedule](/cloud/reference/pricing)
+- View our [pricing schedule](/cloud/pricing)
 
 ```mdx-code-block
 </TabItem>

docs/sdk/examples.mdx

@@ -1,5 +1,5 @@
 ---
-sidebar_position: 50
+sidebar_position: 31
 title: Examples
 description: Example squids
 ---

docs/sdk/faq.md

@@ -13,21 +13,14 @@ Here is an incomplete list:
 - Mining smart contract deployments and the bytecode
 - Real-time bots (\<1sec delay) triggered by on-chain activity
 
-### What is the Premium plan?
 
-The access to Archives is free. Subsquid Cloud offers a free plan and a premium plan. The Premium plan is currently available by invite via the [form](https://docs.google.com/forms/d/e/1FAIpQLSchqvWxRhlw7yfBlfiudizLJI9hEfeCEuaSlk3wOcwB1HQf6g/viewform?usp=sf_link). When released to the public, Premium will be based on the fixed subscription fee + pay-as-you-go model. Details will be announced in the official Subsquid channels.
+### How does Squid SDK handle unfinalized blocks?
 
-### Do you have a roadmap?
-
-Yes, see the issue in [the official repo](https://github.com/subsquid/squid-sdk/issues/70).
-
-### How does Subsquid handle unfinalized blocks?
-
-Archives index only finalized blocks. Handling unfinalized blocks and potential block reorganizations is supported by the Squid SDK since the ArrowSquid release, see [Indexing unfinalized blocks](/sdk/resources/unfinalized-blocks).
+The Subsquid Network only serves finalized blocks and is typically ~1000 blocks behind the tip. The most recent blocks, as well as the unfinalized blocks are seemlesly handled by the SDK from a complementary RPC data source, set by the `chain` config. Potential chain reorgs are automatically handled under the hood. See [Indexing unfinalized blocks](/sdk/resources/unfinalized-blocks) for details.
 
 ### What is the latency for the data served by the squid? 
 
-Since the ArrowSquid release, the Squid SDK has the option to ingest unfinalized blocks directly from an RPC endpoint, making the indexing real-time. Archive-only squids without an RPC data source typically have a latency of a few minutes to a few hours.
+Since the ArrowSquid release, the Squid SDK has the option to ingest unfinalized blocks directly from an RPC endpoint, making the indexing real-time. 
 
 ### How do I enable GraphQL subscriptions for local runs?
 
@@ -46,3 +39,7 @@ drop schema squid_processor cascade;
 to reset the processor status.
 
 Squids that store their data in [file-based datasets](/sdk/resources/persisting-data/file) store their status in `status.txt` by default. This can be overridden by defining custom [database hooks](/sdk/resources/persisting-data/file/#filesystem-syncs-and-dataset-partitioning).
+
+### Is there a healthcheck endpoint for the indexer?
+
+Yes, the processor exposes the key prometheus metrics at the `${process.env.PROMETHEUS_PORT}/metric` endpoint. The squids deployed to the Subsquid Cloud also publicly explose the metrics, see [Monitoring in the Cloud](/cloud/resources/monitoring/)

docs/sdk/overview.mdx

@@ -7,9 +7,13 @@ description: Architecture of squid indexers
 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
 
-# Squids overview
+# Overview
 
-_Squids_ are indexing projects that use [Squid SDK](https://github.com/subsquid/squid-sdk) to retrieve and process blockchain data from Subsquid data lakes, then save it or make it avaiable via an API.
+A _squid_ is an indexing project built with [Squid SDK](https://github.com/subsquid/squid-sdk) to retrieve and process blockchain data from the [Subsquid Network](/subsquid-network/overview)
+(either permissioned or decentralized instance). The Squid SDK is a set of typescript-based open-source libraries to decode, transform, persist and 
+make the transformed data avaiable via an API. All stages of the indexing pipeline, from the data extraction
+to transformation to persistence are designed to be batch-based to maximize the indexing speed. The modular architecture of the SDK
+makes it possible to extend the indexing projects (squids) with custom plugins and data targets.
 
 ## Required squid components
 

docs/sdk/reference/processors/evm-batch/state-diffs.md

@@ -7,7 +7,7 @@ description: >-
 # Storage state diffs
 
 :::tip
-State diffs for historical blocks are [currently available](/subsquid-network/reference/evm-networks) from Archives on the same basis as all other data stored there: for free. If you deploy a squid that indexes traces [in real-time](/sdk/resources/unfinalized-blocks) to Subsquid Cloud and use our [RPC proxy](/cloud/reference/rpc-proxy), the necessary `trace_` or `debug_` RPC calls made will be counted alongside all other calls and [the price](/cloud/reference/pricing/#rpc-requests) will be computed for the total count. There are no surcharges for traces or state diffs.
+State diffs for historical blocks are [currently available](/subsquid-network/reference/evm-networks) from Archives on the same basis as all other data stored there: for free. If you deploy a squid that indexes traces [in real-time](/sdk/resources/unfinalized-blocks) to Subsquid Cloud and use our [RPC proxy](/cloud/reference/rpc-proxy), the necessary `trace_` or `debug_` RPC calls made will be counted alongside all other calls and [the price](/cloud/pricing/#rpc-requests) will be computed for the total count. There are no surcharges for traces or state diffs.
 :::
 
 #### `addStateDiff(options)` {#add-state-diff}

docs/sdk/reference/processors/evm-batch/traces.md

@@ -7,7 +7,7 @@ description: >-
 # Traces
 
 :::tip
-Traces for historical blocks are [currently available](/subsquid-network/reference/evm-networks) from Archives on the same basis as all other data stored there: for free. If you deploy a squid that indexes traces [in real-time](/sdk/resources/unfinalized-blocks) to Subsquid Cloud and use our [RPC proxy](/cloud/reference/rpc-proxy), the necessary `trace_` or `debug_` RPC calls made will be counted alongside all other calls and [the price](/cloud/reference/pricing/#rpc-requests) will be computed for the total count. There are no surcharges for traces or state diffs.
+Traces for historical blocks are [currently available](/subsquid-network/reference/evm-networks) from Archives on the same basis as all other data stored there: for free. If you deploy a squid that indexes traces [in real-time](/sdk/resources/unfinalized-blocks) to Subsquid Cloud and use our [RPC proxy](/cloud/reference/rpc-proxy), the necessary `trace_` or `debug_` RPC calls made will be counted alongside all other calls and [the price](/cloud/pricing/#rpc-requests) will be computed for the total count. There are no surcharges for traces or state diffs.
 :::
 
 #### `addTrace(options)` {#add-trace}

docs/sdk/resources/self-hosting.md

@@ -7,7 +7,7 @@ description: Deploy squid locally or on-premises
 # Self-hosting
 
 :::tip
-Many tips from our [best practices guide](/cloud/resources/best-practices) apply to self-hosted squids, too. Check it out if you intend to use your squid in production.
+Many tips from our [best practices guide](/cloud/best-practices) apply to self-hosted squids, too. Check it out if you intend to use your squid in production.
 :::
 
 To deploy a squid locally or on-premises, use the following Dockerfile template to build a single image for both `api` and `processor` services:

docs/sdk/squid-from-scratch.mdx

@@ -1,13 +1,13 @@
 ---
 sidebar_position: 20
-title: Squid from scratch
+title: Indexer from scratch
 description: Learn to compose SDK packages
 ---
 
 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
 
-Here's an example of how SDK packages can be combined into a working squid indexer.
+Here's an example of how SDK packages can be combined into a working indexer (called *squid*).
 
 This page goes through all the technical details to make the squid architecture easier to understand. If you would like to get to a working indexer ASAP, [bootstrap from a template](/sdk/squid-development/#templates).
 
@@ -125,8 +125,8 @@ To make the indexer, follow these steps:
      const processor = new EvmBatchProcessor()
        .setDataSource({
          archive: lookupArchive('eth-mainnet'),
-         chain: { // for real-time updates
-           url: 'https://rpc.ankr.com/eth',
+         chain: { // set RPC endpoint in .env 
+           url: process.env.RPC_ETH_HTTP`,
            rateLimit: 10
          }
        })

redirectRules.js

@@ -257,7 +257,7 @@ const urlList = [
     },
     {
         "from": "/deploy-squid/best-practices",
-        "to": "/cloud/resources/best-practices"
+        "to": "/cloud/best-practices"
     },
     {
         "from": "/deploy-squid/organizations",
@@ -289,7 +289,7 @@ const urlList = [
     },
     {
         "from": "/deploy-squid/pricing",
-        "to": "/cloud/reference/pricing"
+        "to": "/cloud/pricing"
     },
     {
         "from": "/deploy-squid/promote-to-production",