Skip to content

Latest commit

 

History

History
132 lines (106 loc) · 7.84 KB

PINNING-SERVICE.md

File metadata and controls

132 lines (106 loc) · 7.84 KB

IPBIS - An Experimental Peer-To-Peer IPFS-Pinning and blockchain indexing Service

The index & pinning service has two functions:

  1. pinning IPFS CIDs
  2. indexing blockchain blocks for certain transactions containing references to CIDs on the IPFS network and pinning payment data

The pinning service in the ipbis project is designed to manage persistence of data on the IPFS network. It can receive new CIDs via gossip-sub from browser peers or other peers running libp2p in various programming languages such as JavaScript, Go, Rust, Python, C/C++, and Java.

As a CID is received by this pinning service, it gathers the CID from the IPFS network. This implies that the file was previously added to a Helia (IPFS) node in a browser. The relay adds the CID to its IPFS storage and will pin it as soon as a payment is received. The pinning service examines the CID's data and checks if it is a normal file (e.g., image or video) or if it is a metadata.json which contains a reference to another CID, which is then also loaded from the IPFS network.

The pinning service responds with size and pricing information in cryptocurrency for the CID to be pinned. In the current version, the pinning service waits for a transaction made to a certain address on the Doichain blockchain, an experimental fork of the Namecoin blockchain. Theoretically, in future versions, it can support other blockchains such as Bitcoin, Namecoin, or blockchains like Ethereum or Polygon, etc., to watch for payment data for pinning.

+-------------------+ +-------------------+ | Requesting Peer | | Pinning Service | +-------------------+ +-------------------+ | | | 1. NEW-CID Request | |----------------------------->| | | | | | 2. Gather CID from IPFS | |<-----------------------------| | | | | | 3. ADDING-CID Broadcast | |<-----------------------------| | (Includes CID, size, fee, | | payment address) | | | | | | 4. Payment Confirmation | |----------------------------->| | (Transaction on blockchain) | | | | | | 5. CID-PINNED-WITH-NAME | |<-----------------------------| | (If applicable, includes | | nameId, expirationDate) | | | | | | 6. ADDED-CID Broadcast | |<-----------------------------| | (Confirms successful pinning)| | | | | | 7. Metadata Storage | |<-----------------------------| | (Store metadata in OrbitDB) | | | +-------------------+ +-------------------+ | Requesting Peer | | Pinning Service | +-------------------+ +-------------------+

Limitations:

  • Participating nodes need to agree on a multisig address for the payments made (this can be configured in .env).
  • If other pinning nodes join the same network (same gossip-sub topic), this cannot be prevented, but the peers requesting the pinning so far would have to trust a certain set of nodes, and they could offer their own prices.
  • Participating nodes are mirroring the full set of data.
  • Participating nodes are not validating each other for the completeness of their data.

Todos

  • Participating nodes automatically mirroring their data via a distributed OrbitDB table.
  • As more nodes join the network, shards are automatically increased/decreased.
  • Depending on the number of participating nodes, multiple nodes are mirroring their data.
  • Daily incentives.
  • Stake is necessary to join the network.
  • Slashing algorithm for missing data.

Key Components

  • Helia: Used for interacting with the IPFS network.
  • OrbitDB: A distributed database used to store metadata about pinned content.
  • Electrum: Interacts with PoW blockchains for payment validation.
  • PubSub: Facilitates message broadcasting and receiving for pinning operations.

How It Works

1. Initialization

The PinningService is initialized with Helia, OrbitDB, and Electrum.

2. PubSub Setup

The setupPubsub function subscribes to a specific content topic. It listens for messages related to pinning requests and broadcasts updates.

3. Handling PubSub Messages

When a message is received on the content topic, the service processes it based on its type:

  • LIST Requests: Delivers indexed NameOps and optionally associated pinned media files.
  • NEW-CID Requests: Initiates the pinning process for new content identifiers (CIDs).

4. Content Pinning

The pinContent method handles the pinning process, calculating fees, validating payments, and storing metadata in OrbitDB.

5. PubSub Message Broadcasting

  • ADDING-CID: Broadcasted when the service begins processing a new CID for pinning. It includes:

    • cid: The content identifier being processed.
    • sizes: Information about the file size.
    • fee: The expected fee for pinning, including:
      • amount: The calculated fee.
      • durationMonths: The duration for which the content will be pinned.
      • paymentAddress: The address where the payment should be sent.

    When an ADDING-CID message is received, the relay service gathers the CID from a browser or other node to add it to the local node. As soon as a NameOp transaction with payment is confirmed on the blockchain, the CID will be pinned.

  • CID-PINNED-WITH-NAME: Sent if the CID is already pinned with an associated name. It includes:

    • cid: The content identifier.
    • nameId: The identifier of the associated name.
    • expirationDate: The date when the pinning will expire.
    • remainingDays: The number of days remaining until expiration.

  • ADDED-CID: Broadcasted after successfully pinning the content, indicating the completion of the operation. It includes:

    • cid: The content identifier that has been pinned.

    When an ADDED-CID message is received, it confirms that the CID has been successfully added to the local node.

6. Pinning Metadata

Metadata about the pinning operation, such as CID, file name, size, and payment details, is stored in an OrbitDB document store.

7. Pin Status Check

The shouldRemainPinned method checks if content should remain pinned based on expiration and payment status.

8. Error Handling

The service logs errors and broadcasts error messages via pubsub to inform peers of any issues during the pinning process.

Planned Features

  • Members of the MultiSig pinning the data fully decentralized, meaning as more pinning nodes are added, the safer it becomes.
  • Members of the MultiSig will check other members to ensure they have truly pinned the data promised. If not, they would lose their stake paid into the MultiSig.
  • Monthly revenue payouts to the MultiSig members.
  • DAO functionalities.

This integration with pubsub allows the pinning service to efficiently handle requests and communicate status updates across the network, ensuring a responsive and interactive peer-to-peer environment. The detailed pubsub messages, such as ADDING-CID, CID-PINNED-WITH-NAME, and ADDED-CID, provide clear communication about the status of pinning operations, including specific attributes like cid, fee, nameId, and expirationDate, which help in tracking and managing the pinning lifecycle.