merging dynamic-pricing to main

.dockerignore

@@ -0,0 +1,4 @@
+node_modules
+helia*
+relay/node_modules
+relay/helia*

.github/workflows/test.yml

@@ -0,0 +1,38 @@
+name: Run Tests
+
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Set up Docker BuildX 
+      uses: docker/setup-buildx-action@v2
+    - name: Cache Docker layers
+      uses: actions/cache@v3
+      with:
+        path: /tmp/.buildx-cache
+        key: ${{ runner.os }}-buildx-${{ github.sha }}
+        restore-keys: |
+          ${{ runner.os }}-buildx-
+    - name: Set up Docker Compose
+      uses: hoverkraft-tech/[email protected]
+      with:
+        compose-file: "docker-compose-test.yml"
+        up-flags: "--build"
+    - name: Check Docker Containers
+      run: |
+        echo "🔍 Checking Docker Containers..."
+        docker compose -f "docker-compose-test.yml" ps
+    - name: Check Relay Service Logs
+      run: |
+        echo "🔍 Checking Relay Service Logs..."
+        sleep 30 && docker compose -f docker-compose-test.yml logs relay-service
+        # docker compose -f docker-compose-test.yml logs relay-service | grep "Actual listen addresses: "
+    - name: Running-Tests
+      run: |  
+        echo "🧪 Running tests..."
+        docker compose -f docker-compose-test.yml exec relay-service npm test

.gitignore

@@ -1,6 +1,18 @@
-node_modules/
-relay/node_modules
-relay/helia*
-data/
 .env
+.env*
 .idea
+data/
+helia-blocks/
+helia-data/
+node_modules/
+relay/helia*
+relay/helia-blocks
+relay/helia-data
+relay/ipns-keys/
+relay/leveldb
+relay/leveldb/
+relay/node_modules
+relay/orbitdb
+relay/relay.lock
+relay/scanning-state*
+relay/scanning-state.json

README.md

@@ -11,7 +11,7 @@ This is a docker image and docker compose file which runs a
 ### via nodejs
 0. Copy .env.example to .env and change according to your environment
     - RELAY_DEV_MODE=true in order to disable gossip-sub tresholds
-    - SERVER_NAME=your-server-name.domain.xyz
+    - SERVER_NAME=your-server-ipnsInstance.domain.xyz
     - RELAY_PUBSUB_PEER_DISCOVERY_TOPICS=your-topic._peer-discovery._p2p._pubsub
     - RELAY_PRIVATE_KEY=how to generate a good private key for your node?
 1. Run ```node relay/src/relay.js```
@@ -24,11 +24,38 @@ This is a docker image and docker compose file which runs a
 3. Run ```docker-compose up -d``` (nginx, letsencrypt, relay-service starting)
 4. Run ```docker-compose logs``` and find the libp2p address to connect your peers
 
-## Todo
-- [x] move private key from relay.js into .env
-- [x] SERVER_NAME inside docker-compose.yml should go into .env
-- [ ] init-letsencrypt.sh should take SERVER_NAME as an argument or environment variable 
-- [x] make listen addresses of relay service configurable via .env / docker-compose
-- [x] make announce: ['/dns4/ipfs.le-space.de//tcp/443/wss'] configurable via .env / docker-compose
-- [x] make bootstrap seed nodes configurable via .env / docker-compose
-- [x] make pubsubPeerDiscovery topics configurable via .env / docker-compose
+### Ideas
+When starting a relay-node for libp2p, it would be nice to have place where other peers could find the multi address of our new peer.
+It is also the case that webtransport and webrtc peers and generate new hashes to connect. (E.g. webtransport hashes need to renewed every 2 weeks).
+
+The idea is now to store those multi addresses on a blockchain such as Bitcoin (via Ordinals) or Doichain (a merged-mining Namecoin fork)
+
+What will it do?
+1. During start, connect to the blockchain (e.g., Electrumx) and query a certain namespace for seed nodes and 
+2. If our node is not yet stored on blockchain and a private key is inside .env it should send a transaction to a blockchain
+
+# Adaptation of this relay
+- [ ] pinner should calculate fee based on file size 
+- [ ] pinner should only pin if cid is inside of nameOp transaction and 
+  - a minimum fee per kb is paid to a certain doichain address (-> multisig)
+- [x] BUG: fix scanning blocks on new tip
+- [x] tipWatcher:  when every a new block comes in the scan should be restarted. (Remark: src/pinner/tipWatcher already implemented but functionality not reviewed nor untestet)
+- [x] adding a Telegram bot to send certain messages e.g. status, minted nameOps etc
+- [x] telegram bot send message when somebody adds / added a file (including name, description, filename, filetype and size)
+- [x] telegram bot should send message when a nameOp with ipfs:// cid got pinned. It should display amount of pins and the space they are taking on the hard drive
+- [x] pin only when nameOps arrive and inform network by pubusb  
+- [x] nameop_cids and failed_cids should go into a OrbitDB. So we can use the scanning functions also in every peer (e.g. browser and mobile app)
+- [x] when a scan is discovering a nameOp it should write it into data/failed_cids.json
+  - [ ] write a test 
+- [x] when a peer connects it retries failed and removes successful loads from data/failed_cids.json
+- [x] display also "unconfirmed"
+- [x] the relay is scanning Doichain blockchain for name-ops
+- [x] if a name-ops is being found, it needs to be investigated 
+  - [x] create a file which contains all name-ops of today an ipns name-ops (with the today date e.g. 2024-10-09) and references the files cid
+  - [x] everytime a new name-ops is found this file is being added again and the new resulting cid updated to the ipns name-ops
+  - [x] if name-op contains an ipfs:// url the CID needs to be gathered from the ipfs node (or the network)
+    - [x] if its available wie pin it  
+    - [x] or if its not we store the cid in failed-cids 
+- [x] every browser can call LIST-DATE (e.g. LIST-20241014) 
+- [x] every relay who has the name_ops of today is publishing it (not sure if that is efficient enough)
+- [x] pinner should only pin if cid is inside of nameOp transaction 

docker-compose-test.yml

@@ -0,0 +1,49 @@
+services:
+  regtest:
+    image: "doichain/core:dc0.20.1.15"
+    container_name: regtest
+    ports:
+      - "18332:18332"
+      - "18443:18443"
+      - "18445:18445"
+    # command: ["/usr/local/bin/doichaind"]
+    volumes:
+      - ./docker/doichain-regtest.conf:/home/doichain/.doichain/doichain.conf
+
+  electrumx:
+    container_name: electrumx
+    hostname: electrumx
+    image: "doichain/electrumx:dc1.2.1"
+    depends_on:
+      - regtest
+    volumes:
+      # - ./db:/var/lib/electrumx
+      - ./docker/startElectrumDOIRegTest.sh:/startElectrumDOIRegTest.sh
+    entrypoint: ["/startElectrumDOIRegTest.sh"]
+    ports:
+      - "50001:50001"
+      - "50002:50002"
+      - "8443:8443"
+    stdin_open: true
+    tty: true
+    privileged: true
+
+  relay-service:
+    build:
+      context: .
+      dockerfile: docker/Dockerfile
+    volumes:
+      - ./docker/entrypoint.sh:/usr/src/app/entrypoint.sh
+      # - ./:/usr/src/app/
+    entrypoint: ["/usr/src/app/entrypoint.sh", "generate-key"]
+    ports:
+      - "1235:1235"
+      - "9091:9091"
+      - "12345:12345"
+      - "3000:3000"
+    depends_on:
+      - regtest
+      - electrumx
+
+volumes:
+  doichain-data:

docker-compose.monitoring.yml

@@ -0,0 +1,71 @@
+services:
+  prometheus:
+    image: prom/prometheus:v3.0.0
+    container_name: prometheus
+    ports:
+      - 90:9090
+    command:
+      - '--config.file=/etc/prometheus/prometheus.yaml'
+    volumes:
+      - ./prometheus.yaml:/etc/prometheus/prometheus.yaml
+      - ./prometheus_data:/prometheus
+    restart: unless-stopped
+    networks:
+      - local-first-network
+      - monitoring
+
+  grafana:
+    image: grafana/grafana-oss:latest
+    container_name: grafana
+    ports:
+      - 3001:3000
+    volumes:
+      - grafana-data:/var/lib/grafana
+    restart: unless-stopped
+    networks:
+      - local-first-network
+      - monitoring
+
+  node_exporter:
+    image: quay.io/prometheus/node-exporter:v1.8.2
+    container_name: node_exporter
+    command: '--path.rootfs=/host'
+    pid: host
+    restart: unless-stopped
+    volumes:
+      - /:/host
+    networks:
+      - local-first-network
+      - monitoring
+
+  cadvisor:
+    image: gcr.io/cadvisor/cadvisor:v0.47.0
+    container_name: cadvisor
+    command:
+      - '-port=8098'
+    volumes:
+      - /:/rootfs:ro
+      - /etc/machine-id:/etc/machine-id:ro
+      - /var/run:/var/run:ro
+      - /sys:/sys:ro
+      - /var/lib/docker/:/var/lib/docker:ro
+      - /dev/disk/:/dev/disk:ro
+    devices:
+      - /dev/kmsg
+    privileged: true
+    restart: unless-stopped
+    networks:
+      - local-first-network
+      - monitoring
+
+volumes:
+  prometheus-data:
+    driver: local
+  grafana-data:
+    driver: local
+
+networks:
+  local-first-network:
+    external: true
+  monitoring:
+    driver: bridge

docker-compose.yml

@@ -1,4 +1,3 @@
-version: '3.2'
 services:
   nginx:
     image: nginx:1.22.0
@@ -17,15 +16,20 @@ services:
       - ./data/certbot/www:/var/www/certbot
     entrypoint: "/docker-entrypoint.sh"
     command: ["nginx"]
+    networks:
+      - local-first-network
   certbot:
     image: certbot/certbot
     volumes:
       - ./data/certbot/conf:/etc/letsencrypt
       - ./data/certbot/www:/var/www/certbot
     entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
+    networks:
+      - local-first-network
   relay-service:
     build: .
     ports:
+      - "3000:3000"
       - "1235:1235"
       - "12345:12345"
     volumes:
@@ -34,3 +38,12 @@ services:
     environment:
       - DEBUG=libp2p:*
       - NODE_ENV=production
+    #command: ["node", "relay/src/start-relay.js", "--disable-scanning"]
+    # command: ["node", "relay/src/start-relay.js"] 
+    command: ["node", "--max-old-space-size=8192", "relay/src/start-relay.js"]
+    networks:
+      - local-first-network
+
+networks:
+  local-first-network:
+    external: true

docker/Dockerfile

@@ -0,0 +1,8 @@
+FROM node:20
+WORKDIR /usr/src/app
+COPY relay/ ./
+COPY docker/entrypoint.sh ./
+COPY relay/.env.example ./.env
+RUN rm -rf node_modules && npm install
+ENTRYPOINT [ "./entrypoint.sh" ]
+CMD [ "node", "src/relay.js"]

docker/doichain-regtest.conf

@@ -0,0 +1,27 @@
+regtest=1
+daemon=1
+server=1
+wallet=1
+rpcuser=admin
+rpcpassword=adminpw
+rpcallowip=0.0.0.0/0
+rpcbind=0.0.0.0
+txindex=1
+fallbackfee=0.00001
+namehistory=1
+#blocknotify=curl -X GET http://localhost:3000/api/v1/blocknotify?block=%s
+#walletnotify=curl -X GET http://localhost:3000/api/v1/walletnotify?tx=%s
+
+[test]
+rpcport=18332
+rpcbind=0.0.0.0
+rpcallowip=0.0.0.0/0
+wallet=1
+port=18445
+
+[regtest]
+rpcport=18332
+rpcbind=0.0.0.0
+rpcallowip=0.0.0.0/0
+wallet=1
+port=18445

docker/entrypoint.sh

@@ -0,0 +1,28 @@
+#!/bin/bash 
+rm scanning-state.json
+
+if [ "$1" == "generate-key" ]; then
+    echo "Generating private key..."
+    npm run generate-keypair
+    echo "--------------------------------"
+    key_output=$(cat .env.privateKey)
+
+    echo "Private key generated: $key_output"
+    # Replace or add the RELAY_PRIVATE_KEY in the .env file
+    if grep -q "^RELAY_PRIVATE_KEY=" .env; then
+        sed -i "s/^RELAY_PRIVATE_KEY=.*/$key_output/" .env
+    else
+        echo "$key_output" >> .env
+    fi
+    cat .env
+    npm run start
+elif [ "$1" == "start" ]; then
+    echo "Starting node..."
+    npm run start
+else
+    echo "Invalid command. Use 'generate-key' or 'start'."
+    exit 1
+fi
+
+# Start the application if a valid command was provided
+# exec "$@"