Scripts to setup 4 entropy-tss VMs for test network

[ameba23]

I have been experimenting with setting up 4 entropy-tss servers in 4 TDX enclaves, for a test network (4 is the minimum we need to be able to test the protocols and have a functional network).

To do this we need 4 images on the host, named eg: tdx-guest-ubuntu-24.04-generic-entropy-tss-alice.qcow2, tdx-guest-ubuntu-24.04-generic-entropy-tss-bob.qcow2, etc.

Here is the script i am running on the host: (adapted from the script in the
canonical tdx repo)

#!/bin/bash

cleanup() {
    rm -f /tmp/tdx-guest-*.log &> /dev/null
    rm -f /tmp/tdx-demo-*-monitor.sock &> /dev/null

    PROCESS_NAME="entropy-tss-$1"
    PID_FILE="/tmp/tdx-${PROCESS_NAME}-pid.pid"
    PID_TD=$(cat ${PID_FILE} 2> /dev/null)

    [ ! -z "$PID_TD" ] && echo "Cleanup, kill TD with PID: ${PID_TD}" && kill -TERM ${PID_TD} &> /dev/null
    sleep 3
}

run-tss () {
	SSH_PORT=$1
	PROCESS_NAME="entropy-tss-$2"
	TD_IMG="${PWD}/image/tdx-guest-ubuntu-24.04-generic-${PROCESS_NAME}.qcow2"
	ENTROPY_TSS_HTTP_PORT=$3
	LOGFILE="/tmp/tdx-guest-td-${PROCESS_NAME}.log"
	PID_FILE="/tmp/tdx-${PROCESS_NAME}-pid.pid"
	# approach 1 : userspace in the guest talks to QGS (on the host) directly
	QUOTE_VSOCK_ARGS="-device vhost-vsock-pci,guest-cid=$4"
	# approach 2 : tdvmcall; see quote-generation-socket in qemu command line

	echo "Starting ${PROCESS_NAME} using image ${TD_IMG}"
	qemu-system-x86_64 -D $LOGFILE \
		-accel kvm \
		-m 2G -smp 16 \
		-name ${PROCESS_NAME},process=${PROCESS_NAME},debug-threads=on \
		-cpu host \
		-object '{"qom-type":"tdx-guest","id":"tdx","quote-generation-socket":{"type": "vsock", "cid":"2","port":"4050"}}' \
		-machine q35,kernel_irqchip=split,confidential-guest-support=tdx,hpet=off \
		-bios ${TDVF_FIRMWARE} \
		-nographic -daemonize \
		-nodefaults \
                -device virtio-net-pci,netdev=nic0_td -netdev user,id=nic0_td,hostfwd=tcp::${SSH_PORT}-:22,hostfwd=tcp::${ENTROPY_TSS_HTTP_PORT}-:3001 \
		-drive file=${TD_IMG},if=none,id=virtio-disk0 \
		-device virtio-blk-pci,drive=virtio-disk0 \
		${QUOTE_VSOCK_ARGS} \
		-pidfile ${PID_FILE}

	ret=$?
	if [ $ret -ne 0 ]; then
		echo "Error: Failed to create TD VM. Please check logfile \"$LOGFILE\" for more information."
		exit $ret
	fi

	PID_TD=$(cat ${PID_FILE})

	echo "TD, PID: ${PID_TD}, SSH : ssh -p ${SSH_PORT} root@localhost"
}

declare -a names=("alice" "bob" "charlie" "dave")

for i in "${names[@]}"
do
	cleanup $i
done
if [ "$1" = "clean" ]; then
    exit 0
fi

TDVF_FIRMWARE=/usr/share/ovmf/OVMF.fd

if ! groups | grep -qw "kvm"; then
    echo "Please add user $USER to kvm group to run this script (usermod -aG kvm $USER and then log in again)."
    exit 1
fi

# ssh-port, name, tss-port, guest-cid
run-tss 10023 alice 3001 4
run-tss 10024 bob 3002 5
run-tss 10025 charlie 3003 6
run-tss 10026 dave 3004 7

When run a second time, the script will kill existing processes before starting new ones. It can also be run with the clean command line argument to only kill processes and remove logs, and not start the VMs.

Each VM image needs to be already provisioned with the entropy-tss binary and an enabled systemd service file customised for alice, bob, etc. For example for alice:

[Unit]
Description=Entropy Threshold Signature Server
After=multi-user.target

[Service]
ExecStart=/usr/bin/entropy-tss --alice --chain-endpoint ws://10.0.2.2:9944 --threshold-url 10.0.2.15:3001
Type=simple

[Install]
WantedBy=multi-user.target

I have not yet automated the process of provisioning the VMs, but it could be done as part of this script: https://github.com/canonical/tdx/blob/387764af665cddce319af1667c1648ee988947e5/guest-tools/image/create-td-image.sh

Then we also need to run 4 chain nodes on the host. I am unsure if to use a bash script, docker-compose or by adapting the helper function we use for spinning up the 4 chain nodes we use for entropy-tss tests: https://github.com/entropyxyz/entropy-core/blob/425b2819b6c309adf78e2ce54217937163072eca/crates/testing-utils/src/substrate_context.rs#L112

[ameba23]

In production all nodes will ideally run the same image - that is, we will not be able to set a custom chain endpoint for each node as a command line argument when running entropy-tss . Rather we should use QEMU guest forwarding so that client connections to a standard chain endpoint get forwarded to the actual desired chain endpoint for that node.

From the qemu-system-x86_64 man page:

 guestfwd=[tcp]:server:port-dev; guestfwd=[tcp]:server:port-cmd:command
                     Forward  guest  TCP connections to the IP address server on port port to the character device dev or to a program executed by cmd:command which gets spawned for each
                     connection. This option can be given multiple times.

                     You can either use a chardev directly and have that one used throughout QEMU's lifetime, like in the following example:

                        # open 10.10.1.1:4321 on bootup, connect 10.0.2.100:1234 to it whenever
                        # the guest accesses it
                        qemu-system-x86_64 -nic user,guestfwd=tcp:10.0.2.100:1234-tcp:10.10.1.1:4321

So for our 4 node test network could have the following in the qemu-sytem-x86_64 command for each TSS node:
guestfwd=tcp:10.0.2.2:9944-tcp:10.10.1.1:9944
guestfwd=tcp:10.0.2.2:9945-tcp:10.10.1.1:9944
guestfwd=tcp:10.0.2.2:9946-tcp:10.10.1.1:9944
guestfwd=tcp:10.0.2.2:9947-tcp:10.10.1.1:9944

and in the CVM image, always run entropy-tss with:

entropy-tss --chain-endpoint ws://10.10.1.1:9944 --threshold-url 10.0.2.15:3001