Cluster is not bootstrapped properly

[mightymouse2045]

Not sure what's going on but I've tried 3 different cluster deployments and they all get stuck, In Rancher the cluster is just sitting there:

Updating test Elemental RKE2 v1.28.8+rke2r1 — — 0/0
Non-ready bootstrap machine(s) test-pch-5bd8596f47xb6zfc-vw6gq and join url to be available on bootstrap node

I am getting this in the cattle-cluster-agent after deploying elemental cluster:

time="2024-05-06T19:31:50Z" level=error msg="error syncing 'rancher-partner-charts': handler helm-clusterrepo-download: >update failure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner->charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 fetch origin -- main error: exit status 128, >detail: error: RPC failed; curl 18 Recv failure: Connection reset by peer\nfetch-pack: unexpected disconnect while reading >sideband packet\nfatal: early EOF\nfatal: fetch-pack: invalid index-pack output\n, requeuing"

Keep getting this repeating shown in systemctl rancher-system-agent:

o msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: and exit code: 127"
o msg="[K8s] updated plan secret fleet-default/test-bootstrap-template-zk24x-machine-plan with feedback"
o msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20240506-193119/d4337b207d9fc58d8de17ab3e9a0ed24bae4aed3274e0d2b8008c53b29611f40_0"
o msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
o msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: and exit code: 127"
o msg="[K8s] updated plan secret fleet-default/test-bootstrap-template-zk24x-machine-plan with feedback"
o msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20240506-193419/d4337b207d9fc58d8de17ab3e9a0ed24bae4aed3274e0d2b8008c53b29611f40_0"
o msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
o msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: and exit code: 127"
o msg="[K8s] updated plan secret fleet-default/test-bootstrap-template-zk24x-machine-plan with feedback"

[mightymouse2045]

Anyone have any suggestions or help with this error?

[mightymouse2045]

Really can't get around this one. I'm running Rancher 2.8.3, I've tried rke2 1.28.8, 1.27.12 - is it the version of Rancher perhaps?

[fgiudici]

Hey @mightymouse2045 👋🏼 ,
can you elaborate a bit more on your setup and what steps have you done?
Have you tried a deployment similar to what is described in the quickstart (https://elemental.docs.rancher.com/quickstart-cli or https://elemental.docs.rancher.com/quickstart-ui) but deploying the Elemental chart from the Rancher apps?
How many and which kind of machines do you want to join to the cluster?

[mightymouse2045]

Ok - so just to clarify I am running the default quickstart, but with a custom iso build as the seed image. I have just run the quickstart with the default ISO seedimage and it works and creates a cluster and connects into Rancher successfully. So it is clearly something I'm doing wrong with the custom ISO.

I'll dig into that some more and provide an update back here...

[mightymouse2045]

So adding a node that was built with my custom ISO to the cluster that was initialised with the default seed ISO works

[mightymouse2045]

Ok so I have deployed a new cluster with a custom iso and it is stuck here again:

This is my ISO build process here:

1. Grab the base docker image

FROM registry.suse.com/suse/sle-micro/5.5:2.0.2
COPY EFI/BOOT /usr/share/grub2/x86_64-efi
ARG IMAGE_REPO='merepo-elemental-img'
ARG IMAGE_TAG='v1.1.1'
RUN
sed -i -e "s/^IMAGE_REPO=./IMAGE_REPO="${IMAGE_REPO}"/g" /etc/os-release &&
sed -i -e "s/^IMAGE_TAG=./IMAGE_TAG="${IMAGE_TAG}"/g" /etc/os-release &&
sed -i -e "s/^IMAGE=.*/IMAGE="${IMAGE_REPO}:${IMAGE_TAG}"/g" /etc/os-release

2. Create the ISO

FROM australia-southeast1-docker.pkg.dev/cpas-269913/public/pch-elemental-img:v1.1.1 AS os
FROM merepo/elemental-img:v1.1.1 AS builder
LABEL stage=builder
ARG TARGETARCH=x86_64
WORKDIR /iso
COPY --from=os / rootfs
COPY overlay overlay
RUN echo "some_ip some_host" >> rootfs/etc/hosts &&
echo "nameserver 192.168.86.1" >> rootfs/etc/resolv.conf &&
echo "options eth0 trust-ad" >> rootfs/etc/resolv.conf &&
echo "search ." >> rootfs/etc/resolv.conf
RUN --mount=type=bind,source=./,target=/output,rw
elemental build-iso
dir:rootfs
--bootloader-in-rootfs
--squash-no-compression
--overlay-iso /iso/overlay
-o /output -n "elemental-${TARGETARCH}"

So as you can see I'm not doing anything to the base build at this stage, but it fails to connect to rancher with the same bootstrap status.

PODS look good:

[mightymouse2045]

As a work around - I can build with the default seed image. Add the other nodes with my custom ISO, and then perform an upgrade on the seed node to get it to the same version as the others...

ugly but it gets me there. Just gotta try the upgrade side of things to see if that works

[mightymouse2045]

Ok following the upgrade side of things doesn't work for me. https://elemental.docs.rancher.com/upgrade

I have tried pointing to my custom image but it sits there with a wait status and doesn't progress at all. Is this documentation up-to-date and complete, because it doesn't appear to be doing anything... can you let me know what logs etc I can supply to assist in getting this sorted? I need to be able to upgrade the cluster with custom images, so need to get this sorted if I can.

[mightymouse2045]

I have also taken a look at elemental toolkit, which seems to cater for additional distributions (ubuntu etc) which i see as a bit of a hybrid approach to what I am doing currently with elemental teal and customisations.

The elemental extension and tight integration into Rancher is definitely the way to go, but I'd like to ensure I'm investing time into something that is going to be maintained long term - is that the toolkit approach or teal? Or is teal essentially an officially maintained elemental toolkit derivative?

[mightymouse2045]

Moved this conversation to the slack channel - hopefully get a bit of help in there