spike: optimize OS upgrades bandwidth

[mudler]

Problem

Nodes in a edge cluster might be having poor networking capabilities. Kairos users could create custom images which are way more bigger (e.g. by bringing many kernel drivers, etc) and they do not want necessarly split the image to handle upgrade of extensions separately.

one of the current issue is that, on upgrade each node of the cluster would have to re-download the image entirely from scratch before applying the upgrade.

This card is to explore alternatives

Ideas

• Have the agent downloading the image and serving it for the other nodes of the network (that can be used with a suc plan, or without) in a transparent manner (no kubernetes setup required)
• Have a local service to pull upgrade images and serve it as at local OCI registry. The local OCI registry would have to distribute the image evenly to the nodes in a transparent manner
• Have a cache for the cluster for pulling images remotely (that would also work with OS images)
• Use something like https://github.com/spegel-org/spegel - and have it installed by default in our images, and make it of use when calling the suc plan
• Use something like https://github.com/uber/kraken or https://github.com/dragonflyoss/dragonfly

[mudler]

e.g. k3s does support already spegel: https://docs.k3s.io/installation/registry-mirror?_highlight=spegel

[phillebaba]

If you need any help integrating Spegel let me know. Seeing as Kairos already has a p2p component we could find a way to integrate it instead of using the built in p2p solution in Spegel.

[mudler]

If you need any help integrating Spegel let me know. Seeing as Kairos already has a p2p component we could find a way to integrate it instead of using the built in p2p solution in Spegel.

That would be really cool. One of the major points would be to support Spegel ootb for other providers too in a way or another.

Currently Kairos does have a default provider which supports k3s, but ideally we would like to bake that in to a lower level (we ship an agent with all the images) so it is working with other kubernetes distro (and also without kubernetes at all).

I'm still having a closer look to Spegel to see any matching points, but would be nice if we could sync so we would speed up integration quite faster!

[mudler]

After discussing with @jimmykarily we think that would be a good idea to investigate if we can support pulling from a standalone spegel service directly from the kairos-agent whenever pulling an image with containerd, even better if we could somehow consume spegel as a library.

context: kairos-agent consumes the containerd library directly to pull images during upgrades.

This PR: kairos-io/provider-kairos#675 enables spegel at k3s level, which covers also our upgrade controller use-cases, however, it would be good to extend support whenver calling kairos-agent with a specific OCI source also inside a suc script