Catalogd+Rukpak Maintenance Complexity Reduction: Unpacking related changes

[everettraven]

I've noticed there are a few issues related to making changes in the unpacking process:

• Update the image source to not be Kubernetes-centric  #673
• Explore alternative methods for unpacking catalog data catalogd#66
  • This is catalogd, but related to rukpak's unpacking process because there have been discussions around trying to reduce the maintenance complexity between catalogd and rukpak by getting them to share an unpacking process.
• Support out-of-tree provisioner implementations #425 (out of tree provisioners need to use an unpacker)
  • There is also some conversation around catalogd implementing an out of tree provisioner.

I wanted to use this discussion to come up with some ideas for how we can reduce the maintenance complexity across catalogd and rukpak by answering:

• How can/should catalogd be updated to reuse logic from rukpak for unpacking?
• What are some improvements we can make to rukpak's unpacking logic that results in an improvement for both rukpak and catalogd?

[everettraven]

How can/should catalogd be updated to reuse logic from rukpak for unpacking?

One thought that has been tossed around for a bit is that we can implement a Bundle "provisioner" in catalogd that is used for unpacking, storing, and serving catalog contents. This would allow us to keep the Catalog resource that delegates to the custom Bundle "provisioner" that reuses most of the logic from rukpak. With the way that the rukpak internal libraries are implemented there is still room for customization in this path as we could create custom implementations of specific interfaces as we deem necessary (for example, there is currently an effort in catalogd to implement the now externalized Storage interface from rukpak. This could be reused since Bundle provisioners use a Storage interface for storing Bundle contents.). A driver behind taking this approach is that rukpak and catalogd seem to have a large amount of similarities in how they need to unpack, store, and serve contents and a Bundle provisioner implementation has the benefit of being able to reuse most of the rukpak logic with room for customization as needed.

Another path we could take here is only exporting the source library from rukpak and importing and using it in catalogd. This would be a fairly straightforward drop-in replacement for catalogd as it currently uses a mostly copy-paste implementation for unpacking from rukpak.

What are some improvements we can make to rukpak's unpacking logic that results in an improvement for both rukpak and catalogd?

One of the main improvements that can be made here is as mentioned in #673

An additional potential optimization is to no longer always pass around an fs.FS in the unpacking result as this could be memory intensive

[joelanford]

There has also been some exploration of unpacking bundles via a CLI that is running on a user's workstation. I think my overall perspective is that we should refactor the APIs in rukpak's source/unpack package to remove rukpak and kubernetes specific assumptions.

That way we can re-use the unpack logic across a wider range of use cases:

• Rukpak
• Catalogd
• CLIs (e.g. opm, rukpakctl, others?)