As part of #6149 and the fix in #6152, I spent a lot of time looking at the zone-setup binary and the related zone-network-setup SMF service. This code, specifically the zone-setup common-networking subcommand, runs in all the zones at startup, and does a few things:

• Creates an IP interface on its data link (the link is a VNIC over the underlay etherstub)
• Sets the MTU for both IPv4 and IPv6 on the interface
• If the tool is provided with a non-localhost IPv6 address, i.e., an address on the underlay, it creates the necessary IP addresses (link-local and static IPv6)
• Creates a default route to the provided gateway (the sled-agent's underlay IP address)
• Populates /etc/hosts with a few convenience addresses

This works for most zones because we know all their networking information when they start. For example, for CockroachDB zones, the sled-agent does not launch the zone until we already have the static underlay address that it's server needs to listen on.

The switch zone is different. It's required for bootstrapping the rest of the control plane. In this case, the zone-setup service is provided with only the static address ::1, and so cannot create the underlay IP address inside the zone. In fact, it currently requires that the sled-agent zlogin to the zone later to add that underlay IP address in order to complete -- the route add command will retry until that's done, because it will fail since the sled-agent's underlay IP isn't on a directly-connected network (i.e., the same sled `/64).

So in the case of the switch zone, there's an inherent overlap in the responsibilities of the zone-network-setup service and the sled-agent. It would be nice to sort these out a bit more, and in the spirit of the big thread of work to make the zones self-assembling, completely remove the need for the sled-agent to zlogin into the zone to create the underlay address.

I'm sure there are many ways to do this. One approach would be to have a different zone-setup subcommand that runs in the switch zone, which is (more) persistent rather than one-shot. For example, instead of logging into the zone, the sled-agent could add the underlay address to the SMF properties and then refresh the service. The zone-setup binary could watch for this refresh, reread its SMF properties, and ensure that whatever new values it gets are applied. Note that the sled agent currently adds the same default route as the zone-setup binary -- this would be removed in this proposal, since the zone-setup binary's attempt to add a default route cannot work until it has that address anyway.