Define Linux Network Devices #1271
Conversation
aojea
requested review from
AkihiroSuda,
crosbymichael,
cyphar,
dqminh,
giuseppe,
hqhq,
kolyshkin,
mrunalp,
thaJeztah,
tianon and
utam0k
as code owners
|
/assign @samuelkarp |
|
|
||
| **`netdevices`** (object, OPTIONAL) set of network devices that MUST be available in the container. The runtime MAY supply them however it likes. | ||
|
|
||
| The name of the network device is the entry key. |
There was a problem hiding this comment.
Does the map order matter? If so, implementation can be complicated for Go
There was a problem hiding this comment.
the linux kernel guarantees the uniqueness of the name in the runtime namespace, so a set is ok. Order is not important , each network device should be independent of each other ...
There was a problem hiding this comment.
Should we recommend a runtime performs a uniqueness check as well?
There was a problem hiding this comment.
Uniqueness inside container should be checked, e.g. that rename operation was successful
There was a problem hiding this comment.
added more text to clarify runtime checks and network devices lifecycle, PTAL
|
https://github.com/opencontainers/runtime-spec/blob/main/features.md should be updated too |
aojea
force-pushed
the
network-devices
branch
2 times, most recently
from
51e5104 to
3a666eb
Compare
updated and addressed the comments |
|
|
||
| **`netdevices`** (object, OPTIONAL) set of network devices that MUST be available in the container. The runtime MAY supply them however it likes. | ||
|
|
||
| The name of the network device is the entry key. |
There was a problem hiding this comment.
Should we recommend a runtime performs a uniqueness check as well?
|
AI @aojea (document the cleanup and destroy of the network interfaces) |
|
From the in-person discussion today:
|
config-linux.md
Outdated
|
|
||
| This schema focuses solely on moving existing network devices identified by name into the container namespace. It does not cover the complexities of network device creation or network configuration, such as IP address assignment, routing, and DNS setup. | ||
|
|
||
| **`netDevices`** (object, OPTIONAL) set of network devices that MUST be available in the container. The runtime is responsible for providing these devices; the underlying mechanism is implementation-defined. |
There was a problem hiding this comment.
This spec said "MUST" but, I think it can't do it in the rootless container because the rootless container doesn't have CAP_NET_ADMIN, right?
There was a problem hiding this comment.
I'm not sure we should take care of the rootless container.
There was a problem hiding this comment.
Could be an error in the case of a rootless container, if the runtime is not able to satisfy the MUST condition.
There was a problem hiding this comment.
Could be an error in the case of a rootless container, if the runtime is not able to satisfy the MUST condition.
+1 but It'd be better to clarify it in the spec.
There was a problem hiding this comment.
added mor explanations about runtime and network devices lifecycle and runtime checks, PTAL
Pushed a new commit addressing those comments, the changelog is
|
|
Changelog since the last review:
@akerouanton I succesfully implemented the new behavior of bringing back the interfaces, despite are virtual, in runc. Please see opencontainers/runc#4538 , that implements this spec probing its feasibility |
|
Kindly ping to reviewers, I think I addressed all the comments, also have an implementation on opencontainers/runc#4538 with tests to validate assumptions |
|
Can you please elaborate a bit more on why this is something that What are the pros and cons? What are the risks (especially to some of the more esoteric runtimes, like those implemented via VMs)? When should bundle authors use this new field instead of what they're doing currently (and how do they accurately determine if what they're already doing is something that can be converted to this new field losslessly)? |
|
Oops, some of what I'm after is in #1239, my apologies. That being said, I think the use cases for this are still pretty thin since it's just moves of pre-existing interfaces, and most container networking operations will be about new interfaces (veth at the very least). It feels inconsistent to include that bit of networking here when we don't have any other parts (nor plans to include them, because they're understandably complex essentially right away) -- for a higher-order runtime like Docker to take advantage of this, it'd have to split "network creation" processing code into separate paths based on what type of networking is requested (such as pre-creating veth devices before passing a bundle along so they can be moved by runc, which I think has some big rough edges too). |
From the Kubernetes side, we have a considerable number of workloads and demand, specially those related to AI/ML and Telco , that require ONLY the addition of existing network interfaces on the node to be moved to the containers ... existing implementation based on CNI need to rely on out-of-band mechanisms to compensate this problem with a lot of glue and red tape that makes these operations very brittle ... this problem can be easily solved with a declaratively way to indicate "please, move mlx0 to the container" ... that is this proposal
This proposal is about "network interface" as device interface not "networks" as "docker networks" or CNI, I've tried to make this distinction clear in several places to avoid conflating both problems that are completely different. For a high level runtime these should just be |
config-linux.md
Outdated
| * **`addresses`** *(array of strings, OPTIONAL)* - the IP addresses, IPv4 and or IPv6, of the device within the container in CIDR format (IP address / Prefix). All IPv4 addresses SHOULD be expressed in their decimal format, consisting of four decimal numbers separated by periods. Each number ranges from 0 to 255 and represents an octet of the address. IPv6 addresses SHOULD be represented in their canonical form as defined in RFC 5952. | ||
| The runtime MAY limit the number of addresses allowed. | ||
| The runtime MAY decide to revert back the original addreses. | ||
| * **`hardwareAddress`** *(string, OPTIONAL)* - represents the hardware address (e.g. MAC Address) of the device's network interface. |
There was a problem hiding this comment.
nit: the string representation of the MAC address should be clarified
There was a problem hiding this comment.
done, clarified to mach the golang ParseMac function https://pkg.go.dev/net#ParseMAC
There was a problem hiding this comment.
Sounds too specific to Go implementation.
Probably just need to support the single standard form (HEX:HEX:HEX:HEX:HEX:HEX)
There was a problem hiding this comment.
Probably just need to support the single standard form (HEX:HEX:HEX:HEX:HEX:HEX)
that is the IEEE 802 MAC-48
There was a problem hiding this comment.
I think golang implementation just got all the common formats https://networkengineering.stackexchange.com/a/82800
config-linux.md
Outdated
| The runtime MUST set the network device state to "up" after moving it to the network namespace to allow the container to send and receive network traffic through that device. | ||
|
|
||
| Notice that after deleting a network namespace, all its migratable network devices are moved to the default network namespace, virtual devices (veth, macvlan, ...) are destroyed. | ||
| The runtime MAY decide to move back or destroy the network device before the network namespace is deleted. If the network device is moved back, the runtime MUST set its state to "down" before moving it back to ensure that the interface is no longer active and won't interfere with other network operations or cause IP address conflicts. |
There was a problem hiding this comment.
Pretty much the same comment as above ^, why MAY decide to move back and not MUST?
There was a problem hiding this comment.
You are the experts in the runtime worlds so I will follow your advice, if you prefer and agree it should be MUST I don't have more argument than I'm suggesting MAY because I always prefer to be conservative when defining APIs and spec, I'm worried if some runtimes may not be able to participate in deletion or it can be very complex for them to do it, like I imaging can be the case for kata and gvisor. Being flexible allows them to implement the spec and be fully complaint instead of having to document it as limitation.
There was a problem hiding this comment.
Got it. Thanks.
Being flexible allows them to implement the spec and be fully complaint instead of having to document it as limitation.
I think either way they'd need to specify whether they support moving back ifaces into the original netns.
I'm not sure whether MUST is better than MAY here or not. If no else has anything to say, let's keep it as is.
There was a problem hiding this comment.
I think describing the current flow for intended use cases will help inform what we should do here. Are the interfaces that are moved into the containers reused or destroyed or both are valid cases?
We could have a cleanup policy field for each device if we need to support both.
There was a problem hiding this comment.
put it that way, I think that should be a MUST, since a virtual interface has to be created out of this flow, does not make sense to add the responsability to destroy it to the runtime ... it also makes more clear the spec, with @mrunalp questions I recognize there is a lot of ambiguity, and despite it can be more amenable for runtime implementers it sounds like it be more confusing for users.
I will push a change to make this a MUST
There was a problem hiding this comment.
@mrunalp @akerouanton added a last commit with MUST return back the interface in down state 47c4d3c , so it removes ambiguity and makes simpler to build automation over it,
- node has interface A or user creates interface A
- user tells runtime "run with --net-device A"
- user tells runtime "stop container"
- node has interface A in down state
then is up to the user to do what it wants with interface A, if already exist it can not do much, if the user created the interface then is on him to delete or reuse it
aojea
force-pushed
the
network-devices
branch
from
5ed1640 to
d8eb25f
Compare
|
@aojea Thanks for your hard work! It looks good to me. Also, I agree that it includes 1.3.0. |
|
|
||
| * **`name`** *(string, OPTIONAL)* - the name of the network device inside the container namespace. If not specified, the host name is used. The network device name is unique per network namespace, if an existing network device with the same name exists that rename operation will fail. The runtime MAY check that the name is unique before the rename operation. | ||
| The runtime MUST revert back the original name to guarantee the idempotence of operations, so a container that moves an interfaces and renames it can be created and destroyed multiple times with the same result. | ||
| * **`addresses`** *(array of strings, OPTIONAL)* - the IP addresses, IPv4 and or IPv6, of the device within the container in CIDR format (IP address / Prefix). All IPv4 addresses SHOULD be expressed in their decimal format, consisting of four decimal numbers separated by periods. Each number ranges from 0 to 255 and represents an octet of the address. IPv6 addresses SHOULD be represented in their canonical form as defined in RFC 5952. |
There was a problem hiding this comment.
Is the runtime expected to set this? It looks like it is. Let us say that in the spec.
There was a problem hiding this comment.
This is the input to the runtime, the runtime may choose how to set them meanwhile is consistent.
The context is that from kubernetes we got bitten by this, so is a recommendation because we find very hard to enforce this as input as it may break some clients , more context in https://daniel.haxx.se/blog/2021/04/19/curl-those-funny-ipv4-addresses/
aojea
force-pushed
the
network-devices
branch
from
47c4d3c to
80e758e
Compare
|
Changelog since last review, only the part that was not clear, to decide if runtime MUST or MAY bring back the interface to the host namespace, got updated to make it MUST , since is less ambiguous and covers all use cases, more on #1271 (comment) |
|
Please clean up the commits |
The proposed "netdevices" field provides a declarative way to specify which host network devices should be moved into a container's network namespace. This approach is similar than the existing "devices" field used for block devices but uses a dictionary keyed by the interface name instead. The proposed scheme is based on the existing representation of network device by the `struct net_device` https://docs.kernel.org/networking/netdevices.html. This proposal focuses solely on moving existing network devices into the container namespace. It does not cover the complexities of network configuration or network interface creation, emphasizing the separation of device management and network configuration. Signed-off-by: Antonio Ojea <[email protected]>
squashed |
The proposed "netdevices" field provides a declarative way to specify which host network devices should be moved into a container's network namespace.
This approach is similar than the existing "devices" field used for block devices but uses a dictionary keyed by the interface name instead.
The proposed scheme is based on the existing representation of network device by the
struct net_devicehttps://docs.kernel.org/networking/netdevices.html.
This proposal focuses solely on moving existing network devices into the container namespace. It does not cover the complexities of network configuration or network interface creation, emphasizing the separation of device management and network configuration.
Fixes: #1239