Fails to create envoy container with nerdctl <- docker symlink

[kriswuollett]

Due to #130 I just symlinked the binary to see if it work, but container creation fails.

Logs:

I0912 21:03:53.930481  117970 event.go:389] "Event occurred" object="projectcontour/envoy" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message=<
	Error syncing load balancer: failed to ensure load balancer: failed to create continers kindccm-GZ4YR5X5RDVG26ZXXYAU2GKKAVOAEUPJFJGPZD72 [--detach --tty --label io.x-k8s.cloud-provider-kind.cluster=nyc3-shared --label io.x-k8s.cloud-provider-kind.loadbalancer.name=nyc3-shared/projectcontour/envoy --net kind --init=false --hostname kindccm-GZ4YR5X5RDVG26ZXXYAU2GKKAVOAEUPJFJGPZD72 --privileged --restart=on-failure --sysctl=net.ipv4.ip_forward=1 --sysctl=net.ipv6.conf.all.disable_ipv6=0 --sysctl=net.ipv6.conf.all.forwarding=1 --sysctl=net.ipv4.conf.all.rp_filter=0 --publish=10000/TCP --publish-all envoyproxy/envoy:v1.30.1 bash -c echo -en 'node:
	  cluster: cloud-provider-kind
	  id: cloud-provider-kind-id
	
	dynamic_resources:
	  cds_config:
	    resource_api_version: V3
	    path: /home/envoy/cds.yaml
	  lds_config:
	    resource_api_version: V3
	    path: /home/envoy/lds.yaml
	
	admin:
	  access_log_path: /dev/stdout
	  address:
	    socket_address:
	      address: 0.0.0.0
	      port_value: 10000
	' > /home/envoy/envoy.yaml && touch /home/envoy/cds.yaml && touch /home/envoy/lds.yaml && while true; do envoy -c /home/envoy/envoy.yaml && break; sleep 1; done]: exit status 1
 >

Environment:

$ uname -a
Linux REDACTED 6.1.0-25-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.106-3 (2024-08-26) x86_64 GNU/Linux
$ kind --version
kind version 0.24.0
$ nerdctl version
Client:
 Version:	v1.7.7
 OS/Arch:	linux/amd64
 Git commit:	5882c720f4e7f358fb26b759e514b3ae9dd8ea83
 buildctl:
  Version:	v0.15.2
  GitCommit:	9e14164a1099d3e41b58fc879cbdd6f2b2edb04e

Server:
 containerd:
  Version:	v1.7.22
  GitCommit:	7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:	1.1.14
  GitCommit:	v1.1.14-0-g2c9f5602

nerdctl events:

# nerdctl events
2024-09-12 21:10:49.854188261 +0000 UTC default /tasks/exec-added {"container_id":"0c1ccb1a9660af3f7ba2ca979d2b5ac031fc28badabcbcfe4efb03e7d127c43c","exec_id":"exec-7ee143597330237894d7d3cf4712d262be0f22e5f198f5b3993021d268683491"}
2024-09-12 21:10:49.92051739 +0000 UTC default /tasks/exec-started {"container_id":"0c1ccb1a9660af3f7ba2ca979d2b5ac031fc28badabcbcfe4efb03e7d127c43c","exec_id":"exec-7ee143597330237894d7d3cf4712d262be0f22e5f198f5b3993021d268683491","pid":119353}
2024-09-12 21:10:49.920830243 +0000 UTC default /tasks/exit {"container_id":"0c1ccb1a9660af3f7ba2ca979d2b5ac031fc28badabcbcfe4efb03e7d127c43c","id":"exec-7ee143597330237894d7d3cf4712d262be0f22e5f198f5b3993021d268683491","pid":119353,"exited_at":{"seconds":1726175449,"nanos":920404251}}

[aojea]

is nerdctl not fully compatible? I'm not very familiar with this alternative

[kriswuollett]

is nerdctl not fully compatible? I'm not very familiar with this alternative

I'd assume it should be at least so far as command/API arguments as containerd/nerdctl claims to be:

• nerdctl is a Docker-compatible CLI for containerd.
• Same UI/UX as docker

I don't know enough regarding implementing a load balancer controller in Kubernetes if it would require anything Docker-proprietary networking-wise. I'd assume not.

[aojea]

I don't know enough regarding implementing a load balancer controller in Kubernetes if it would require anything Docker-proprietary networking-wise. I'd assume not.

my statement may be misleading, nerdctl, podman or docker does not need to implement any loadbalancer capabilit, they just run containers ... the commands and the reports of those commands are the things that need to be compatible, as you see in other issue you have reported the problem is that some containers return more than two ips ... we can fix that here, but until now docker and podman always reported one IP per IP family per container

[BenTheElder]

I'd assume it should be at least so far as command/API arguments as containerd/nerdctl claims to be:

While nerdctl has shown willingness to consider differences a bug, they still have to release a fix for them if/when it happens.

In my experience, docker's behavior is complex and "drop in replacements" quickly stop being drop-in compatible when you're doing tricky, barely supported things like kubernetes-in-a-container. We maintain specific interface implementations in kind for nerdctl, podman, and docker because of those small differences.

We should detect if it's actually docker or not, kind does this for podman/docker already with symlinks. We sniff the output of docker version which is not the same for symlinked replacements (unless nerdctl has started faking that).

[BenTheElder]

We should detect "this is actually nerdctl" and use the nerdctl specific paths. nerdctl is really close to compatible but not 100% compatible, and we'll want to be able to mitigate anything that turns up (plus it's not that bad to maintain two copies of the bits that talk to docker or nerdctl respectively, we do this in kubernetes-sigs/kind as well).

[aojea]

Detecting of nerdctl will be added in #175 but I still can't see why it fails to create the container in the logs