Support Docker Desktop for Linux (access from the host to the containers VM)

[AyushGlitch]

OS: Ubuntu 22
Processor: Inter i5 1135g7

Ubuntu Version

Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.4 LTS
Release:	22.04
Codename:	jammy

Kind Version
kind v0.27.0-alpha+3ab1dab1c81267 go1.23.4 linux/amd64

Docker Version
Docker version 27.0.3, build 7d4bcd8

I went through the following steps and the cloud-provider-kind provided an external IP Addr for the load balancer but wasn't able to access the load balancer with it.

Creating Kind Cluster

clustersConfig.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
  - role: worker
  - role: worker

$ kind create cluster --config clustersConfig.yaml

Creating Load Balancer Deployment

loadBalancerDeply.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zeroapp
spec:
  selector:
    matchLabels:
      app: zeroapp
  replicas: 2
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: zeroapp
    spec:
      terminationGracePeriodSeconds: 30
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - zeroapp
            topologyKey: kubernetes.io/hostname
      containers:
      - name: zeroapp
        image: registry.k8s.io/e2e-test-images/agnhost:2.39
        args:
          - netexec
          - --http-port=80
          - --delay-shutdown=30
        ports:
          - name: httpd
            containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: zeroapp
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app: zeroapp
  ports:
    - name: httpd
      port: 80
      targetPort: 80

$ kubectl apply -f loadBalancerDeply.yaml

Allowing load balancers access to control plane nodes

$ kubectl label node kind-control-plane node.kubernetes.io/exclude-from-external-load-balancers-

Staring cloud-provider-kind

$ cloud-provider-kind

External IP provided

Access Load Balancer (Not Working)

$ curl 172.22.0.5:80/hostname

Able to curl when exec into the kindccm-* container

Did I miss any of the steps ?
If not then, is there any setting that has to be changed or any permission that should be provided ?

[TrooperT]

I have couple of initial questions/thoughts:

• Is Docker installed via apt or snap
• I see externalTrafficPolicy: Local on service
  • Try changing test externalTrafficPolicy: Cluster as I recall attempting something similar and would basically get round robin timeouts where my LBs would work sporadically as the CCM configures the envoyproxy LoadBalancer containers with a list of ALL nodes as valid backends, even those not actively participating in the service
  • Can try backing this up with reviewing logs of the KINDCCM-XXXXX container and comparing connection attempts against kubectl to see what pod is on what node

[AyushGlitch]

• Docker Desktop was installed .deb file, it comes with both Docker Engine and Docker Desktop.
• I am using the same deployment as cloud-provider-kind/examples/loadbalancer_deployment.yaml in the github repo hence i don't think that it would matter if we change externalTrafficPolicy: Local to Cluster. Anyway I tried with externalTrafficPolicy: Cluster, but it resulted in the same issue. Also, in Kubecon EU 2024 Antonio told that we need to have externalTrafficPolicy: Local rather than Cluster.
• I don't know what to look for in the logs of KINDCCM-XXX container. Following is screenshot of logs:
  

[aojea]

Interesting, it seems that docker desktop runs in a VM https://docs.docker.com/desktop/troubleshoot-and-support/faqs/linuxfaqs/#why-does-docker-desktop-for-linux-run-a-vm so we need to identify that is docker desktop and create the tunnel in the host, otherwise you should only be able to access the portmaps on the host , you can set the flag --enable-lb-port-mapping=true so the envoy container will publish the ports on the hosts

Can you give me some way to identify docker desktop from the host, the output of docker info maybe?

[TrooperT]

Ah! Docker desktop strikes again. I actually ran into the exact issue as @AyushGlitch during my setup for cloud-provider-kind albeit on a MacOS device running Docker Desktop. I ended up utilizing https://github.com/chipmk/docker-mac-net-connect to provide a Layer 3 connection from the host machine into the Docker VM. It sets up and manages a wireguard tunnel with IP routing of the VM's docker networks. It would be a bit kludgy but I posit we could do something similar in cloud-provider-kind in instances where it detects docker desktop.

@aojea if you'd like i can provide docker info from my MacOS machine tomorrow once in office so you have another data point

[aojea]

@aojea if you'd like i can provide docker info from my MacOS machine tomorrow once in office so you have another data point

we can detect mac from the GOOS environment variables, the problem is that when we are in linux seems we need to discriminate now between "normal linux" where containers are routable and "docker desktop linux" where containers are behind a VM and are not reachable

[AyushGlitch]

Can you give me some way to identify docker desktop from the host, the output of docker info maybe?

docker info with the above configs and --enable-lb-port-mapping="true" for cloud-provider-kind

Client: Docker Engine - Community
 Version:    27.0.3
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0-desktop.1
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0-desktop.2
    Path:     /usr/lib/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.29
    Path:     /usr/lib/docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /usr/lib/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     /usr/lib/docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /usr/lib/docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     /usr/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/lib/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.8.0
    Path:     /usr/lib/docker/cli-plugins/docker-scout

Server:
 Containers: 5
  Running: 4
  Paused: 0
  Stopped: 1
 Images: 6
 Server Version: 26.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.26-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.578GiB
 Name: docker-desktop
 ID: 31330744-e699-41be-98a8-4ddfc3f3614c
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///home/ayush/.docker/desktop/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

I tried the same steps but after switching the docker context from desktop-linux which is for Docker Desktopto default and it ran perfectly

But the one of the problem is that we can't see/access these containers in Docker Desktop

docker info for default context

Client: Docker Engine - Community
 Version:    27.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0-desktop.1
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0-desktop.2
    Path:     /usr/lib/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.29
    Path:     /usr/lib/docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /usr/lib/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     /usr/lib/docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /usr/lib/docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     /usr/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/lib/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.8.0
    Path:     /usr/lib/docker/cli-plugins/docker-scout

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 27.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-49-generic
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.36GiB
 Name: ayush-NKi511TL165S
 ID: 680321fb-8423-4365-9857-c6495a3cc473
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[aojea]

I can't find a consistent way to figure out if is Desktop Linux https://www.docker.com/blog/how-to-check-docker-version/

[TrooperT]

Consistent across all OSes or within Linux?

Can docker info be run either by exec or through the docker go packages and parsed for the Server Version with a switch?

[aojea]

Linux only that is the only platform that has "native" docker and desktop mode.

Can docker info be run either by exec

Yeah, we already do that in kind, but base on the output I only see a reference in the Operating System field and I'm not sure if this is a reliable API

Operating System: Docker Desktop

[TrooperT]

Reading through their docs a bit and the page you linked would lead me to believe that they know in a stable fashion what version you are running on the server version string: "Server: Docker Engine - Community" vs "Server: Docker Desktop 4.31.0 (153195)" for example

https://www.docker.com/blog/how-to-check-docker-version/#highlighter_867586

I'd imagine that'd be stable enough to key off of

[aojea]

I'd imagine that'd be stable enough to key off of

That's what I thought but I don't see that in your output pasted