Unable to add gke cluster with dns-endpoint enabled

[eranreshef-starkware]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I am trying to add a gke cluster to my argocd instance and it fails with
FATA[0005] rpc error: code = Unauthenticated desc = the server has asked for the client to provide credentials
The gke cluster is configure to work with dns-based-control-plane-access (see this and this for details)

To Reproduce

1. Create a gke cluster with --enable-dns-access
2. Try to add this new cluster to an existing remote argocd instance with argocd cluster add ...

Expected behavior

Argocd is expected to successfully communicate and manage the target cluster

Screenshots

None

Version

argocd: v2.4.11+3d9e9f2
  BuildDate: 2022-08-22T09:35:38Z
  GitCommit: 3d9e9f2f95b7801b90377ecfc4073e5f0f07205b
  GitTreeState: clean
  GoVersion: go1.18.5
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.9.6+ba62a0a

Logs

Client log:

$ argocd cluster add gcp-dns-testing --yes --grpc-web
INFO[0001] ServiceAccount "argocd-manager" already exists in namespace "kube-system" 
INFO[0002] ClusterRole "argocd-manager-role" updated    
INFO[0003] ClusterRoleBinding "argocd-manager-role-binding" updated 
FATA[0006] rpc error: code = Unauthenticated desc = the server has asked for the client to provide credentials 

Server Log:

argocd-server-c44fdc568-n8kjh server time="2024-12-26T15:26:00Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = the server has asked for the client to provide credentials" grpc.code=Unauthenticated grpc.method=Create grpc.service=cluster.ClusterService grpc.start_time="2024-12-26T15:25:59Z" grpc.time_ms=1109.873 span.kind=server system=grpc