Azure workload identity needs to be setup in podLabels which isn't currently possible.

[malnor]

In order to use Azure Workload identity, I have to be able to set a label on the pod (azure.workload.identity/use: "true") but currently I'm only able to set annotations on the pods, so I cannot use the helm chart, but have to setup the manifests manually.

[isindir]

@malnor , I added ability to specify podLabels as part of #191 , will merge shortly

[isindir]

merged, please let me know if that fixes the issue

[malnor]

@isindir YES! That does fix the issue! Now I'm able to decrypt using Azure Workload identity. Thanks you 🙏

no I only have the problem that the actual secrets are not produced, have no idea why.

[malnor]

@isindir well, I might have spoken to early, it does solve one problem (that the token isn't mounted), but I still have issues with the decryption. I'll be back once I know what is going on.

[malnor]

@isindir Okay, so there is another part to this puzzle. In order to azure to be able to authorize the Workload Identity it is expecting a specific Service Account name. So In order to fix this I would also need to actually set the ServiceAccountName in the deployment.

This is the part of the error that has this clue: "AADSTS700213: No matching federated identity record found for presented assertion subject 'system:serviceaccount:sops:sops-sops-secrets-operator'"

Full error:

Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  URL: https://xxx.vault.azure.net/keys/sops-key/yyy: FAILED
    - failed to decrypt SOPS data key with Azure Key Vault key
    - URL: 'https://kv-aks-dev-k8s-base-swec.vault.azure.net/keys/sops-key/yyy'
    - DefaultAzureCredential authentication failed:
      - Token acquisition failed.
      - Attempted credentials:
        - EnvironmentCredential: incomplete environment variable configuration. Only AZURE_TENANT_ID and AZURE_CLIENT_ID are set.
        - WorkloadIdentityCredential: authentication failed.
      - FromAssertion() error:
        - HTTP call to https://login.microsoftonline.com/13625fe7-3c99-4410-bb37-a862f17daee4/oauth2/v2.0/token (POST)
        - Error: reply status code was 401:
          {
            \"error\": \"invalid_client\",
            \"error_description\": \"AADSTS700213: No matching federated identity record found for presented assertion subject 'system:serviceaccount:sops:sops-sops-secrets-operator'.\"
            \"Please note that the matching is done using a case-sensitive comparison. Check your federated identity credential Subject, Audience, and Issuer against the presented assertion.\"
            For more information, visit: https://learn.microsoft.com/entra/workload-id/workload-identity-federation
          }
        - Trace ID: 569a3605-7fdd-446f-b843-fc7352dc7500
        - Correlation ID: 683b9a26-1cb4-48b1-8e62-134dc4ad7362
        - Timestamp: 2024-11-04 14:18:59Z"

[isindir]

I have used it with azure fairly long time ago, but from this documentation:

• https://github.com/isindir/sops-secrets-operator?tab=readme-ov-file#azure
  one need to set tenantId, clientId and clientSecret .

From error message I catch following:

• Token acquisition failed.
• Only AZURE_TENANT_ID and AZURE_CLIENT_ID are set.
• No matching federated identity record found for presented assertion subject 'system:serviceaccount:sops:sops-sops-secrets-operator'

Later suggests that some cloud configuration bit is missing for Kubernetes service account - it is not authorised to fetch token. in AWS EKS on service account there should be set an annotation like eks.amazonaws.com/role-arn and this role should be configured properly on the AWS side as well. It feels like similar should be done in Azure AKS. For me, the error does not suggest that k8s Service Account name is at fault.

please check also https://learn.microsoft.com/en-gb/entra/workload-id/workload-identity-federation

[malnor]

Yeah, Azure is fun... What you’re seeing is DefaultAzureCredential encountering issues because it tries multiple authentication methods in a specific order, and if none are correctly configured, it doesn’t recognize any of them as valid and fails - that is why the error is confusing. Here’s a quick overview of the three main authentication methods it can use:

3 methods for authentication:

Service Principal

Which requires environment variables AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_CLIENT_SECRET. This is the only one of the three methods that requires a password, with AZURE_CLIENT_SECRET serving as that credential. Service Principal credentials typically have an expiration date, so at some point, the password will stop working unless it’s rotated, which can make this method inconvenient for long-term production use. It’s useful in development or scenarios where secret management is feasible, but I generally avoid it in production if I can use a password-free option.

In your scenario, this will work if all the variables are set, so this is supported already.

Workload Identity (Pod-specific authentication through OIDC)

This approach doesn’t need a password but does require a Service Account and specific annotations. The Service Account should include the annotations azure.workload.identity/client-id set to the Azure AD app’s client ID and azure.workload.identity/tenant-id for the tenant. Additionally, a pod label azure.workload.identity/use: "true". Workload Identity requires that the AKS cluster be set up to use OIDC, and it allows each pod to authenticate directly with Azure resources using its own identity, making it both secure and password-free. The Service Account has in my case already been created for the namespace (through terraform). Note also that the name of the service account can be important, to just setting the labels on "your" Service Account will not fly - at least not in my case.

In your scenario, this is partly required as the pod label now is allowed, but it also requires that the service account can be pointed to instead of created.

Managed Identity (k8s Node-based authentication)

also doesn’t require a password and allows access to Azure resources at the node level. For System-Assigned Managed Identity, it’s enabled by default on AKS nodes, so no additional pod configuration is needed. For User-Assigned Managed Identity, you’ll need to create the identity in Azure, assign it the necessary permissions, and then add the annotation azure.identity.client-id=<user-assigned-managed-identity-client-id> to the pod. Managed Identity provides broad access across the node and can be effective for scenarios where multiple pods on a node require access to Azure resources, though it may be more permissive than necessary if the access needs to be limited to specific pods.

In your scenario, this should also work as the annotations for the pods was already there.

In this case, it might make sense to use a Managed identity, but I since I don't own this cluster, I cannot set it up. So I need the workload identity. Also, if I do template out the resources and add them instead of using the charts and use the existing service account it does work.

[isindir]

Ok, so the requirement is :

• ability to use custom service account, externally created (with terraform in your case)
• or alternatively ability to set annotations (already supported by chart) and specify custom name for the service account

[malnor]

Yes,

Workload, needs pod labels (which you fixed earlier) and to set the service account (which is lacking today).

The other should be possible with what is there today!

[isindir]

@malnor please see #192 , do you think this would fix it for you and or do you spot any problems with this approach ?

[isindir]

@malnor would you mind adding a subsection into readme describing the setup you are using on Azure once it succeeds ? Or you could create new readme file in /docs and link it from top level readme ?

I merged #192

[malnor]

@isindir the deployment needs to point to the ServiceAccountName of my choosing, not sure it did? Doing too many things at the same time :-( Sorry about that.

[isindir]

I think I covered that already: https://github.com/isindir/sops-secrets-operator/pull/192/files#diff-db6b1ba4bcca23297327a6daa538c0fd9a8261ba3ca11b7dce5a749c8b9d866eR32

what you need to do is disable sa and specify name of your own sa via values see: serviceAccount section

[malnor]

Perfect! Then it should work. :-)

[malnor]

It works now - Thank you for your swift response 🙏