Remove duplicate keys from multi-document YAML file? (Helm manifest)

[boekhold]

Hi,

I'm working with a massive Helm chart/subcharts, and some of the subcharts generate duplicate keys out of their templates.

Is there a way with yq to merge or remove duplicate keys from this file? E.g. something like:

help template my-release . | yq '' > all.yaml

(to show how "massive" this chart is, all.yaml is currently over 45K lines, and I've only enabled part of it).

The end objective is that I need to run "kustomize build ." on this output, which is currently failing due to the duplicate keys.

About the logic on which value to keep for duplicate keys, it doesn't really matter to me, but slightly prefer to keep the last occurrence in each document.

[boekhold]

The end objective is that I need to run "kustomize build ." on this output, which is currently failing due to the duplicate keys.

The more I think about it, the more I think that kustomize would not meet my needs anyways, and that I would be much better off using 'yq' as helm post renderer anyways.

Are there any resources/quick recipes available that show how to do the most common 'manifest manipulations' as a single yq command? Or would I need to pipe several yq commands together using shell pipes?

• select specific yaml document/Kubernetes resource
• update a key
• delete a key
• do a string substitution on the value for a specific key
• insert a key
• append to an array
• delete a specific entry in an array, with regex/substring match on the entry value
• update a specific entry in an array, with regex/substring match on the entry

It's late here, can't experiment now (I will tomorrow!) but can't get this out of my head, hence the post...

[boekhold]

My initial progress is as follows:

yq '
    select(.kind == "ConfigMap" and .metadata.name == "app1-config")
        .data."application.properties" |=
        sub("(db.catalog)=schema1", "$1=schema2")
    |
    select(.kind == "ConfigMap" and .metadata.name == "app?-config")
        .data."application.properties" |=
        sub("(long.option.name)=true", "$1=false")
' 

This seems to work as expected, and I am confident I can extend this to my other use-cases. I'm still wondering though if there is a better/more preferred approach?

Also: can I somehow include comments inside the expression, so I can document what each block is doing?

[mikefarah]

That looks good to me. I don't functionality for comments yet :(