Appproj annotation not modified in `argocd proj create --update -f my-proj-file.yaml

[eranreshef-starkware]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I've created a project using the following command:

argocd proj create -f my-proj.yaml

Where my-proj.yaml includes annotations for slack notifications subscriptions.
A few moments later I wanted to update the project deceleration both in spec content but also the name of the target slack channels names and I ran the following command:

argocd proj create --upsert -f my-proj.yaml

The command exited successfully but I noticed that slack notifications are still arriving to the old channel. When I examined the project's live object I've noticed that the spec content has changed as expected but the annotations for the notifications subscriptions didn't change at all.

To Reproduce

1. Create a project spec yaml which includes annotations for notifications subscriptions.
2. Apply the yaml with argocd cli
3. Edit the notifications subscriptions annotations (change channel name for example)
4. Re-apply the yaml with argocd cli and the --upsert flag

Expected behavior

The live project manifest will include the annotations according to the new version.

Screenshots

N/A

Version

argocd: v2.4.11+3d9e9f2
  BuildDate: 2022-08-22T09:35:38Z
  GitCommit: 3d9e9f2f95b7801b90377ecfc4073e5f0f07205b
  GitTreeState: clean
  GoVersion: go1.18.5
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.9.3+6eba5be

Logs

N/A