Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content

[thibveni]

What steps did you take and what happened:
I have a schedule backup manifest which every night do a backup of my apps.
From time to time i have a warning during night backups, (once every two weeks i would say)

Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-742508d0-6ed9-4fe3-82fd-4e98af246ccf: "snapshot controller failed to update snapcontent-742508d0-6ed9-4fe3-82fd-4e98af246ccf on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-742508d0-6ed9-4fe3-82fd-4e98af246ccf\": the object has been modified; please apply your changes to the latest version and try again"

Looking in my logs i can retrieve the warning

time="2023-12-10T03:50:44Z" level=warning msg="VolumeSnapshot has a temporary error Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation on the content snapcontent-742508d0-6ed9-4fe3-82fd-4e98af246ccf: \"snapshot controller failed to update snapcontent-742508d0-6ed9-4fe3-82fd-4e98af246ccf on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \\\"snapcontent-742508d0-6ed9-4fe3-82fd-4e98af246ccf\\\": the object has been modified; please apply your changes to the latest version and try again\". Snapshot controller will retry later." cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:250" pluginName=velero-plugin-for-csi
The warning will appear like 20 times within 8 minutes

When looking at the details of my backup i can see it is in state PartiallyFailed,
But without touching anything, on the next morning the backup is completely done perfectly, and snapcontent is updated without an error.

What did you expect to happen:
I expect it to correctly update the snapcontent annotation,

The following information will help us better understand what's going on:
Full logs in this gist: https://gist.github.com/thibveni/15e0e585952dc52f4df295f1f43f8ad9
including

• deployment of velero
• schedule manifest
• backup describe
• backup logs

Anything else you would like to add:
I have this warning since i installed velero 9 months ago and i updated to latest velero version,

Environment:

• Velero version (use velero version):
• Client:
  Version: v1.12.0
  Git commit: -
  Server:
  Version: v1.12.2
• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:57:26Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v4.5.7
  Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:14:49Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
  Scaleway provider
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[kaovilai]

I expect it to correctly update the snapcontent annotation,

It in this case is not Velero, but https://github.com/kubernetes-csi/external-snapshotter/

Please track resolution via kubernetes-csi/external-snapshotter#748

Unless there's a common workaround you'd like built into velero, the underlying issue itself should be tracked elsewhere.

[thibveni]

Thanks a lot for this fast response, indeed i will track resolution in the links you have me.