Get error when creating volume snapshot

[igorgonibm]

When creating snapshot I get the following messages in csi-snapshotter logs:
Full log, snap content describe and relevant yamls attached

I0421 09:23:09.254741 1 snapshot_controller.go:606] setAnnVolumeSnapshotBeingCreated: set annotation [snapshot.storage.kubernetes.io/volumesnapshot-being-created:yes] on content [snapcontent-3f3c2bc8-2305-4b26-95f6-0e641e8e3762].
I0421 09:23:09.259302 1 snapshot_controller.go:179] updateContentStatusWithEvent[snapcontent-3f3c2bc8-2305-4b26-95f6-0e641e8e3762]
I0421 09:23:09.263085 1 snapshot_controller.go:200] updating VolumeSnapshotContent[snapcontent-3f3c2bc8-2305-4b26-95f6-0e641e8e3762] error status failed Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io "snapcontent-3f3c2bc8-2305-4b26-95f6-0e641e8e3762": the object has been modified; please apply your changes to the latest version and try again
E0421 09:23:09.263110 1 snapshot_controller.go:139] createSnapshot [create-snapcontent-3f3c2bc8-2305-4b26-95f6-0e641e8e3762]: error occurred in createSnapshotWrapper: failed to add VolumeSnapshotBeingCreated annotation on the content snapcontent-3f3c2bc8-2305-4b26-95f6-0e641e8e3762: "snapshot controller failed to update snapcontent-3f3c2bc8-2305-4b26-95f6-0e641e8e3762 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io "snapcontent-3f3c2bc8-2305-4b26-95f6-0e641e8e3762": the object has been modified; please apply your changes to the latest version and try again"
E0421 09:23:09.263208 1 goroutinemap.go:150] Operation for "create-snapcontent-3f3c2bc8-2305-4b26-95f6-0e641e8e3762" failed. No retries permitted until 2020-04-21 09:23:09.763165038 +0000 UTC m=+6.819361135 (durationBeforeRetry 500ms). Error: "failed to add VolumeSnapshotBeingCreated annotation on the content snapcontent-3f3c2bc8-2305-4b26-95f6-0e641e8e3762: "snapshot controller failed to update snapcontent-3f3c2bc8-2305-4b26-95f6-0e641e8e3762 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io \"snapcontent-3f3c2bc8-2305-4b26-95f6-0e641e8e3762\": the object has been modified; please apply your changes to the latest version and try again""
yamls_and_log.zip

[xing-yang]

I see the following error message in your logs. It looks like you have not created the secret but you have that secret referenced in the snapshot class.

E0421 09:29:03.988544       1 snapshot_controller.go:535] Failed to get credentials for snapshot snapcontent-c62126d0-15af-4c45-a950-99e056a12604: error getting secret secret-67a in namespace kube-system: secrets "secret-67a" not found

[igorgonibm]

@xing-yang I used a wrong secret at first but then fixed to use the right one.
This error happened after the fix.
(As I specified snapshot was created on storage. If secret was wrong it would not happen)

[igorgonibm]

@xing-yang I forget to mention that I am in team which develop CSI driver and I am adding snapshots support. Any idea what could be wrong? Its really blocking us prom proceeding. Thank you!

[xing-yang]

@igorgonibm what image versions of snapshot-controller and csi-snapshotter sidecar are you using?

Can you provide "kubectl describe volumesnapshot"? Those error messages you posted are failures updating objects in the API server but the retries were successful later as VolumeSnapshotContent has its status updated and ReadyToUse is set true. I don't know what is the status of VolumeSnapshot. Can you set log level to 5 and reproduce the logs? I can't find any messages on updating the VoumeSnapshot status in the existing logs.

[igorgonibm]

I created new snapshot and attached relevant yamls. describe of class, snapshot and snapshotcomntent and snapshotter log
Log level is set to 5
thank you
logs.zip

[xing-yang]

According to "kubectl describe volumesnapshot", the snapshot is created successfully.

Name:         snap20
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  snapshot.storage.k8s.io/v1beta1
Kind:         VolumeSnapshot
Metadata:
  Creation Timestamp:  2020-04-22T14:07:52Z
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  Generation:        1
  Resource Version:  2052306
  Self Link:         /apis/snapshot.storage.k8s.io/v1beta1/namespaces/default/volumesnapshots/snap20
  UID:               b52df980-4158-4af1-b0ad-bffa7d1a2937
Spec:
  Source:
    Persistent Volume Claim Name:  pvc-snapshot
  Volume Snapshot Class Name:      snapclass
Status:
  Bound Volume Snapshot Content Name:  snapcontent-b52df980-4158-4af1-b0ad-bffa7d1a2937
  Creation Time:                       2020-04-22T14:08:03Z
  Ready To Use:                        true
  Restore Size:                        1Gi
Events:                                <none>

Ready To Use is true. Creation time, restore size, and bound volume snapshot content name are all set in the VolumeSnapshot status.

[igorgonibm]

@xing-yang so why is there error message in the log? It is related to the current snap content.

E0422 14:08:03.234487 1 snapshot_controller_base.go:250] could not sync content "snapcontent-b52df980-4158-4af1-b0ad-bffa7d1a2937": failed to remove VolumeSnapshotBeingCreated annotation from the content snapcontent-b52df980-4158-4af1-b0ad-bffa7d1a2937: "snapshot controller failed to update snapcontent-b52df980-4158-4af1-b0ad-bffa7d1a2937 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io "snapcontent-b52df980-4158-4af1-b0ad-bffa7d1a2937": the object has been modified; please apply your changes to the latest version and try again"

[xing-yang]

This is because VolumeSnapshotBeingCreated annotation was already removed earlier. This syncContent call has an older version of the object so it failed to remove it again, but it is actually already removed.

I'll take a look to see how to avoid this error.

[xing-yang]

/assign @xing-yang

[igorgonibm]

This is because VolumeSnapshotBeingCreated annotation was already removed earlier. This syncContent call has an older version of the object so it failed to remove it again, but it is actually already removed.

I'll take a look to see how to avoid this error.

Thank you. Will wait for your answer.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten