RWO Volume request failed

[kotreshhr]

[vagrant@kube1 ~]$ kubectl describe pvc glusterblock-20
Name:          glusterblock-20
Namespace:     default
StorageClass:  glustervirtblock-csi
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: org.gluster.glustervirtblock
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Events:
  Type       Reason                Age                From                                                                                                  Message
  ----       ------                ----               ----                                                                                                  -------
  Normal     Provisioning          11s (x2 over 53s)  org.gluster.glustervirtblock_csi-glustervirtblock-provisioner-0_48a9096c-2e93-11e9-a99b-0a580ae94107  External provisioner is provisioning volume for claim "default/glusterblock-20"
  Warning    ProvisioningFailed    6s (x2 over 26s)   org.gluster.glustervirtblock_csi-glustervirtblock-provisioner-0_48a9096c-2e93-11e9-a99b-0a580ae94107  failed to provision volume with StorageClass "glustervirtblock-csi": rpc error: code = Internal desc = failed to create block volume: pvc-96704fe9-2e9f-11e9-ab2f-525400995719 err: could not obtain lock: another conflicting transaction may be in progress
  Normal     ExternalProvisioning  5s (x8 over 65s)   persistentvolume-controller                                                                           waiting for a volume to be created, either by external provisioner "org.gluster.glustervirtblock" or manually created by system administrator
Mounted By:  gluster-2

[vagrant@kube1 ~]$ kubectl get pvc 
NAME              STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
glusterblock-1    Bound     pvc-95a5746a-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   88s
glusterblock-10   Bound     pvc-95deb06a-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   87s
glusterblock-11   Bound     pvc-95ecb8ac-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   87s
glusterblock-12   Bound     pvc-95f6a68b-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   87s
glusterblock-13   Bound     pvc-96180f68-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   87s
glusterblock-14   Bound     pvc-962202dc-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   87s
glusterblock-15   Bound     pvc-962f2d00-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   87s
glusterblock-16   Bound     pvc-9637c45d-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   87s
glusterblock-17   Bound     pvc-964fb24a-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   87s
glusterblock-18   Bound     pvc-966014a7-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   87s
glusterblock-19   Bound     pvc-96656bc6-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   87s
glusterblock-2    Bound     pvc-95aba858-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   88s
glusterblock-20   Pending                                                                        glustervirtblock-csi   86s
glusterblock-21   Bound     pvc-967af625-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   86s
glusterblock-22   Bound     pvc-9684f208-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   86s
glusterblock-23   Bound     pvc-968f0725-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   86s
glusterblock-24   Pending                                                                        glustervirtblock-csi   86s
glusterblock-3    Bound     pvc-95b05185-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   88s
glusterblock-4    Bound     pvc-95b459c9-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   88s
glusterblock-5    Bound     pvc-95bbbe36-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   88s
glusterblock-6    Bound     pvc-95c31353-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   88s
glusterblock-7    Bound     pvc-95c8095f-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   88s
glusterblock-8    Bound     pvc-95d11727-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   88s
glusterblock-9    Bound     pvc-95d462e9-2e9f-11e9-ab2f-525400995719   100Mi      RWO            glustervirtblock-csi   88s
[vagrant@kube1 ~]$ 

[kotreshhr]

Glusterd2 logs from the node.

time="2019-02-12 08:20:49.381749" level=info msg="10.233.65.7 - - [12/Feb/2019:08:20:49 +0000] \"GET /v1/peers HTTP/1.1\" 200 812" reqid=1b2aad2e-8832-402d-ba38-65e103e4368c
time="2019-02-12 08:20:50.794884" level=info msg="client connected" address="10.233.66.13:49151" server=sunrpc source="[server.go:148:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2019-02-12 08:20:50.833154" level=info msg="client connected" address="10.233.66.13:1023" server=sunrpc source="[server.go:148:sunrpc.(*SunRPC).acceptLoop]" transport=tcp
time="2019-02-12 08:20:50.834550" level=info msg="client disconnected" address="10.233.66.13:1023" server=sunrpc source="[server.go:109:sunrpc.(*SunRPC).pruneConn]"
time="2019-02-12 08:21:19.686324" level=info msg="10.233.66.1 - - [12/Feb/2019:08:21:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=c8b6d0e3-bc00-4505-a238-cb287842707b
time="2019-02-12 08:22:19.685364" level=info msg="10.233.66.1 - - [12/Feb/2019:08:22:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=0e4c2478-f1bd-4606-bd5c-aae38bd6bbff
time="2019-02-12 08:23:19.685525" level=info msg="10.233.66.1 - - [12/Feb/2019:08:23:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=775dbb45-df76-495f-bdc7-c0c688d3a080
time="2019-02-12 08:23:53.008514" level=info msg="10.233.65.7 - - [12/Feb/2019:08:23:52 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-1125e02f-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 204 0" reqid=f9f3c2cd-9b93-4308-a502-e58a84eba7f7
time="2019-02-12 08:23:53.176259" level=info msg="10.233.65.7 - - [12/Feb/2019:08:23:52 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-112b6aad-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 204 0" reqid=fb185961-72b3-4b4c-8828-11526cde54ab
time="2019-02-12 08:23:54.100340" level=info msg="10.233.65.7 - - [12/Feb/2019:08:23:54 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-11583441-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=16324faf-5d22-43c2-b779-7488c26a6ec2
time="2019-02-12 08:23:54.140917" level=info msg="10.233.65.7 - - [12/Feb/2019:08:23:53 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-1141c3d8-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 204 0" reqid=cc2c830c-6eac-4ab9-b19b-0323296b1a57
time="2019-02-12 08:24:03.124718" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:02 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-1181dd86-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 204 0" reqid=95d28272-8e98-4d44-9526-822088d11f4d
time="2019-02-12 08:24:03.406890" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:02 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-11b0f5da-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 204 0" reqid=df1d6326-ef20-44bc-9bda-ed3e35f610d2
time="2019-02-12 08:24:03.535776" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:03 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-11ebc52e-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=3fcd115b-d806-447a-b4f6-c1b300458fb0
time="2019-02-12 08:24:04.647908" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:03 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-11c11bf7-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 204 0" reqid=6d009f9d-848d-4222-8769-5a789223f428
time="2019-02-12 08:24:08.641742" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:08 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-114d88b3-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=ad62b11a-830a-41d9-9bcc-84d81de6f740
time="2019-02-12 08:24:18.640362" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:18 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-11ebc52e-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=275e0c15-bc41-4e40-882a-e2188b78be4a
time="2019-02-12 08:24:19.686095" level=info msg="10.233.66.1 - - [12/Feb/2019:08:24:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=a0765567-7877-4368-959e-128908542306
time="2019-02-12 08:24:20.213604" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:16 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=dc41a749-db21-4db6-a6f8-9154952bcc45
time="2019-02-12 08:24:21.300029" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:19 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=35200ef0-d8bb-45ed-b0f4-1d67998e9b86
time="2019-02-12 08:24:22.313401" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:20 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=ed40dd32-a2e1-4ba6-861e-4b3281704d6a
time="2019-02-12 08:24:23.479884" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:21 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=1898b399-d284-4d4e-a433-90df2f899d40
time="2019-02-12 08:24:24.005789" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:22 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=aec69878-a68f-479d-83a2-1519afaf901a
time="2019-02-12 08:24:25.362623" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:25 +0000] \"GET /v1/peers HTTP/1.1\" 200 812" reqid=e3c6ef2b-6750-4b5b-985a-0165f0fa2713
time="2019-02-12 08:24:26.047034" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:23 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=27891918-b0b9-4835-ba54-3b13d294e84f
time="2019-02-12 08:24:26.236353" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:24 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=cca5d5d7-56c1-4258-abed-94025ea2f89b
time="2019-02-12 08:24:26.369764" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:26 +0000] \"GET /v1/peers HTTP/1.1\" 200 812" reqid=a76de0f8-ad18-4ec5-910b-817d1e983122
time="2019-02-12 08:24:28.743582" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:27 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=6c734a35-aebf-42c1-8c29-256ea9ba09ae
time="2019-02-12 08:24:28.748878" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:28 +0000] \"GET /v1/peers HTTP/1.1\" 200 812" reqid=cf65a502-f05f-475c-832d-de47937574d4
time="2019-02-12 08:24:29.150398" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:29 +0000] \"GET /v1/peers HTTP/1.1\" 200 812" reqid=14c16d6a-0a3c-4a68-8231-297106ea76af
time="2019-02-12 08:24:35.280816" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:30 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=e85ddda3-bd43-4789-8739-eceb17b9b4e2
time="2019-02-12 08:24:38.258086" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:35 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 201 167" reqid=a3c57e7b-5a9f-44a0-8bb7-3d9407e98826
time="2019-02-12 08:24:38.283329" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:38 +0000] \"GET /v1/peers HTTP/1.1\" 200 812" reqid=a8f6fe97-ecfa-4bc4-9f34-9bc87042813d
time="2019-02-12 08:24:48.739426" level=info msg="10.233.65.7 - - [12/Feb/2019:08:24:48 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-11ebc52e-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=5c4baa30-6efb-4563-94ff-8e4c58d5e652
time="2019-02-12 08:25:16.821361" level=error msg="error in acquiring cluster lock" error="could not obtain lock: another conflicting transaction may be in progress" source="[volume.go:94:utils.ResizeBlockHostingVolume]"
time="2019-02-12 08:25:16.825462" level=error msg="failed in updating hostvolume _block-hosting-available-size metadata" block_name=pvc-96704fe9-2e9f-11e9-ab2f-525400995719 error="could not obtain lock: another conflicting transaction may be in progress" hostvol=block_hosting_volume_c0e4d154-0f93-41d7-8408-32e1e5ff61ba requested_block_size=104857600 source="[gluster_virtblock.go:110:gluster-virtblock.(*GlusterVirtBlk).CreateBlockVolume]"
time="2019-02-12 08:25:16.825636" level=info msg="10.233.65.7 - - [12/Feb/2019:08:25:11 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 500 110" reqid=efc63e35-21c9-49a7-a1ff-47a8493558ec
time="2019-02-12 08:25:19.687135" level=info msg="10.233.66.1 - - [12/Feb/2019:08:25:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=dde1646a-aa47-4aed-8e35-e754c92d351f
time="2019-02-12 08:25:44.537604" level=info msg="10.233.65.7 - - [12/Feb/2019:08:25:44 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-114d88b3-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=700c518c-afa1-4ff3-9599-81f26e738a84
time="2019-02-12 08:26:19.687209" level=info msg="10.233.66.1 - - [12/Feb/2019:08:26:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=c840a6c6-6997-4526-85f7-8e454609b837
time="2019-02-12 08:26:51.733639" level=error msg="error in acquiring cluster lock" error="could not obtain lock: another conflicting transaction may be in progress" source="[volume.go:94:utils.ResizeBlockHostingVolume]"
time="2019-02-12 08:26:51.733703" level=error msg="failed in updating hostvolume _block-hosting-available-size metadata" block_name=pvc-96a4dd44-2e9f-11e9-ab2f-525400995719 error="could not obtain lock: another conflicting transaction may be in progress" hostvol=block_hosting_volume_c0e4d154-0f93-41d7-8408-32e1e5ff61ba requested_block_size=104857600 source="[gluster_virtblock.go:110:gluster-virtblock.(*GlusterVirtBlk).CreateBlockVolume]"
time="2019-02-12 08:26:51.733758" level=info msg="10.233.65.7 - - [12/Feb/2019:08:26:46 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 500 110" reqid=4554eac7-7aab-46dd-9d47-d1e3739dd5f9
time="2019-02-12 08:27:19.687883" level=info msg="10.233.66.1 - - [12/Feb/2019:08:27:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=ea07874f-63d5-469d-9cd5-fd6c067a5e85
time="2019-02-12 08:27:44.577533" level=info msg="10.233.65.7 - - [12/Feb/2019:08:27:44 +0000] \"DELETE /v1/blockvolumes/virtblock/pvc-114d88b3-2e9f-11e9-ab2f-525400995719 HTTP/1.1\" 500 59" reqid=881eedda-f39b-49ac-97f0-7454baa39dd4
time="2019-02-12 08:28:19.685340" level=info msg="10.233.66.1 - - [12/Feb/2019:08:28:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=7e3dc719-53f5-4070-913a-3dd36211b835
time="2019-02-12 08:29:03.178720" level=error msg="error in acquiring cluster lock" error="could not obtain lock: another conflicting transaction may be in progress" source="[volume.go:94:utils.ResizeBlockHostingVolume]"
time="2019-02-12 08:29:03.178919" level=error msg="failed in updating hostvolume _block-hosting-available-size metadata" block_name=pvc-96704fe9-2e9f-11e9-ab2f-525400995719 error="could not obtain lock: another conflicting transaction may be in progress" hostvol=block_hosting_volume_c0e4d154-0f93-41d7-8408-32e1e5ff61ba requested_block_size=104857600 source="[gluster_virtblock.go:110:gluster-virtblock.(*GlusterVirtBlk).CreateBlockVolume]"
time="2019-02-12 08:29:03.179350" level=info msg="10.233.65.7 - - [12/Feb/2019:08:28:57 +0000] \"POST /v1/blockvolumes/virtblock HTTP/1.1\" 500 110" reqid=a3dbf8ca-29c4-42c6-b7cc-30b6b2d72001
time="2019-02-12 08:29:19.687312" level=info msg="10.233.66.1 - - [12/Feb/2019:08:29:19 +0000] \"GET /ping HTTP/1.1\" 200 0" reqid=52f2e59c-731a-47f1-ae53-d30cb498d8fe

[amarts]

Is it the same on latest container? Did we get any further ?