Created
June 17, 2024 15:19
-
-
Save abhilashshetty04/dcb73d32a8c6eecbe3160629b9be4f52 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Current behavior when we have unique vg name across nodes and no topology in sc. | |
| root@node-2-193524:~# vgs | |
| VG #PV #LV #SN Attr VSize VFree | |
| lvmvg3 1 0 0 wz--n- <10.00g <10.00g | |
| root@node-0-193524:~# vgs | |
| VG #PV #LV #SN Attr VSize VFree | |
| lvmvg1 1 0 0 wz--n- <10.00g <10.00g | |
| root@node-1-193524:~# vgs | |
| VG #PV #LV #SN Attr VSize VFree | |
| lvmvg 1 0 0 wz--n- <10.00g <10.00g | |
| sc: | |
| abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat sc-one-vg.yaml | |
| apiVersion: storage.k8s.io/v1 | |
| kind: StorageClass | |
| metadata: | |
| name: openebs-lvmpv-one | |
| parameters: | |
| storage: "lvm" | |
| volgroup: "lvmvg" | |
| provisioner: local.csi.openebs.io | |
| abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat sc-zero-vg.yaml | |
| apiVersion: storage.k8s.io/v1 | |
| kind: StorageClass | |
| metadata: | |
| name: openebs-lvmpv-zero //zero indicates that volgroup is in node-0 | |
| parameters: | |
| storage: "lvm" | |
| volgroup: "lvmvg1" | |
| provisioner: local.csi.openebs.io | |
| abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat sc-two-vg.yaml | |
| apiVersion: storage.k8s.io/v1 | |
| kind: StorageClass | |
| metadata: | |
| name: openebs-lvmpv-two | |
| parameters: | |
| storage: "lvm" | |
| volgroup: "lvmvg3" | |
| provisioner: local.csi.openebs.io | |
| pvc: | |
| abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat pvc.yaml | |
| kind: PersistentVolumeClaim | |
| apiVersion: v1 | |
| metadata: | |
| name: csi-lvmpv-one | |
| spec: | |
| storageClassName: openebs-lvmpv-one | |
| accessModes: | |
| - ReadWriteOnce | |
| resources: | |
| requests: | |
| storage: 4Gi | |
| abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat pvc-zero.yaml | |
| kind: PersistentVolumeClaim | |
| apiVersion: v1 | |
| metadata: | |
| name: csi-lvmpv-zero | |
| spec: | |
| storageClassName: openebs-lvmpv-zero | |
| accessModes: | |
| - ReadWriteOnce | |
| resources: | |
| requests: | |
| storage: 4Gi | |
| abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat pvc-two.yaml | |
| kind: PersistentVolumeClaim | |
| apiVersion: v1 | |
| metadata: | |
| name: csi-lvmpv-two | |
| spec: | |
| storageClassName: openebs-lvmpv-two | |
| accessModes: | |
| - ReadWriteOnce | |
| resources: | |
| requests: | |
| storage: 4Gi | |
| test1: | |
| All vg had 10G free space and pvc for all vg were requesting 4G volume. | |
| No matter in which order i create these PVC, provisioner was creating lvmvolume cr spcifying node which doesnt have the vg. | |
| Whats weird is it always selected the same node for a perticular pvc request each time. | |
| For example in below log | |
| I0614 06:19:13.058733 1 controller.go:314] scheduling the volume ^lvmvg$/pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f on node node-0-193524 | |
| Its selecting node-0 to provision pvc which has lvmvg in sc parameter. Whereas node-1 has the vg. For the other 2 volumes also it was selecting wrong nodes. | |
| controller log: | |
| I0614 06:19:13.045548 1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateVolume requests {"accessibility_requirements":{"preferred":[{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}},{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}},{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}}],"requisite":[{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}},{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}}]},"capacity_range":{"required_bytes":4294967296},"name":"pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f","parameters":{"csi.storage.k8s.io/pv/name":"pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f","csi.storage.k8s.io/pvc/name":"csi-lvmpv-one","csi.storage.k8s.io/pvc/namespace":"default","storage":"lvm","volgroup":"lvmvg"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]} | |
| I0614 06:19:13.058733 1 controller.go:314] scheduling the volume ^lvmvg$/pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f on node node-0-193524 | |
| I0614 06:19:13.067941 1 volume.go:94] provisioned volume pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f | |
| I0614 06:19:14.078140 1 volume.go:104] deprovisioned volume pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f | |
| E0614 06:19:15.083249 1 grpc.go:79] GRPC error: rpc error: code = ResourceExhausted desc = no vg available to serve volume request having regex="^lvmvg$" & capacity="4294967296" | |
| lvmcr: | |
| abhilashshetty@abhilashshetty-mbp1 mayamanifest % kubectl get lvmvolume -n openebs -oyaml | |
| apiVersion: v1 | |
| items: | |
| - apiVersion: local.openebs.io/v1alpha1 | |
| kind: LVMVolume | |
| metadata: | |
| creationTimestamp: "2024-06-14T06:14:58Z" | |
| generation: 2 | |
| labels: | |
| kubernetes.io/nodename: node-0-193524 | |
| name: pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f | |
| namespace: openebs | |
| resourceVersion: "450514" | |
| uid: 4574afc9-6b56-479c-884f-cc880aa5a730 | |
| spec: | |
| capacity: "4294967296" | |
| ownerNodeID: node-0-193524 | |
| shared: "no" | |
| thinProvision: "no" | |
| vgPattern: ^lvmvg$ | |
| volGroup: "" | |
| status: | |
| error: | |
| code: Internal | |
| message: no vg available to serve volume request having regex="^lvmvg$" & capacity="4294967296" | |
| state: Failed | |
| kind: List | |
| metadata: | |
| resourceVersion: "" | |
| Since we use spaceweight algorithm by default. Its supposed to prioritise node with higest free space vg for scheduling. So created some lv manually on 2 nodes to influence the scheduling. | |
| root@node-0-193524:~# vgs | |
| VG #PV #LV #SN Attr VSize VFree | |
| lvmvg1 1 0 0 wz--n- <10.00g <10.00g | |
| root@node-1-193524:~# vgs | |
| VG #PV #LV #SN Attr VSize VFree | |
| lvmvg 1 2 0 wz--n- <10.00g <3.00g | |
| root@node-2-193524:~# vgs | |
| VG #PV #LV #SN Attr VSize VFree | |
| lvmvg3 1 2 0 wz--n- <10.00g <3.00g | |
| applied pvc-zero.yaml which has lvmvg1 in volgroup. It still selected node-1-193524. Below are the logs. | |
| I0614 07:22:38.231973 1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateVolume requests {"accessibility_requirements":{"preferred":[{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}},{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}}],"requisite":[{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}},{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}}]},"capacity_range":{"required_bytes":4294967296},"name":"pvc-23b303b7-77f6-4d2c-b108-ee9d892135ff","parameters":{"csi.storage.k8s.io/pv/name":"pvc-23b303b7-77f6-4d2c-b108-ee9d892135ff","csi.storage.k8s.io/pvc/name":"csi-lvmpv-zero","csi.storage.k8s.io/pvc/namespace":"default","storage":"lvm","volgroup":"lvmvg1"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]} | |
| I0614 07:22:38.245137 1 controller.go:314] scheduling the volume ^lvmvg1$/pvc-23b303b7-77f6-4d2c-b108-ee9d892135ff on node node-1-193524 | |
| I0614 07:22:38.250035 1 volume.go:94] provisioned volume pvc-23b303b7-77f6-4d2c-b108-ee9d892135ff | |
| I0614 07:22:39.260035 1 volume.go:104] deprovisioned volume pvc-23b303b7-77f6-4d2c-b108-ee9d892135ff | |
| E0614 07:22:40.266879 1 grpc.go:79] GRPC error: rpc error: code = ResourceExhausted desc = no vg available to serve volume request having regex="^lvmvg1$" & capacity="4294967296" | |
| Since its always picking node-1 for lvmvg1 pvc. Then created lvmvg1 there with less capacity. | |
| root@node-1-193524:~# vgs | |
| VG #PV #LV #SN Attr VSize VFree | |
| lvmvg 1 2 0 wz--n- <10.00g <3.00g | |
| lvmvg1 1 0 0 wz--n- <1024.00g <1024.00g | |
| root@node-0-193524:~# vgs | |
| VG #PV #LV #SN Attr VSize VFree | |
| lvmvg1 1 0 0 wz--n- <10.00g <10.00g | |
| root@node-2-193524:~# vgs | |
| VG #PV #LV #SN Attr VSize VFree | |
| lvmvg3 1 2 0 wz--n- <10.00g <3.00g | |
| Its picked node-2. Still not picking node-1. | |
| created lvmvg1 on node-1 with less size. | |
| root@node-2-193524:~# vgs | |
| VG #PV #LV #SN Attr VSize VFree | |
| lvmvg1 1 1 0 wz--n- <1024.00g <1020.00g | |
| lvmvg3 1 2 0 wz--n- <10.00g <3.00g | |
| Then the provisioning works and picks lvmvg1. This has become a supported scenaio according to current behavior. All node has the vg and picks node which has higest free space. | |
| I0614 08:31:21.544627 1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateVolume requests {"accessibility_requirements":{"preferred":[{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}},{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}}],"requisite":[{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}},{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}}]},"capacity_range":{"required_bytes":4294967296},"name":"pvc-f32be8f7-dd35-4382-baf3-21bb6008ba24","parameters":{"csi.storage.k8s.io/pv/name":"pvc-f32be8f7-dd35-4382-baf3-21bb6008ba24","csi.storage.k8s.io/pvc/name":"csi-lvmpv-zero","csi.storage.k8s.io/pvc/namespace":"default","storage":"lvm","volgroup":"lvmvg1"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]} | |
| I0614 08:31:21.556272 1 controller.go:314] scheduling the volume ^lvmvg1$/pvc-f32be8f7-dd35-4382-baf3-21bb6008ba24 on node node-2-193524 | |
| I0614 08:31:21.559589 1 volume.go:94] provisioned volume pvc-f32be8f7-dd35-4382-baf3-21bb6008ba24 | |
| I0614 08:31:22.571777 1 volume.go:104] deprovisioned volume pvc-f32be8f7-dd35-4382-baf3-21bb6008ba24 | |
| E0614 08:31:23.579056 1 grpc.go:79] GRPC error: rpc error: code = ResourceExhausted desc = no vg available to serve volume request having regex="^lvmvg1$" & capacity="4294967296" | |
| Non existing vg | |
| abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat sc-non.yaml | |
| apiVersion: storage.k8s.io/v1 | |
| kind: StorageClass | |
| metadata: | |
| name: openebs-lvmpv-non | |
| parameters: | |
| storage: "lvm" | |
| volgroup: "lvmvgn" | |
| provisioner: local.csi.openebs.io | |
| abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat pvc-non.yaml | |
| kind: PersistentVolumeClaim | |
| apiVersion: v1 | |
| metadata: | |
| name: csi-lvmpv-non | |
| spec: | |
| storageClassName: openebs-lvmpv-non | |
| accessModes: | |
| - ReadWriteOnce | |
| resources: | |
| requests: | |
| storage: 1Gi | |
| This vg is not present in any node. It picked node-o randomly. | |
| I0614 08:47:21.303187 1 controller.go:314] scheduling the volume ^lvmvgn$/pvc-92710b9c-e095-4f72-9d42-bbaa88427bcd on node node-0-193524 | |
| Provisioning failed as expected. Seeing the failure provisioner will delete the cr. When the create volume came again vg was created on the node it previously tried. | |
| root@node-0-193524:~# vgs | |
| VG #PV #LV #SN Attr VSize VFree | |
| lvmvg1 1 0 0 wz--n- <10.00g <10.00g | |
| lvmvgn 1 0 0 wz--n- <2.00t <2.00t | |
| I0614 08:53:49.362168 1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateVolume requests {"accessibility_requirements":{"preferred":[{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}},{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}}],"requisite":[{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}},{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}}]},"capacity_range":{"required_bytes":1073741824},"name":"pvc-92710b9c-e095-4f72-9d42-bbaa88427bcd","parameters":{"csi.storage.k8s.io/pv/name":"pvc-92710b9c-e095-4f72-9d42-bbaa88427bcd","csi.storage.k8s.io/pvc/name":"csi-lvmpv-non","csi.storage.k8s.io/pvc/namespace":"default","storage":"lvm","volgroup":"lvmvgn"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]} | |
| I0614 08:53:49.373913 1 controller.go:314] scheduling the volume ^lvmvgn$/pvc-92710b9c-e095-4f72-9d42-bbaa88427bcd on node node-1-193524 | |
| I0614 08:53:49.378807 1 volume.go:94] provisioned volume pvc-92710b9c-e095-4f72-9d42-bbaa88427bcd | |
| I0614 08:53:50.391486 1 volume.go:104] deprovisioned volume pvc-92710b9c-e095-4f72-9d42-bbaa88427bcd | |
| E0614 08:53:51.405182 1 grpc.go:79] GRPC error: rpc error: code = ResourceExhausted desc = no vg available to serve volume request having regex="^lvmvgn$" & capacity="1073741824" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment