Created
June 17, 2024 15:19
-
-
Save abhilashshetty04/dcb73d32a8c6eecbe3160629b9be4f52 to your computer and use it in GitHub Desktop.
Revisions
-
abhilashshetty04 created this gist
Jun 17, 2024 .There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,251 @@ Current behavior when we have unique vg name across nodes and no topology in sc. root@node-2-193524:~# vgs VG #PV #LV #SN Attr VSize VFree lvmvg3 1 0 0 wz--n- <10.00g <10.00g root@node-0-193524:~# vgs VG #PV #LV #SN Attr VSize VFree lvmvg1 1 0 0 wz--n- <10.00g <10.00g root@node-1-193524:~# vgs VG #PV #LV #SN Attr VSize VFree lvmvg 1 0 0 wz--n- <10.00g <10.00g sc: abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat sc-one-vg.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: openebs-lvmpv-one parameters: storage: "lvm" volgroup: "lvmvg" provisioner: local.csi.openebs.io abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat sc-zero-vg.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: openebs-lvmpv-zero //zero indicates that volgroup is in node-0 parameters: storage: "lvm" volgroup: "lvmvg1" provisioner: local.csi.openebs.io abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat sc-two-vg.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: openebs-lvmpv-two parameters: storage: "lvm" volgroup: "lvmvg3" provisioner: local.csi.openebs.io pvc: abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat pvc.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: csi-lvmpv-one spec: storageClassName: openebs-lvmpv-one accessModes: - ReadWriteOnce resources: requests: storage: 4Gi abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat pvc-zero.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: csi-lvmpv-zero spec: storageClassName: openebs-lvmpv-zero accessModes: - ReadWriteOnce resources: requests: storage: 4Gi abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat pvc-two.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: csi-lvmpv-two spec: storageClassName: openebs-lvmpv-two accessModes: - ReadWriteOnce resources: requests: storage: 4Gi test1: All vg had 10G free space and pvc for all vg were requesting 4G volume. No matter in which order i create these PVC, provisioner was creating lvmvolume cr spcifying node which doesnt have the vg. Whats weird is it always selected the same node for a perticular pvc request each time. For example in below log I0614 06:19:13.058733 1 controller.go:314] scheduling the volume ^lvmvg$/pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f on node node-0-193524 Its selecting node-0 to provision pvc which has lvmvg in sc parameter. Whereas node-1 has the vg. For the other 2 volumes also it was selecting wrong nodes. controller log: I0614 06:19:13.045548 1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateVolume requests {"accessibility_requirements":{"preferred":[{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}},{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}},{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}}],"requisite":[{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}},{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}}]},"capacity_range":{"required_bytes":4294967296},"name":"pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f","parameters":{"csi.storage.k8s.io/pv/name":"pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f","csi.storage.k8s.io/pvc/name":"csi-lvmpv-one","csi.storage.k8s.io/pvc/namespace":"default","storage":"lvm","volgroup":"lvmvg"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]} I0614 06:19:13.058733 1 controller.go:314] scheduling the volume ^lvmvg$/pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f on node node-0-193524 I0614 06:19:13.067941 1 volume.go:94] provisioned volume pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f I0614 06:19:14.078140 1 volume.go:104] deprovisioned volume pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f E0614 06:19:15.083249 1 grpc.go:79] GRPC error: rpc error: code = ResourceExhausted desc = no vg available to serve volume request having regex="^lvmvg$" & capacity="4294967296" lvmcr: abhilashshetty@abhilashshetty-mbp1 mayamanifest % kubectl get lvmvolume -n openebs -oyaml apiVersion: v1 items: - apiVersion: local.openebs.io/v1alpha1 kind: LVMVolume metadata: creationTimestamp: "2024-06-14T06:14:58Z" generation: 2 labels: kubernetes.io/nodename: node-0-193524 name: pvc-684bdf13-b8a8-488e-84b3-cf8b611f0c7f namespace: openebs resourceVersion: "450514" uid: 4574afc9-6b56-479c-884f-cc880aa5a730 spec: capacity: "4294967296" ownerNodeID: node-0-193524 shared: "no" thinProvision: "no" vgPattern: ^lvmvg$ volGroup: "" status: error: code: Internal message: no vg available to serve volume request having regex="^lvmvg$" & capacity="4294967296" state: Failed kind: List metadata: resourceVersion: "" Since we use spaceweight algorithm by default. Its supposed to prioritise node with higest free space vg for scheduling. So created some lv manually on 2 nodes to influence the scheduling. root@node-0-193524:~# vgs VG #PV #LV #SN Attr VSize VFree lvmvg1 1 0 0 wz--n- <10.00g <10.00g root@node-1-193524:~# vgs VG #PV #LV #SN Attr VSize VFree lvmvg 1 2 0 wz--n- <10.00g <3.00g root@node-2-193524:~# vgs VG #PV #LV #SN Attr VSize VFree lvmvg3 1 2 0 wz--n- <10.00g <3.00g applied pvc-zero.yaml which has lvmvg1 in volgroup. It still selected node-1-193524. Below are the logs. I0614 07:22:38.231973 1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateVolume requests {"accessibility_requirements":{"preferred":[{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}},{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}}],"requisite":[{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}},{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}}]},"capacity_range":{"required_bytes":4294967296},"name":"pvc-23b303b7-77f6-4d2c-b108-ee9d892135ff","parameters":{"csi.storage.k8s.io/pv/name":"pvc-23b303b7-77f6-4d2c-b108-ee9d892135ff","csi.storage.k8s.io/pvc/name":"csi-lvmpv-zero","csi.storage.k8s.io/pvc/namespace":"default","storage":"lvm","volgroup":"lvmvg1"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]} I0614 07:22:38.245137 1 controller.go:314] scheduling the volume ^lvmvg1$/pvc-23b303b7-77f6-4d2c-b108-ee9d892135ff on node node-1-193524 I0614 07:22:38.250035 1 volume.go:94] provisioned volume pvc-23b303b7-77f6-4d2c-b108-ee9d892135ff I0614 07:22:39.260035 1 volume.go:104] deprovisioned volume pvc-23b303b7-77f6-4d2c-b108-ee9d892135ff E0614 07:22:40.266879 1 grpc.go:79] GRPC error: rpc error: code = ResourceExhausted desc = no vg available to serve volume request having regex="^lvmvg1$" & capacity="4294967296" Since its always picking node-1 for lvmvg1 pvc. Then created lvmvg1 there with less capacity. root@node-1-193524:~# vgs VG #PV #LV #SN Attr VSize VFree lvmvg 1 2 0 wz--n- <10.00g <3.00g lvmvg1 1 0 0 wz--n- <1024.00g <1024.00g root@node-0-193524:~# vgs VG #PV #LV #SN Attr VSize VFree lvmvg1 1 0 0 wz--n- <10.00g <10.00g root@node-2-193524:~# vgs VG #PV #LV #SN Attr VSize VFree lvmvg3 1 2 0 wz--n- <10.00g <3.00g Its picked node-2. Still not picking node-1. created lvmvg1 on node-1 with less size. root@node-2-193524:~# vgs VG #PV #LV #SN Attr VSize VFree lvmvg1 1 1 0 wz--n- <1024.00g <1020.00g lvmvg3 1 2 0 wz--n- <10.00g <3.00g Then the provisioning works and picks lvmvg1. This has become a supported scenaio according to current behavior. All node has the vg and picks node which has higest free space. I0614 08:31:21.544627 1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateVolume requests {"accessibility_requirements":{"preferred":[{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}},{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}}],"requisite":[{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}},{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}}]},"capacity_range":{"required_bytes":4294967296},"name":"pvc-f32be8f7-dd35-4382-baf3-21bb6008ba24","parameters":{"csi.storage.k8s.io/pv/name":"pvc-f32be8f7-dd35-4382-baf3-21bb6008ba24","csi.storage.k8s.io/pvc/name":"csi-lvmpv-zero","csi.storage.k8s.io/pvc/namespace":"default","storage":"lvm","volgroup":"lvmvg1"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]} I0614 08:31:21.556272 1 controller.go:314] scheduling the volume ^lvmvg1$/pvc-f32be8f7-dd35-4382-baf3-21bb6008ba24 on node node-2-193524 I0614 08:31:21.559589 1 volume.go:94] provisioned volume pvc-f32be8f7-dd35-4382-baf3-21bb6008ba24 I0614 08:31:22.571777 1 volume.go:104] deprovisioned volume pvc-f32be8f7-dd35-4382-baf3-21bb6008ba24 E0614 08:31:23.579056 1 grpc.go:79] GRPC error: rpc error: code = ResourceExhausted desc = no vg available to serve volume request having regex="^lvmvg1$" & capacity="4294967296" Non existing vg abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat sc-non.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: openebs-lvmpv-non parameters: storage: "lvm" volgroup: "lvmvgn" provisioner: local.csi.openebs.io abhilashshetty@abhilashshetty-mbp1 lvmmanifest % cat pvc-non.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: csi-lvmpv-non spec: storageClassName: openebs-lvmpv-non accessModes: - ReadWriteOnce resources: requests: storage: 1Gi This vg is not present in any node. It picked node-o randomly. I0614 08:47:21.303187 1 controller.go:314] scheduling the volume ^lvmvgn$/pvc-92710b9c-e095-4f72-9d42-bbaa88427bcd on node node-0-193524 Provisioning failed as expected. Seeing the failure provisioner will delete the cr. When the create volume came again vg was created on the node it previously tried. root@node-0-193524:~# vgs VG #PV #LV #SN Attr VSize VFree lvmvg1 1 0 0 wz--n- <10.00g <10.00g lvmvgn 1 0 0 wz--n- <2.00t <2.00t I0614 08:53:49.362168 1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateVolume requests {"accessibility_requirements":{"preferred":[{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}},{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}}],"requisite":[{"segments":{"kubernetes.io/hostname":"node-1-193524","openebs.io/nodename":"node-1-193524"}},{"segments":{"kubernetes.io/hostname":"node-2-193524","openebs.io/nodename":"node-2-193524"}},{"segments":{"kubernetes.io/hostname":"node-0-193524","openebs.io/nodename":"node-0-193524"}}]},"capacity_range":{"required_bytes":1073741824},"name":"pvc-92710b9c-e095-4f72-9d42-bbaa88427bcd","parameters":{"csi.storage.k8s.io/pv/name":"pvc-92710b9c-e095-4f72-9d42-bbaa88427bcd","csi.storage.k8s.io/pvc/name":"csi-lvmpv-non","csi.storage.k8s.io/pvc/namespace":"default","storage":"lvm","volgroup":"lvmvgn"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]} I0614 08:53:49.373913 1 controller.go:314] scheduling the volume ^lvmvgn$/pvc-92710b9c-e095-4f72-9d42-bbaa88427bcd on node node-1-193524 I0614 08:53:49.378807 1 volume.go:94] provisioned volume pvc-92710b9c-e095-4f72-9d42-bbaa88427bcd I0614 08:53:50.391486 1 volume.go:104] deprovisioned volume pvc-92710b9c-e095-4f72-9d42-bbaa88427bcd E0614 08:53:51.405182 1 grpc.go:79] GRPC error: rpc error: code = ResourceExhausted desc = no vg available to serve volume request having regex="^lvmvgn$" & capacity="1073741824"