Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

following the docs results in "FailedScheduling: 0/1 nodes are available: 1 node(s) did not have enough free storage." #349

Open
condaatje opened this issue Nov 25, 2024 · 1 comment
Open

following the docs results in "FailedScheduling: 0/1 nodes are available: 1 node(s) did not have enough free storage." #349

condaatje opened this issue Nov 25, 2024 · 1 comment

Comments

@condaatje
Copy link

condaatje commented Nov 25, 2024 •
edited
Loading

following the docs here: https://openebs.io/docs/user-guides/local-storage-user-guide/local-pv-lvm/lvm-installation#prerequisites

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.]

to reproduce (microk8s for easy testing on fresh ec2 instance, for example):

sudo snap install microk8s --classic --channel=1.31

truncate -s 5G /tmp/disk.img
sudo losetup -f /tmp/disk.img --show

# use the name of the disk from that last command and put it here:
sudo pvcreate /dev/loop<?>
sudo vgcreate lvmvg /dev/loop<?>       ## here lvmvg is the volume group name to be created

microk8s helm repo add openebs https://openebs.github.io/openebs
microk8s helm install openebs --namespace openebs openebs/openebs --set engines.replicated.mayastor.enabled=false --set engines.local.zfs.enabled=false --create-namespace

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-lvm
provisioner: local.csi.openebs.io
parameters:
  storage: "lvm"
  volgroup: "lvmvg"
volumeBindingMode: WaitForFirstConsumer     ## note: when this is replaced with "Immediate" it creates a pv successfully, but the pod still can't use it for some reason

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: lvm-local-pvc
spec:
  storageClassName: openebs-lvmpv
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1G
    limits:
      storage: 1G

apiVersion: v1
kind: Pod
metadata:
  name: lvm-task-pv-pod-2
spec:
  volumes:
    - name: lvm-task-pv-storage
      persistentVolumeClaim:
        claimName: lvm-local-pvc
  containers:
    - name: task-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/hypermount"
          name: lvm-task-pv-storage

in the pod events:

Warning FailedScheduling 99s default-scheduler 0/1 nodes are available: 1 node(s) did not have enough free storage. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

What did you expect to happen:

normal pod bootup

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other Pastebin is fine.)

  • kubectl logs -f openebs-lvm-localpv-controller-7b6d6b4665-fk78q -n openebs -c openebs-lvm-plugin
ubuntu@ip-172-31-23-102:~$ microk8s kubectl logs -f openebs-lvm-localpv-controller-84844f9c47-4k6pf -n openebs -c openebs-lvm-plugin
I1125 20:26:42.788607       1 main.go:149] LVM Driver Version :- 1.6.1 - commit :- a3c08bad6ad3ae64581f135e076d20b2ca26636c
I1125 20:26:42.789559       1 main.go:150] DriverName: local.csi.openebs.io Plugin: controller EndPoint: unix:///var/lib/csi/sockets/pluginproxy/csi.sock NodeID:  SetIOLimits: false ContainerRuntime: containerd RIopsPerGB: [] WIopsPerGB: [] RBpsPerGB: [] WBpsPerGB: []
I1125 20:26:42.789750       1 driver.go:49] enabling volume access mode: SINGLE_NODE_WRITER
I1125 20:26:42.790574       1 controller.go:213] waiting for k8s & lvm node informer caches to be synced
I1125 20:26:42.890728       1 controller.go:217] synced k8s & lvm node informer caches
I1125 20:26:42.890756       1 controller.go:219] initializing csi provisioning leak protection controller
I1125 20:26:42.915898       1 grpc.go:190] Listening for connections on address: &net.UnixAddr{Name:"//var/lib/csi/sockets/pluginproxy/csi.sock", Net:"unix"}
I1125 20:26:42.916943       1 leak_protection.go:134] "starting up csi pvc controller"
I1125 20:26:42.917031       1 shared_informer.go:311] Waiting for caches to sync for CSI Provisioner
I1125 20:26:42.917069       1 shared_informer.go:318] Caches are synced for CSI Provisioner
I1125 20:26:43.046136       1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I1125 20:26:43.050388       1 grpc.go:81] GRPC response: {}
I1125 20:26:43.054687       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I1125 20:26:43.054883       1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"1.6.1"}
I1125 20:26:43.056291       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginCapabilities requests {}
I1125 20:26:43.056409       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I1125 20:26:43.057896       1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I1125 20:26:43.058015       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I1125 20:26:43.059390       1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I1125 20:26:43.059489       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I1125 20:26:43.313026       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I1125 20:26:43.313086       1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"1.6.1"}
I1125 20:26:43.314005       1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I1125 20:26:43.314050       1 grpc.go:81] GRPC response: {}
I1125 20:26:43.314396       1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I1125 20:26:43.314525       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I1125 20:26:43.680817       1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I1125 20:26:43.680881       1 grpc.go:81] GRPC response: {}
I1125 20:26:43.681776       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I1125 20:26:43.681826       1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"1.6.1"}
I1125 20:26:43.682592       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginCapabilities requests {}
I1125 20:26:43.682640       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I1125 20:26:43.683304       1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I1125 20:26:43.683374       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
  • kubectl logs -f openebs-lvm-localpv-node-[xxxx] -n openebs -c openebs-lvm-plugin
ubuntu@ip-172-31-23-102:~$ microk8s kubectl logs -f openebs-lvm-localpv-node-l96hs -n openebs -c openebs-lvm-plugin
I1125 20:27:12.475692       1 main.go:149] LVM Driver Version :- 1.6.1 - commit :- a3c08bad6ad3ae64581f135e076d20b2ca26636c
I1125 20:27:12.475759       1 main.go:150] DriverName: local.csi.openebs.io Plugin: agent EndPoint: unix:///plugin/csi.sock NodeID: ip-172-31-23-102 SetIOLimits: false ContainerRuntime: containerd RIopsPerGB: [] WIopsPerGB: [] RBpsPerGB: [] WBpsPerGB: []
I1125 20:27:12.475780       1 driver.go:49] enabling volume access mode: SINGLE_NODE_WRITER
I1125 20:27:12.476870       1 grpc.go:190] Listening for connections on address: &net.UnixAddr{Name:"//plugin/csi.sock", Net:"unix"}
I1125 20:27:12.477703       1 builder.go:84] Creating event broadcaster
I1125 20:27:12.477959       1 builder.go:90] Creating lvm volume controller object
I1125 20:27:12.478048       1 builder.go:101] Adding Event handler functions for lvm volume controller
I1125 20:27:12.478112       1 start.go:73] Starting informer for lvm volume controller
I1125 20:27:12.478149       1 start.go:75] Starting Lvm volume controller
I1125 20:27:12.478199       1 volume.go:295] Starting Vol controller
I1125 20:27:12.478230       1 volume.go:298] Waiting for informer caches to sync
I1125 20:27:12.478683       1 builder.go:84] Creating event broadcaster
I1125 20:27:12.478830       1 builder.go:90] Creating lvm snapshot controller object
I1125 20:27:12.478959       1 builder.go:99] Adding Event handler functions for lvm snapshot controller
I1125 20:27:12.479090       1 start.go:72] Starting informer for lvm snapshot controller
I1125 20:27:12.479600       1 start.go:74] Starting Lvm snapshot controller
I1125 20:27:12.479665       1 snapshot.go:195] Starting Snap controller
I1125 20:27:12.479690       1 snapshot.go:198] Waiting for informer caches to sync
I1125 20:27:12.499763       1 builder.go:95] Creating lvm node controller object
I1125 20:27:12.500609       1 builder.go:110] Adding Event handler functions for lvm node controller
I1125 20:27:12.501868       1 start.go:98] Starting informer for lvm node controller
I1125 20:27:12.501897       1 start.go:101] Starting Lvm node controller
I1125 20:27:12.501926       1 lvmnode.go:223] Starting Node controller
I1125 20:27:12.501931       1 lvmnode.go:226] Waiting for informer caches to sync
I1125 20:27:12.504719       1 lvmnode.go:153] Got add event for lvm node openebs/ip-172-31-23-102
I1125 20:27:12.578452       1 volume.go:302] Starting Vol workers
I1125 20:27:12.578489       1 volume.go:309] Started Vol workers
I1125 20:27:12.580181       1 snapshot.go:202] Starting Snap workers
I1125 20:27:12.580201       1 snapshot.go:209] Started Snap workers
I1125 20:27:12.602733       1 lvmnode.go:231] Starting Node workers
I1125 20:27:12.602763       1 lvmnode.go:238] Started Node workers
I1125 20:27:12.649811       1 lvmnode.go:306] Successfully synced 'openebs/ip-172-31-23-102'
I1125 20:27:13.348389       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I1125 20:27:13.358406       1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"1.6.1"}
I1125 20:28:12.651499       1 lvmnode.go:306] Successfully synced 'openebs/ip-172-31-23-102'
I1125 20:29:12.650382       1 lvmnode.go:306] Successfully synced 'openebs/ip-172-31-23-102'
I1125 20:30:12.654277       1 lvmnode.go:306] Successfully synced 'openebs/ip-172-31-23-102'
I1125 20:31:12.660316       1 lvmnode.go:306] Successfully synced 'openebs/ip-172-31-23-102'
I1125 20:32:12.505344       1 lvmnode.go:165] Got update event for lvm node openebs/ip-172-31-23-102
I1125 20:32:12.550347       1 lvmnode.go:306] Successfully synced 'openebs/ip-172-31-23-102'
I1125 20:32:12.657691       1 lvmnode.go:306] Successfully synced 'openebs/ip-172-31-23-102'
I1125 20:33 ...
... (this continues for some time)
  • kubectl get pods -n openebs
ubuntu@ip-172-31-23-102:~$ microk8s kubectl get pods -n openebs
NAME                                              READY   STATUS    RESTARTS   AGE
openebs-localpv-provisioner-65db9df479-vmkdf      1/1     Running   0          3h3m
openebs-lvm-localpv-controller-84844f9c47-4k6pf   5/5     Running   0          3h3m
openebs-lvm-localpv-node-l96hs                    2/2     Running   0          3h3m
  • kubectl get lvmvol -A -o yaml
ubuntu@ip-172-31-23-102:~$ microk8s kubectl get lvmvol -A -o yaml
apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

  • LVM Driver version:
  • Kubernetes version (use kubectl version): v1.31.2
  • Kubernetes installer & version: microk8s 1.31
  • Cloud provider or hardware configuration: freshly created ec2 instance
  • OS (e.g. from /etc/os-release): Ubuntu 24.04.1 LTS
@condaatje condaatje mentioned this issue Nov 25, 2024
Open
@abhilashshetty04
Copy link
Contributor

abhilashshetty04 commented Nov 28, 2024 •
edited
Loading

Hi @condaatje ,

storageClassName in the pvc.spec doesnt match with storageClass.name you have share. Is that a typo?

Is this one worker node cluster? Can you share output of kubectl get lvmnode -n openebs -oyaml

We dont see any /CreateVolume calls in the csi-controller logs. Can you please share the complete log. Can you also share csi-provisioner log after reproducing again.

What event you see on the pod describe when volumeBindingMode is Immediate?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@condaatje @abhilashshetty04