Lower performance after migrating from Ceph to OpenEbs (Elasticsearch)

[Soltaniant]

Description

We have previously gotten 16 warm-data-nodes (running on Kuber) all having the same resource (32Gb heap, 64Gb Ram, 3 Tb hdd).
Since it is not much recommended to use a distributed storage system for Elasticsearch, we have decided to use local disks instead.

As we were running on kuber and it provided us several advantages, we were looking for a kuber storage to have a neer-local disk performance and we have found the OpenEbs(Local PV: LVM) technology to do so.

I have also asked the problem from elastic community, and it seems that they also think the problem should be somewhere else.

Expected Behavior

The new cluster has 10 warm nodes each having the same Ram and CPU as before, with 8 Tb of hdd storage, which we expected to at least provide tha same performace and latency as our previous cluster does.

Despite the fact that reducing the number of nodes may reduce perfomance a bit, though we think that this is not the real problem, and the root cause is the openEbs itself.

Current Behavior

We have encountered about 10x higher latency (0.2 ms -> 2ms) and 10X higher took time than our previous cluster.

Your Environment

[root@master1 ~]# k get pvc  -n elastic 
NAME                               STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
data-elasticsearch-cord-0          Bound     pvc-434736ae-9ccd-49c1-b582-759517520363   2Gi        RWO            ceph-ssd-r3         2y130d
data-elasticsearch-cord-1          Bound     pvc-8469749b-93e6-4b8b-a6e2-94bdd2088cb3   2Gi        RWO            ceph-ssd-r3         2y130d
data-elasticsearch-cord-2          Bound     pvc-f5756c53-b61d-4223-b055-ad04cb1781d2   2Gi        RWO            ceph-ssd-r3         2y130d
data-elasticsearch-data-hot-0      Bound     pvc-d89afd4f-7254-4feb-b6cc-ab0cedca7a74   500Gi      RWO            ceph-ssd-r3         725d
data-elasticsearch-data-hot-1      Bound     pvc-9456b971-55ff-4316-b108-1d7bc2e5b7d6   500Gi      RWO            ceph-ssd-r3         725d
data-elasticsearch-data-hot-2      Bound     pvc-b4c70b20-1fc8-487e-95a6-05e1ac8b5c76   500Gi      RWO            ceph-ssd-r3         725d
data-elasticsearch-data-hot-3      Bound     pvc-6cfa199b-a99b-4c97-ad8a-c4ab795390e2   500Gi      RWO            ceph-ssd-r3         725d
data-elasticsearch-data-hot-4      Bound     pvc-14c1884e-153a-4370-9f57-1fc90965b8c0   500Gi      RWO            ceph-ssd-r3         725d
data-elasticsearch-data-hot-5      Bound     pvc-00855e98-080e-48a3-9b45-82e1d1e52bf1   500Gi      RWO            ceph-ssd-r3         725d
data-elasticsearch-data-hot-6      Bound     pvc-874d7c97-4916-47f2-abe0-e937ebfd1f99   500Gi      RWO            ceph-ssd-r3         173d
data-elasticsearch-data-hot-7      Bound     pvc-2fb8dce2-85cf-444b-89be-920662a70f2e   500Gi      RWO            ceph-ssd-r3         173d
data-elasticsearch-data-warm-0     Bound     pvc-30496e1c-5bab-458a-9bb9-e3693916b3f4   3Ti        RWO            ceph-hdd-r3         563d
data-elasticsearch-data-warm-1     Bound     pvc-ab2ab4b0-2f64-4a1e-8d2f-0847902c854c   3Ti        RWO            ceph-hdd-r3         562d
data-elasticsearch-data-warm-10    Bound     pvc-967d5bad-e143-41b4-8bb8-891dc97e4241   3Ti        RWO            ceph-hdd-r3         2y33d
data-elasticsearch-data-warm-11    Bound     pvc-7142565b-6b71-4c9a-bb13-806a804d0db7   3Ti        RWO            ceph-hdd-r3         2y33d
data-elasticsearch-data-warm-12    Bound     pvc-8c6df6b6-ef1c-4c5e-85ed-bede2ce25fdf   3Ti        RWO            ceph-hdd-r3         615d
data-elasticsearch-data-warm-13    Bound     pvc-72a0bf6b-939c-4a2f-a213-468e0997aeb3   3Ti        RWO            ceph-hdd-r3         612d
data-elasticsearch-data-warm-14    Bound     pvc-cead4f2d-2ded-44fb-9fe4-6649fd4c6d2f   3Ti        RWO            ceph-hdd-r3         612d
data-elasticsearch-data-warm-15    Bound     pvc-4a9828b5-ad66-47ab-9117-a0f7a685c6a9   3Ti        RWO            ceph-hdd-r3         354d
data-elasticsearch-data-warm-2     Bound     pvc-0f6fdef6-6b68-4cae-a67d-a6ab618dc456   3Ti        RWO            ceph-hdd-r3         562d
data-elasticsearch-data-warm-3     Bound     pvc-8bca26d5-c279-4e14-bf9f-4dde52b31678   3Ti        RWO            ceph-hdd-r3         562d
data-elasticsearch-data-warm-4     Bound     pvc-b938da18-3f8d-4172-b9c2-2a25d50f504d   3Ti        RWO            ceph-hdd-r3         516d
data-elasticsearch-data-warm-5     Bound     pvc-bd86e3f4-3d30-4732-9803-56d5eb149c1e   3Ti        RWO            ceph-hdd-r3         2y52d
data-elasticsearch-data-warm-6     Bound     pvc-a621861c-6baf-416e-9a30-7857984fdc60   3Ti        RWO            ceph-hdd-r3         2y33d
data-elasticsearch-data-warm-7     Bound     pvc-14bc5727-93b4-437d-a849-24fa602a6d2d   3Ti        RWO            ceph-hdd-r3         2y33d
data-elasticsearch-data-warm-8     Bound     pvc-59eef81e-7df4-48b3-b38b-04bbafcb4ecd   3Ti        RWO            ceph-hdd-r3         550d
data-elasticsearch-data-warm-9     Bound     pvc-c688f17f-f254-4598-959f-ae9582914420   3Ti        RWO            ceph-hdd-r3         2y33d
data-elasticsearch-master-0        Bound     pvc-5ee0cc5d-aa52-49dd-96fd-bc1625f3d885   2Gi        RWO            ceph-ssd-r3         562d
data-elasticsearch-master-1        Bound     pvc-a5696799-adaf-439e-8be1-8a5f5bc959d7   2Gi        RWO            ceph-ssd-r3         516d
data-elasticsearch-master-2        Bound     pvc-0125575e-46fe-4604-a4af-27b8a6bf7ffa   2Gi        RWO            ceph-ssd-r3         562d
data-elasticsearch-warm-ebs113-0   Bound     pvc-42b52a91-61bd-4851-9a41-def7a697c9c8   7451Gi     RWO            openebs-lvmpv-hdd   6d19h
data-elasticsearch-warm-ebs113-1   Bound     pvc-f14cf3f8-e6fb-49e9-a5d4-0f16450aa51b   7451Gi     RWO            openebs-lvmpv-hdd   6d19h
data-elasticsearch-warm-ebs114-0   Bound     pvc-50a1aba6-72b9-45af-8b4e-e997337144cb   7451Gi     RWO            openebs-lvmpv-hdd   6d19h
data-elasticsearch-warm-ebs114-1   Bound     pvc-c4a85532-fbd2-43d7-9145-31088dc735a4   7451Gi     RWO            openebs-lvmpv-hdd   6d19h
data-elasticsearch-warm-ebs115-0   Bound     pvc-2c92a344-aba0-4bd2-9d8d-cb441e8fdb91   7451Gi     RWO            openebs-lvmpv-hdd   6d19h
data-elasticsearch-warm-ebs115-1   Bound     pvc-49839744-33e4-4d5e-a92e-ae2bff0e9d49   7451Gi     RWO            openebs-lvmpv-hdd   6d19h
data-elasticsearch-warm-ebs14-0    Bound     pvc-e02edffd-98a9-461d-9a87-93912ebbe6f2   7451Gi     RWO            openebs-lvmpv-hdd   6d19h
data-elasticsearch-warm-ebs14-1    Bound     pvc-8b1db2ca-e035-42c5-90f2-1104c1d7583f   7451Gi     RWO            openebs-lvmpv-hdd   6d19h
data-elasticsearch-warm-ebs15-0    Bound     pvc-32677343-81cd-4fb3-98b8-8ecd0246ad1d   7451Gi     RWO            openebs-lvmpv-hdd   6d19h
data-elasticsearch-warm-ebs15-1    Pending                                                                        openebs-lvmpv-hdd   4d4h
data-elasticsearch-warm-ebs16-0    Bound     pvc-882f9714-ddb3-4676-93f0-ec34bf291006   7451Gi     RWO            openebs-lvmpv-hdd   6d19h
mnt                                Bound     pvc-732949db-7bda-46a1-bdf5-d2879e41cd16   1Gi        RWX            cephfs-ssd-r3       147d

os_release
NAME="Oracle Linux Server"
VERSION="8.7"

[dsharma-dc]

Responded here: openebs/openebs#3731

[dsharma-dc]

Closing this issue as there hasn't been any activity here, and there is a duplicate open which will be kept open for some more time expecting comments from reporter.