Suggestion for my benchmark using nanobind

[tsumli]

Hi,

I've recently created a repository to compare the performance of pybind, nanobind, and pyo3. During my tests, I noticed that the implementation with nanobind is slower compared to pybind. So, I'm struggling to find the reason for this difference.

I would appreciate it if someone could take a look at my code and provide any insights or suggestions for improvement. The repository is available here, and you can find the code using nanobind under the python_binding_nb/

Thank you in advance for your time and assistance!

[yosh-matsuda]

Adding NOMINSIZE and LTO to nanobind_add_module maximizes the performance.

[tsumli]

After revising the CMake file, I have observed that it is now faster than Pybind. Thanks for the quick reply, and I really appreciate your help!

[wjakob]

Is there a page showing your benchmark results?

[tsumli]

Benchmark result is visualized in this jupyter notebook.
Each experiment is repeated five times, varying the length of data and the order parameter in scipy.signal.argrelmax.

[wjakob]

Daer @tsumli,

I was quite surprised when looking at these visualizations:

Depending on the setup, both C++ and Rust are eventually compiled by very similar compiler backends, potentially even the same one (LLVM). During this process, Rust may need to add bounds checks that aren't there in the C++ version. It makes simply no sense to me that the Rust/PyO3 of your benchmark would be several times faster than the nanobind/C++ version.

In this specific benchmark, most of the cost occurs within the argrelmax function, hence this benchmark mainly measures how fast that function is implemented, with only a small portion coming from overheads of the binding tool.

One immediate observation is that you aren't specifying a target architecture for the C++ compiler. So it will compile for the oldest platform (some ancient Intel Pentium processor) to maximize compatibility/redistributability of the binary.

diff --git a/python_binding_nb/CMakeLists.txt b/python_binding_nb/CMakeLists.txt
index 66a1c64..9161822 100644
--- a/python_binding_nb/CMakeLists.txt
+++ b/python_binding_nb/CMakeLists.txt
@@ -8,7 +8,7 @@ if (NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
 endif()

 # Set flags
-set(CMAKE_CXX_FLAGS "-Wall -Wextra")
+set(CMAKE_CXX_FLAGS "-Wall -Wextra -march=native")

 # Detect the installed nanobind package and import it into CMake
 execute_process(

Another observation is that you're not using array views, which are explained in the ndarray documentation. Without a view, every access traverses the array descriptor somewhere on the heap, which impedes optimization opportunities. I believe that your Rust code is doing something similar with in the line let arr = arr.as_array();. There is also no need to pass around the ndarray by const reference, it's the const float template parameter that matters. Finally, nanobind accepts non-contiguous arrays by default, which also reduces performance. I suspect that Rust may not do so -- in any case, it might be worth comparing.

diff --git a/python_binding_nb/src/argrelmax.cpp b/python_binding_nb/src/argrelmax.cpp
index da693b7..f14464f 100644
--- a/python_binding_nb/src/argrelmax.cpp
+++ b/python_binding_nb/src/argrelmax.cpp
@@ -1,5 +1,3 @@
-#include <omp.h>
-
 #include <iostream>
 #include <typeinfo>
 #include <vector>
@@ -10,20 +8,21 @@

 namespace nb = nanobind;

-auto process(std::vector<bool> &results, const nb::ndarray<const float, nb::shape<nb::any>> &py_arr,
+auto process(std::vector<bool> &results, nb::ndarray<const float, nb::shape<nb::any>, nb::c_contig> &py_arr,
              const int shift, const size_t len) {
+    auto view = py_arr.view();
     for (auto i = 0; i < static_cast<int>(len); i++) {
         if (!results[i]) {
             continue;
         }
-        const auto plus = py_arr(std::min(i + shift, static_cast<int>(len - 1)));
-        const auto minus = py_arr(std::max(i - shift, 0));
-        const auto data = py_arr(i);
+        const auto plus = view(std::min(i + shift, static_cast<int>(len - 1)));
+        const auto minus = view(std::max(i - shift, 0));
+        const auto data = view(i);
         results[i] = (data > plus) && (data > minus);
     }
 }

-auto argrelmax(const nb::ndarray<const float, nb::shape<nb::any>> &py_arr, const int order) {
+auto argrelmax(nb::ndarray<const float, nb::shape<nb::any>, nb::c_contig> py_arr, const int order) {
     const auto len = py_arr.shape(0);
     auto results = std::vector<bool>(len, true);
     for (size_t shift = 1; shift <= static_cast<size_t>(order); shift++) {

These are just a few things I noticed at a first glance -- benchmarking things is difficult. In this case, I don't think it is an apples-to-apples comparison.