[BUG]: Troubles with overloaded methods taking std::function with different signatures

[thelfer]

Problem description

Let us consider the following class:

struct Test{
  void setFunction(std::function<double(double)>);
  void setFunction(std::function<double(double, double)>);
  void call();
private:
  std::function<double(double)> f1;
  std::function<double(double, double)> f2;
};

It seems that exposing the two overloads of the setFunction method leads to a conflict, one hiding the other.

Please find here a minimal test case: test.zip

Reproducible example code

The attached C++ file exposes the Test class and the python test fails with the following message:

Traceback (most recent call last):
  File "/tmp/test.py", line 7, in <module>
    t.call()
TypeError: <lambda>() missing 1 required positional argument: 'y'

which indicates that the python lambda was passed to the wrong overload of the setFunction method.

[Yikai-Liao]

Although I'm not the maintainer of this repository, I think I can answer your question to some extend, becuase I'm familiar with pybind11, which is similar to nanobind in the interface.

It is hard to distinguish two python function by their signatures. There are several reasons:

• The type of the return value of a python function is not fixed ( functions could return a Union of something)
• The type of the input argument is still not fixed ( functions could accept a Union of something)
• The numbaer of input argument could be dynamic. (*args, **kwargs)

So If you want to store python functions as callback functions (or for other usages), you'd better store them as nanobind::callable or pybind11::function

And, of course, use 2 different function names to replace setFunction

[thelfer]

@Yikai-Liao Thanks for your answer. I understand how difficult it could be. It is also important to report troubles in a priori legitimate use of the library. From an user point of view, I would have prefered a error stating that the setFunction call is ambiguous.

Thanks also for your advices. The trouble is that I am wrapping an existing C++ library, so using nanobind::callable is not really an option here. Maybe could I dig further using the inspect module and select the appropriate method by hand...

[Yikai-Liao]

If you want to specify a cpp function (May be those functions have been binded) on python side at runtime, the best way I think is to use some Enum to dispatch the right function in case you don't want to suffer a performance loss.

Here I mentioned performance loss because even if you cast the binded cpp function to std::function, the function would be called through the python interpreter, which is slow. That's because the compiler just don't know wether the function you passed from python side is the cpp function above or a native python function.

call the func -> python interpreter -> binding level overhead -> the raw function

Function Pointer

One possible way is to write a lambda function in your binding codes (for non-intrusive binding to c++ libraries) like this:

enum class MyFunc {};
// some codes to bind MyFunc

nanobind::class_<Test>(m, "Test")
      .def("setFunction", [](Test& self, MyFunc tag) {
      switch MyFunc {
          case: // ....      set the func 
      }});

If other prameters are used to create the function, I recommend you to use another function name in python.

If you want to simplify the switch, you could use some Macro tricks such as X-Macro.

By the way, if the function is called frequenly, and it can be just treated as a function pointer, I recommend you to store the c++ function in the function pointer, which could significantly reduce the overhead.

For example, there are 3 ways to store a lambda function (without capturing any arguments):

1. template (like the template comp function in std::sort)
2. function pointer (c style)
3. functional (powerful)

Then the overhead order is that template (no overhead) < pointer (inline is prevented) << std::function (lots of overhead to support its complex features)

If you need to specify the function at runtime, I think function pointer + enum tag would be the best way.

Class with Operator ()

As I saided above, std::function is powerful, it could even store a class with customized operator().

So another possible way is to wrap the function you need into a class!

struct MyFunc1 {
double operator() (double arg) { return arg + 1. ; }
}

// define other functions

nanobind::class_<MyFunc1>(m, "MyFunc1");
nanobind::class_<Test>(m, "Test")
      .def("setFunction", [](Test& self, MyFunc1 func) { self.setFunction(func); })
      .def("setFunction", [](Test& self, MyFunc2 func) { self.setFunction(func); })

In this way the function you binded can be dispatched by the typing system.

[thelfer]

@Yikai-Liao Thanks ! That's interesting. For the moment, perfomance is not an issue: if the user wants to use a python lambda, he will pay the price for it. Otherwise, he has to write a C++ plugin.

I am thinking of exposing to distinct (private) methods using the following design:

enum class MyFunc {};
// some codes to bind MyFunc

nanobind::class_<Test>(m, "Test")
      .def("__setFunction1", ...)
      .def("__setFunction2", ...);

And in my __init__.py, add a setFunction method to the Test class which would use the inspect module to make the dispatch (i.e. select either __setFunction1 and __setFunction2)

[Yikai-Liao]

Thanks to the complexity of the typing system, you should still be able to come up with a lot of abstract solutions, as long as you don't worry about call overhead. You can explore which one best suits your project.

[Yikai-Liao]

By the way, have you found any leaked types or leaked functions warnings in python 3.12 #397 ?
I'm confused about this issue as this warning only appears in python 3.12. And I'm still not sure if this is a problem only appears in my project.

[thelfer]

A lot ! I had to use nanobind::set_leak_warnings(false); to avoid tons of warnings in what seems legitimate code. Note that I did not check what happens with other python versions.

[ThoreWietzke]

I don't know if I understand your problem correctly, but does the overload functionality https://nanobind.readthedocs.io/en/latest/classes.html#overloaded-methods work here?

[thelfer]

@ThoreWietzke I will give a try, thanks for the tips. I did not notice this feature and basically used static_cast to select the right method.

[thelfer]

@ThoreWietzke Unfortunately, it does not help. The implementation of overload_cast shows that it just does a cast with a nicer syntax.

[wjakob]

This does not look to me like an actual nanobind issue -- it's more of a gripe with the existing interface and C++ language itself. I will convert it to a discussion.

[thelfer]

@wjakob I am a bit surprised by your statement.

Is my use of nanobind incorrect ? If so, ok. Of course, please let me know what I did wrong !

If not, then this example:

1. illustrate an issue that shall be fixed if technically possible,
2. show a corner case that can't be avoided du to the design of python/nanobind/C++.

In case 1, we shall keep an issue. I can contribute by (trying to) fix it.

In case 2, this is something that the user shall be aware of and a discussion is ok with that.

Thanks for you help.

[wjakob]

Aha! I missed a bit of the context provided in your .zip file.

This is a limitation of the nanobind function type caster. It accepts any kind of function and does not check that the provided function actually has the signature that it claims to have. If the function is incompatible, using it will raise an exception later on.

Fixing this issue seems hard. I general, function objects don't expose enough information to make those kinds of decisions (in the sense of duck typing: you don't know if the function will "quack" until you tried to call it).

Some functions do expose type information in docstrings. However, you certainly would not want to call inspect to parse the docstring and analyze types at runtime: performance would be horrible. If you want to propose a PR to document this limitation, then you are welcome to do so (not sure where it would fit the best way, there AFAIK isn't a section specifically about the function type caster)