Functions#
There are numerous generations options that can be set in order to change function bindings options.
See options.py: all the function related options begin wth fn_ or fn_params (when they deal with function parameters)
Exclude functions and/or params#
Extract from options.py, showing the related options:
################################################################################
# <functions and method adaptations>
################################################################################
# Exclude certain functions and methods by a regex on their name
fn_exclude_by_name__regex: str = ""
# Exclude certain functions and methods by a regex on any of their parameter type and/or return type
# (those should be decorated type)
# For example:
# options.fn_exclude_by_param_type__regex = "^char\s*$|^unsigned\s+char$|Callback$"
# would exclude all functions having params of type "char *", "unsigned char", "xxxCallback"
#
# Note: this is distinct from `fn_params_exclude_types__regex` which removes params
# from the function signature, but not the function itself.
fn_exclude_by_param_type__regex: str = ""
# ------------------------------------------------------------------------------
# Exclude some params by name or type
# ------------------------------------------------------------------------------
# Remove some params from the python published interface. A param can only be removed if it has a default value
# in the C++ signature
fn_params_exclude_names__regex: str = ""
fn_params_exclude_types__regex: str = ""
As an example, let’s consider the code below, where we would want to:
exclude all functions beginning with “priv_”
exclude a function parameter if its type name starts with “Private”
import litgen
from litgen.demo import litgen_demo
cpp_code = """
void priv_SetOptions(bool v);
void SetOptions(const PublicOptions& options, const PrivateOptions& privateOptions = PrivateOptions());
"""
By default the generated code will be:
options = litgen.LitgenOptions()
litgen_demo.demo(options, cpp_code)
void priv_SetOptions(bool v);
void SetOptions(const PublicOptions& options, const PrivateOptions& privateOptions = PrivateOptions());
def priv_set_options(v: bool) -> None:
pass
def set_options(
options: PublicOptions,
private_options: PrivateOptions = PrivateOptions()
) -> None:
pass
m.def("priv_set_options",
priv_SetOptions, py::arg("v"));
m.def("set_options",
SetOptions, py::arg("options"), py::arg("private_options") = PrivateOptions());
m.def("priv_set_options",
priv_SetOptions, nb::arg("v"));
m.def("set_options",
SetOptions, nb::arg("options"), nb::arg("private_options") = PrivateOptions());
But we can set some options to change this.
In the generated code below, look closely at the C++ binding code: you will see that it takes steps to generate a default value for the parameter of type PrivateOptions
options = litgen.LitgenOptions()
options.fn_exclude_by_name__regex = "^priv_" # Exclude functions whose name begin by "priv_"
options.fn_params_exclude_types__regex = "Private" # Exclude functions params whose type name contains "Private"
litgen_demo.demo(options, cpp_code, show_pydef=True)
void priv_SetOptions(bool v);
void SetOptions(const PublicOptions& options, const PrivateOptions& privateOptions = PrivateOptions());
def set_options(options: PublicOptions) -> None:
pass
m.def("set_options",
[](const PublicOptions & options)
{
auto SetOptions_adapt_exclude_params = [](const PublicOptions & options)
{
SetOptions(options, PrivateOptions());
};
SetOptions_adapt_exclude_params(options);
}, py::arg("options"));
m.def("set_options",
[](const PublicOptions & options)
{
auto SetOptions_adapt_exclude_params = [](const PublicOptions & options)
{
SetOptions(options, PrivateOptions());
};
SetOptions_adapt_exclude_params(options);
}, nb::arg("options"));
Return policy#
See relevant doc from pybind11:
Python and C++ use fundamentally different ways of managing the memory and lifetime of objects managed by them. This can lead to issues when creating bindings for functions that return a non-trivial type. Just by looking at the type information, it is not clear whether Python should take charge of the returned value and eventually free its resources, or if this is handled on the C++ side. For this reason, pybind11 provides a several return value policy annotations that can be passed to the
module_::def()andclass_::def()functions. The default policy isreturn_value_policy::automatic.
See relevant doc from nanobind:
nanobind provides several return value policy annotations that can be passed to
module_::def(),class_::def(), andcpp_function()…
return_value_policy::reference#
In the C++ code below, let’s suppose that C++ is responsible for handling the destruction of the values returned by MakeWidget and MakeFoo: we do not want python to call the destructor automatically.
cpp_code = """
Widget * MakeWidget();
Foo& MakeFoo();
"""
In that case, we can set options.fn_return_force_policy_reference_for_pointers__regex and/or options.fn_return_force_policy_reference_for_references__regex, and the generated pydef binding code, will set the correct return value policy.
options = litgen.LitgenOptions()
options.fn_return_force_policy_reference_for_pointers__regex = r"^Make"
options.fn_return_force_policy_reference_for_references__regex = r"^Make"
litgen_demo.demo(options, cpp_code, show_pydef=True)
Widget * MakeWidget();
Foo& MakeFoo();
def make_widget() -> Widget:
pass
def make_foo() -> Foo:
pass
m.def("make_widget",
MakeWidget, py::return_value_policy::reference);
m.def("make_foo",
MakeFoo, py::return_value_policy::reference);
m.def("make_widget",
MakeWidget, nb::rv_policy::reference);
m.def("make_foo",
MakeFoo, nb::rv_policy::reference);
Custom return value policy#
If you annotate the function declaration with return_value_policy::..., then the generator will use this information:
cpp_code = """
Widget *MakeWidget(); // return_value_policy::take_ownership
"""
options = litgen.LitgenOptions()
litgen_demo.demo(options, cpp_code, show_pydef=True)
Widget *MakeWidget(); // return_value_policy::take_ownership
def make_widget() -> Widget:
""" return_value_policy::take_ownership"""
pass
m.def("make_widget",
MakeWidget,
"return_value_policy::take_ownership",
py::return_value_policy::take_ownership);
m.def("make_widget",
MakeWidget,
"return_value_policy::take_ownership",
nb::rv_policy::take_ownership);
Handle mutable default param values#
See “Why are default values shared between objects?” in the Python FAQ
There is a common pitfall in Python when using mutable default values in function signatures: if the default value is a mutable object, then it is shared between all calls to the function. This is because the default value is evaluated only once, when the function is defined, and not each time the function is called.
The Python code below shows this issue:
def use_elems(elems = []): # elems is a mutable default argument
elems.append(1) # This will affect the default argument, for **all** subsequent calls!
print(elems)
use_elems() # will print [1]
use_elems() # will print [1, 1]
use_elems() # will print [1, 1, 1]
[1]
[1, 1]
[1, 1, 1]
This is fundamentally different from C++ default arguments, where the default value is evaluated each time the function is called.
For bound C++ functions, in most cases the default value still be reevaluated at each call. However, this is not guaranteed, especially when using nanobind!
In order to handle this, litgen provides a set of options that can be set to change the behavior of the generated code. By default,those options are disabled, and the default parameters values might be shared between calls (but your mileage may vary).
Recommended settings for nanobind:
options.fn_params_adapt_mutable_param_with_default_value__to_autogenerated_named_ctor = Trueoptions.fn_params_adapt_mutable_param_with_default_value__regex = r".*"you may call
options.use_nanobind()to set these options as well as the library to nanobind
There are a few related options that can be set to change the behavior of the generated code. See the extract from options.py below:
# ------------------------------------------------------------------------------
# Make "mutable default parameters" behave like C++ default arguments
# (i.e. re-evaluate the default value each time the function is called)
# ------------------------------------------------------------------------------
# Regex which contains a list of regexes on functions names for which this transformation will be applied.
# by default, this is disabled (set it to r".*" to enable it for all functions)
fn_params_adapt_mutable_param_with_default_value__regex: str = r""
# if True, auto-generated named constructors will adapt mutable default parameters
fn_params_adapt_mutable_param_with_default_value__to_autogenerated_named_ctor: bool = False
# if True, a comment will be added in the stub file to explain the behavior
fn_params_adapt_mutable_param_with_default_value__add_comment: bool = True
# fn_params_adapt_mutable_param_with_default_value__fn_is_known_immutable_type
# may contain a user defined function that will determine if a type is considered immutable in python based on its name.
# By default, all the types below are considered immutable in python:
# "int|float|double|bool|char|unsigned char|std::string|..."
fn_params_adapt_mutable_param_with_default_value__fn_is_known_immutable_type: Callable[[str], bool] | None = None
# Same as above, but for values
fn_params_adapt_mutable_param_with_default_value__fn_is_known_immutable_value: Callable[[str], bool] | None = None
If those options are active, litgen will by default wrap the parameter into an Optional[Parameter_type], and then check if the passed value is None. This step will be done for parameters that have a default value which is mutable.
In the example below, use_elems signature in Python becomes use_elems(elems: Optional[List[int]] = None), and the generated code will check if elems is None, and if so, create a new list.
cpp_code = """
void use_elems(const std::vector<int> &elems = {}) {
elems.push_back(1);
std::cout << elems.size() << std::endl;
}
"""
options = litgen.LitgenOptions()
options.fn_params_adapt_mutable_param_with_default_value__regex = r".*"
litgen_demo.demo(options, cpp_code, show_pydef=False)
void use_elems(const std::vector<int> &elems = {}) {
elems.push_back(1);
std::cout << elems.size() << std::endl;
}
def use_elems(elems: Optional[List[int]] = None) -> None:
"""---
Python bindings defaults:
If elems is None, then its default value will be: initialized with default value
"""
pass
m.def("use_elems",
[](const std::optional<const std::vector<int>> & elems = std::nullopt)
{
auto use_elems_adapt_mutable_param_with_default_value = [](const std::optional<const std::vector<int>> & elems = std::nullopt)
{
const std::vector<int>& elems_or_default = [&]() -> const std::vector<int> {
if (elems.has_value())
return elems.value();
else
return {};
}();
use_elems(elems_or_default);
};
use_elems_adapt_mutable_param_with_default_value(elems);
},
py::arg("elems") = py::none(),
"---\nPython bindings defaults:\n If elems is None, then its default value will be: initialized with default value");
m.def("use_elems",
[](const std::optional<const std::vector<int>> & elems = std::nullopt)
{
auto use_elems_adapt_mutable_param_with_default_value = [](const std::optional<const std::vector<int>> & elems = std::nullopt)
{
const std::vector<int>& elems_or_default = [&]() -> const std::vector<int> {
if (elems.has_value())
return elems.value();
else
return {};
}();
use_elems(elems_or_default);
};
use_elems_adapt_mutable_param_with_default_value(elems);
},
nb::arg("elems") = nb::none(),
"---\nPython bindings defaults:\n If elems is None, then its default value will be: initialized with default value");
Below is a more advanced example, where we use an inner struct insider another struct: the autogenerated default constructor with named params will use a wrapped Optional type. You can see that the behavior is nicely explained in the generated stub, as an help for the user.
cpp_code = """
struct Inner {
int a = 0;
Inner(int _a) : a(_a) {}
};
struct SomeStruct {
Inner inner = Inner(42);
};
"""
options = litgen.LitgenOptions()
options.fn_params_adapt_mutable_param_with_default_value__regex = r".*"
options.fn_params_adapt_mutable_param_with_default_value__to_autogenerated_named_ctor = True
litgen_demo.demo(options, cpp_code, show_pydef=False)
struct Inner {
int a = 0;
Inner(int _a) : a(_a) {}
};
struct SomeStruct {
Inner inner = Inner(42);
};
class Inner:
a: int = 0
def __init__(self, _a: int) -> None:
pass
class SomeStruct:
inner: Inner = Inner(42)
def __init__(self, inner: Optional[Inner] = None) -> None:
"""Auto-generated default constructor with named params
---
Python bindings defaults:
If inner is None, then its default value will be: Inner(42)
"""
pass
auto pyClassInner =
py::class_<Inner>
(m, "Inner", "")
.def_readwrite("a", &Inner::a, "")
.def(py::init<int>(),
py::arg("_a"))
;
auto pyClassSomeStruct =
py::class_<SomeStruct>
(m, "SomeStruct", "")
.def(py::init<>([](
const std::optional<const Inner> & inner = std::nullopt)
{
auto r = std::make_unique<SomeStruct>();
if (inner.has_value())
r->inner = inner.value();
else
r->inner = Inner(42);
return r;
})
, py::arg("inner") = py::none()
)
.def_readwrite("inner", &SomeStruct::inner, "")
;
auto pyClassInner =
nb::class_<Inner>
(m, "Inner", "")
.def_rw("a", &Inner::a, "")
.def(nb::init<int>(),
nb::arg("_a"))
;
auto pyClassSomeStruct =
nb::class_<SomeStruct>
(m, "SomeStruct", "")
.def("__init__", [](SomeStruct * self, const std::optional<const Inner> & inner = std::nullopt)
{
new (self) SomeStruct(); // placement new
auto r = self;
if (inner.has_value())
r->inner = inner.value();
else
r->inner = Inner(42);
},
nb::arg("inner") = nb::none()
)
.def_rw("inner", &SomeStruct::inner, "")
;
Handle modifiable immutable params#
Some C++ functions may use a modifiable input/output parameter, for which the corresponding type in python is immutable (e.g. its is a numeric type, or a string).
For example, in the C++ code below, the param inOutFlag is modified by the function.
void SwitchBool(bool* inOutFlag);
In python, a function with the following signature can not change its parameter value, since bool is immutable:
def switch_bool(in_out_v: bool) -> None:
pass
litgen offers two ways to handle those situations:
by using boxed types
by adding the modified value to the function output
Using boxed types#
You can decide to replace this kind of parameters type by a “Boxed” type: this is a simple class that encapsulates the value, and makes it modifiable.
Look at the example below where a BoxedBool is created:
its python signature is given in the stub
its C++ declaration is given in the glue code
the C++ binding code handle the conversion between
bool *andBoxedBool
cpp_code = "void SwitchBool(bool* inOutFlag);"
options = litgen.LitgenOptions()
options.fn_params_replace_modifiable_immutable_by_boxed__regex = r".*" # "Box" all modifiable immutable parameters
litgen_demo.demo(options, cpp_code, show_pydef=True)
void SwitchBool(bool* inOutFlag);
#################### <generated_from:BoxedTypes> ####################
class BoxedBool:
value: bool
def __init__(self, v: bool = False) -> None:
pass
def __repr__(self) -> str:
pass
#################### </generated_from:BoxedTypes> ####################
def switch_bool(in_out_flag: BoxedBool) -> None:
pass
//////////////////// <generated_from:BoxedTypes> ////////////////////
auto pyClassBoxedBool =
py::class_<BoxedBool>
(m, "BoxedBool", "")
.def_readwrite("value", &BoxedBool::value, "")
.def(py::init<bool>(),
py::arg("v") = false)
.def("__repr__",
&BoxedBool::__repr__)
;
//////////////////// </generated_from:BoxedTypes> ////////////////////
m.def("switch_bool",
[](BoxedBool & inOutFlag)
{
auto SwitchBool_adapt_modifiable_immutable = [](BoxedBool & inOutFlag)
{
bool * inOutFlag_boxed_value = & (inOutFlag.value);
SwitchBool(inOutFlag_boxed_value);
};
SwitchBool_adapt_modifiable_immutable(inOutFlag);
}, py::arg("in_out_flag"));
//////////////////// <generated_from:BoxedTypes> ////////////////////
auto pyClassBoxedBool =
nb::class_<BoxedBool>
(m, "BoxedBool", "")
.def_rw("value", &BoxedBool::value, "")
.def(nb::init<bool>(),
nb::arg("v") = false)
.def("__repr__",
&BoxedBool::__repr__)
;
//////////////////// </generated_from:BoxedTypes> ////////////////////
m.def("switch_bool",
[](BoxedBool & inOutFlag)
{
auto SwitchBool_adapt_modifiable_immutable = [](BoxedBool & inOutFlag)
{
bool * inOutFlag_boxed_value = & (inOutFlag.value);
SwitchBool(inOutFlag_boxed_value);
};
SwitchBool_adapt_modifiable_immutable(inOutFlag);
}, nb::arg("in_out_flag"));
struct BoxedBool
{
bool value;
BoxedBool(bool v = false) : value(v) {}
std::string __repr__() const { return std::string("BoxedBool(") + std::to_string(value) + ")"; }
};
Adding the modified value to the function output#
Let’s say that we have a C++ function that modifies a string, and returns a bool that indicates whether it was modified:
bool UserInputString(std::string* inOutStr);
We can ask litgen to add the modified string to the output of the function.
Look at the example below:
the python function returns a
Tuple[bool, str]the C++ binding code adds a lambda that does the necessary transformation
cpp_code = "bool UserInputString(std::string* inOutStr);"
options = litgen.LitgenOptions()
options.fn_params_output_modifiable_immutable_to_return__regex = r".*"
litgen_demo.demo(options, cpp_code, show_pydef=True)
bool UserInputString(std::string* inOutStr);
def user_input_string(in_out_str: str) -> Tuple[bool, str]:
pass
m.def("user_input_string",
[](std::string inOutStr) -> std::tuple<bool, std::string>
{
auto UserInputString_adapt_modifiable_immutable_to_return = [](std::string inOutStr) -> std::tuple<bool, std::string>
{
std::string * inOutStr_adapt_modifiable = & inOutStr;
bool r = UserInputString(inOutStr_adapt_modifiable);
return std::make_tuple(r, inOutStr);
};
return UserInputString_adapt_modifiable_immutable_to_return(inOutStr);
}, py::arg("in_out_str"));
m.def("user_input_string",
[](std::string inOutStr) -> std::tuple<bool, std::string>
{
auto UserInputString_adapt_modifiable_immutable_to_return = [](std::string inOutStr) -> std::tuple<bool, std::string>
{
std::string * inOutStr_adapt_modifiable = & inOutStr;
bool r = UserInputString(inOutStr_adapt_modifiable);
return std::make_tuple(r, inOutStr);
};
return UserInputString_adapt_modifiable_immutable_to_return(inOutStr);
}, nb::arg("in_out_str"));
C style function params#
Immutable C array param#
If a function uses a param whose type is const SomeType[N], then it will be translated automatically, and the C++ binding code will handle the necessary transformations.
cpp_code = "void foo(const int v[2]);"
options = litgen.LitgenOptions()
litgen_demo.demo(options, cpp_code, show_pydef=True)
void foo(const int v[2]);
def foo(v: List[int]) -> None:
pass
m.def("foo",
[](const std::array<int, 2>& v)
{
auto foo_adapt_fixed_size_c_arrays = [](const std::array<int, 2>& v)
{
foo(v.data());
};
foo_adapt_fixed_size_c_arrays(v);
}, py::arg("v"));
m.def("foo",
[](const std::array<int, 2>& v)
{
auto foo_adapt_fixed_size_c_arrays = [](const std::array<int, 2>& v)
{
foo(v.data());
};
foo_adapt_fixed_size_c_arrays(v);
}, nb::arg("v"));
Modifiable C array param#
If a function uses a param whose type is SomeType[N] v, then litgen will understand that any value inside v can be modified, and it will emit code where a C++ signature like this:
void foo(int v[2]);
is transformed into python:
def foo(v_0: BoxedInt, v_1: BoxedInt) -> None:
pass
cpp_code = "void foo(int v[2]);"
options = litgen.LitgenOptions()
litgen_demo.demo(options, cpp_code)
void foo(int v[2]);
#################### <generated_from:BoxedTypes> ####################
class BoxedInt:
value: int
def __init__(self, v: int = 0) -> None:
pass
def __repr__(self) -> str:
pass
#################### </generated_from:BoxedTypes> ####################
def foo(v_0: BoxedInt, v_1: BoxedInt) -> None:
pass
//////////////////// <generated_from:BoxedTypes> ////////////////////
auto pyClassBoxedInt =
py::class_<BoxedInt>
(m, "BoxedInt", "")
.def_readwrite("value", &BoxedInt::value, "")
.def(py::init<int>(),
py::arg("v") = 0)
.def("__repr__",
&BoxedInt::__repr__)
;
//////////////////// </generated_from:BoxedTypes> ////////////////////
m.def("foo",
[](BoxedInt & v_0, BoxedInt & v_1)
{
auto foo_adapt_fixed_size_c_arrays = [](BoxedInt & v_0, BoxedInt & v_1)
{
int v_raw[2];
v_raw[0] = v_0.value;
v_raw[1] = v_1.value;
foo(v_raw);
v_0.value = v_raw[0];
v_1.value = v_raw[1];
};
foo_adapt_fixed_size_c_arrays(v_0, v_1);
}, py::arg("v_0"), py::arg("v_1"));
//////////////////// <generated_from:BoxedTypes> ////////////////////
auto pyClassBoxedInt =
nb::class_<BoxedInt>
(m, "BoxedInt", "")
.def_rw("value", &BoxedInt::value, "")
.def(nb::init<int>(),
nb::arg("v") = 0)
.def("__repr__",
&BoxedInt::__repr__)
;
//////////////////// </generated_from:BoxedTypes> ////////////////////
m.def("foo",
[](BoxedInt & v_0, BoxedInt & v_1)
{
auto foo_adapt_fixed_size_c_arrays = [](BoxedInt & v_0, BoxedInt & v_1)
{
int v_raw[2];
v_raw[0] = v_0.value;
v_raw[1] = v_1.value;
foo(v_raw);
v_0.value = v_raw[0];
v_1.value = v_raw[1];
};
foo_adapt_fixed_size_c_arrays(v_0, v_1);
}, nb::arg("v_0"), nb::arg("v_1"));
struct BoxedInt
{
int value;
BoxedInt(int v = 0) : value(v) {}
std::string __repr__() const { return std::string("BoxedInt(") + std::to_string(value) + ")"; }
};
C style string list#
If a pair of function params look like const char * const items[], int item_count, it will be transformed into a python List[str]:
cpp_code = "void PrintItems(const char * const items[], int item_count);"
options = litgen.LitgenOptions()
options.fn_params_replace_c_string_list__regex = r".*" # apply to all function names (this is the default!)
litgen_demo.demo(options, cpp_code)
void PrintItems(const char * const items[], int item_count);
def print_items(items: List[str]) -> None:
pass
m.def("print_items",
[](const std::vector<std::string> & items)
{
auto PrintItems_adapt_c_string_list = [](const std::vector<std::string> & items)
{
std::vector<const char *> items_ptrs;
for (const auto& v: items)
items_ptrs.push_back(v.c_str());
int item_count = static_cast<int>(items.size());
PrintItems(items_ptrs.data(), item_count);
};
PrintItems_adapt_c_string_list(items);
}, py::arg("items"));
m.def("print_items",
[](const std::vector<std::string> & items)
{
auto PrintItems_adapt_c_string_list = [](const std::vector<std::string> & items)
{
std::vector<const char *> items_ptrs;
items_ptrs.reserve(items.size());
for (const auto& v: items)
items_ptrs.push_back(v.c_str());
int item_count = static_cast<int>(items.size());
PrintItems(items_ptrs.data(), item_count);
};
PrintItems_adapt_c_string_list(items);
}, nb::arg("items"));
C style variadic string format#
If a function uses a pair of parameters like char const* const format, ..., then litgen will transform it into a simple python string.
cpp_code = "void Log(LogLevel level, char const* const format, ...);"
options = litgen.LitgenOptions()
litgen_demo.demo(options, cpp_code)
void Log(LogLevel level, char const* const format, ...);
def log(level: LogLevel, format: str) -> None:
pass
m.def("log",
[](LogLevel level, const char * const format)
{
auto Log_adapt_variadic_format = [](LogLevel level, const char * const format)
{
Log(level, "%s", format);
};
Log_adapt_variadic_format(level, format);
}, py::arg("level"), py::arg("format"));
m.def("log",
[](LogLevel level, const char * const format)
{
auto Log_adapt_variadic_format = [](LogLevel level, const char * const format)
{
Log(level, "%s", format);
};
Log_adapt_variadic_format(level, format);
}, nb::arg("level"), nb::arg("format"));
Passing numeric buffers to numpy#
Simple numeric buffers#
If a function uses a pair (a more) of parameters which look like (double *values, int count), or (const float* values, int nb) (etc.), then litgen can transform this parameter into a numpy array.
Let’s see an example with this function:
void PlotXY(const float *xValues, const float *yValues, size_t how_many);
We would like it to be published as:
def plot_xy(x_values: np.ndarray, y_values: np.ndarray) -> None:
pass
We will need to tell litgen:
Which function are concerned (options.fn_params_replace_buffer_by_array__regex)
The name of the the “count” param if it is not a standard one (count, nb, etc)
Note: if you look at the pybind11 C++ binding code, you will see that litgen handles the transformation, and ensures that the types are correct.
cpp_code = """
void PlotXY(const float *xValues, const float *yValues, size_t how_many);
"""
options = litgen.LitgenOptions()
options.fn_params_replace_buffer_by_array__regex = r"^Plot"
options.fn_params_buffer_size_names__regex += "|how_many"
litgen_demo.demo(options, cpp_code)
void PlotXY(const float *xValues, const float *yValues, size_t how_many);
def plot_xy(x_values: np.ndarray, y_values: np.ndarray) -> None:
pass
m.def("plot_xy",
[](const py::array & xValues, const py::array & yValues)
{
auto PlotXY_adapt_c_buffers = [](const py::array & xValues, const py::array & yValues)
{
// Check if the array is 1D and C-contiguous
if (! (xValues.ndim() == 1 && xValues.strides(0) == xValues.itemsize()) )
throw std::runtime_error("The array must be 1D and contiguous");
// convert py::array to C standard buffer (const)
const void * xValues_from_pyarray = xValues.data();
py::ssize_t xValues_count = xValues.shape()[0];
char xValues_type = xValues.dtype().char_();
if (xValues_type != 'f')
throw std::runtime_error(std::string(R"msg(
Bad type! Expected a numpy array of native type:
const float *
Which is equivalent to
f
(using py::array::dtype().char_() as an id)
)msg"));
// Check if the array is 1D and C-contiguous
if (! (yValues.ndim() == 1 && yValues.strides(0) == yValues.itemsize()) )
throw std::runtime_error("The array must be 1D and contiguous");
// convert py::array to C standard buffer (const)
const void * yValues_from_pyarray = yValues.data();
py::ssize_t yValues_count = yValues.shape()[0];
char yValues_type = yValues.dtype().char_();
if (yValues_type != 'f')
throw std::runtime_error(std::string(R"msg(
Bad type! Expected a numpy array of native type:
const float *
Which is equivalent to
f
(using py::array::dtype().char_() as an id)
)msg"));
PlotXY(static_cast<const float *>(xValues_from_pyarray), static_cast<const float *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
};
PlotXY_adapt_c_buffers(xValues, yValues);
}, py::arg("x_values"), py::arg("y_values"));
m.def("plot_xy",
[](const nb::ndarray<> & xValues, const nb::ndarray<> & yValues)
{
auto PlotXY_adapt_c_buffers = [](const nb::ndarray<> & xValues, const nb::ndarray<> & yValues)
{
// Check if the array is 1D and C-contiguous
if (! (xValues.ndim() == 1 && xValues.stride(0) == 1))
throw std::runtime_error("The array must be 1D and contiguous");
// convert nb::ndarray to C standard buffer (const)
const void * xValues_from_pyarray = xValues.data();
size_t xValues_count = xValues.shape(0);
// Check the type of the ndarray (generic type and size)
// - Step 1: check the generic type (one of dtype_code::Int, UInt, Float, Bfloat, Complex, Bool = 6);
uint8_t dtype_code_python_0 = xValues.dtype().code;
uint8_t dtype_code_cpp_0 = static_cast<uint8_t>(nb::dlpack::dtype_code::Float);
if (dtype_code_python_0 != dtype_code_cpp_0)
throw std::runtime_error(std::string(R"msg(
Bad type! While checking the generic type (dtype_code=Float)!
)msg"));
// - Step 2: check the size of the type
size_t size_python_0 = xValues.dtype().bits / 8;
size_t size_cpp_0 = sizeof(float);
if (size_python_0 != size_cpp_0)
throw std::runtime_error(std::string(R"msg(
Bad type! Size mismatch, while checking the size of the type (for param "xValues")!
)msg"));
// Check if the array is 1D and C-contiguous
if (! (yValues.ndim() == 1 && yValues.stride(0) == 1))
throw std::runtime_error("The array must be 1D and contiguous");
// convert nb::ndarray to C standard buffer (const)
const void * yValues_from_pyarray = yValues.data();
size_t yValues_count = yValues.shape(0);
// Check the type of the ndarray (generic type and size)
// - Step 1: check the generic type (one of dtype_code::Int, UInt, Float, Bfloat, Complex, Bool = 6);
uint8_t dtype_code_python_1 = yValues.dtype().code;
uint8_t dtype_code_cpp_1 = static_cast<uint8_t>(nb::dlpack::dtype_code::Float);
if (dtype_code_python_1 != dtype_code_cpp_1)
throw std::runtime_error(std::string(R"msg(
Bad type! While checking the generic type (dtype_code=Float)!
)msg"));
// - Step 2: check the size of the type
size_t size_python_1 = yValues.dtype().bits / 8;
size_t size_cpp_1 = sizeof(float);
if (size_python_1 != size_cpp_1)
throw std::runtime_error(std::string(R"msg(
Bad type! Size mismatch, while checking the size of the type (for param "yValues")!
)msg"));
PlotXY(static_cast<const float *>(xValues_from_pyarray), static_cast<const float *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
};
PlotXY_adapt_c_buffers(xValues, yValues);
}, nb::arg("x_values"), nb::arg("y_values"));
Template numeric buffers#
If a template function uses a pair of parameters whose signature looks like (const T* values, int count), then it can be transformed into a numpy array.
In the example below, we would like the following C++ function:
template<typename NumberType> void PlotXY(Color color, const NumberType *xValues, const NumberType *yValues, size_t count);
To be published as:
def plot_xy(color: Color, x_values: np.ndarray, y_values: np.ndarray) -> None:
pass
For this we need to:
Set which function names are concerned (options.fn_params_replace_buffer_by_array__regex)
Optionally, add the name of the template param (options.fn_params_buffer_template_types)
Note: if you look at the generated pybind11 C++ binding code, you will see that it handles all numeric types. This is a very efficient way to transmit numeric buffers of all types to python
cpp_code = """
template<typename NumberType>
void PlotXY(Color color, const NumberType *xValues, const NumberType *yValues, size_t count);
"""
options = litgen.LitgenOptions()
options.fn_params_replace_buffer_by_array__regex = r"^Plot"
options.fn_params_buffer_template_types += "|NumberType"
litgen_demo.demo(options, cpp_code, height=80)
template<typename NumberType>
void PlotXY(Color color, const NumberType *xValues, const NumberType *yValues, size_t count);
def plot_xy(color: Color, x_values: np.ndarray, y_values: np.ndarray) -> None:
pass
m.def("plot_xy",
[](Color color, const py::array & xValues, const py::array & yValues)
{
auto PlotXY_adapt_c_buffers = [](Color color, const py::array & xValues, const py::array & yValues)
{
// Check if the array is 1D and C-contiguous
if (! (xValues.ndim() == 1 && xValues.strides(0) == xValues.itemsize()) )
throw std::runtime_error("The array must be 1D and contiguous");
// convert py::array to C standard buffer (const)
const void * xValues_from_pyarray = xValues.data();
py::ssize_t xValues_count = xValues.shape()[0];
// Check if the array is 1D and C-contiguous
if (! (yValues.ndim() == 1 && yValues.strides(0) == yValues.itemsize()) )
throw std::runtime_error("The array must be 1D and contiguous");
// convert py::array to C standard buffer (const)
const void * yValues_from_pyarray = yValues.data();
py::ssize_t yValues_count = yValues.shape()[0];
#ifdef _WIN32
using np_uint_l = uint32_t;
using np_int_l = int32_t;
#else
using np_uint_l = uint64_t;
using np_int_l = int64_t;
#endif
// call the correct template version by casting
char yValues_type = yValues.dtype().char_();
if (yValues_type == 'B')
PlotXY(color, static_cast<const uint8_t *>(xValues_from_pyarray), static_cast<const uint8_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'b')
PlotXY(color, static_cast<const int8_t *>(xValues_from_pyarray), static_cast<const int8_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'H')
PlotXY(color, static_cast<const uint16_t *>(xValues_from_pyarray), static_cast<const uint16_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'h')
PlotXY(color, static_cast<const int16_t *>(xValues_from_pyarray), static_cast<const int16_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'I')
PlotXY(color, static_cast<const uint32_t *>(xValues_from_pyarray), static_cast<const uint32_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'i')
PlotXY(color, static_cast<const int32_t *>(xValues_from_pyarray), static_cast<const int32_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'L')
PlotXY(color, static_cast<const np_uint_l *>(xValues_from_pyarray), static_cast<const np_uint_l *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'l')
PlotXY(color, static_cast<const np_int_l *>(xValues_from_pyarray), static_cast<const np_int_l *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'f')
PlotXY(color, static_cast<const float *>(xValues_from_pyarray), static_cast<const float *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'd')
PlotXY(color, static_cast<const double *>(xValues_from_pyarray), static_cast<const double *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'g')
PlotXY(color, static_cast<const long double *>(xValues_from_pyarray), static_cast<const long double *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'q')
PlotXY(color, static_cast<const long long *>(xValues_from_pyarray), static_cast<const long long *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
// If we reach this point, the array type is not supported!
else
throw std::runtime_error(std::string("Bad array type ('") + yValues_type + "') for param yValues");
};
PlotXY_adapt_c_buffers(color, xValues, yValues);
}, py::arg("color"), py::arg("x_values"), py::arg("y_values"));
m.def("plot_xy",
[](Color color, const nb::ndarray<> & xValues, const nb::ndarray<> & yValues)
{
auto PlotXY_adapt_c_buffers = [](Color color, const nb::ndarray<> & xValues, const nb::ndarray<> & yValues)
{
// Check if the array is 1D and C-contiguous
if (! (xValues.ndim() == 1 && xValues.stride(0) == 1))
throw std::runtime_error("The array must be 1D and contiguous");
// convert nb::ndarray to C standard buffer (const)
const void * xValues_from_pyarray = xValues.data();
size_t xValues_count = xValues.shape(0);
// Check if the array is 1D and C-contiguous
if (! (yValues.ndim() == 1 && yValues.stride(0) == 1))
throw std::runtime_error("The array must be 1D and contiguous");
// convert nb::ndarray to C standard buffer (const)
const void * yValues_from_pyarray = yValues.data();
size_t yValues_count = yValues.shape(0);
using np_uint_l = uint64_t;
using np_int_l = int64_t;
// Define a lambda to compute the letter code for the buffer type
auto _nanobind_buffer_type_to_letter_code = [](uint8_t dtype_code, size_t sizeof_item) -> char
{
#define DCODE(T) static_cast<uint8_t>(nb::dlpack::dtype_code::T)
const std::array<std::tuple<uint8_t, size_t, char>, 11> mappings = {{
{DCODE(UInt), 1, 'B'}, {DCODE(UInt), 2, 'H'}, {DCODE(UInt), 4, 'I'}, {DCODE(UInt), 8, 'L'},
{DCODE(Int), 1, 'b'}, {DCODE(Int), 2, 'h'}, {DCODE(Int), 4, 'i'}, {DCODE(Int), 8, 'l'},
{DCODE(Float), 4, 'f'}, {DCODE(Float), 8, 'd'}, {DCODE(Float), 16, 'g'}
}};
#undef DCODE
for (const auto& [code_val, size, letter] : mappings)
if (code_val == dtype_code && size == sizeof_item)
return letter;
throw std::runtime_error("Unsupported dtype");
};
// Compute the letter code for the buffer type
uint8_t dtype_code_yValues = yValues.dtype().code;
size_t sizeof_item_yValues = yValues.dtype().bits / 8;
char yValues_type = _nanobind_buffer_type_to_letter_code(dtype_code_yValues, sizeof_item_yValues);
// call the correct template version by casting
if (yValues_type == 'B')
PlotXY(color, static_cast<const uint8_t *>(xValues_from_pyarray), static_cast<const uint8_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'b')
PlotXY(color, static_cast<const int8_t *>(xValues_from_pyarray), static_cast<const int8_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'H')
PlotXY(color, static_cast<const uint16_t *>(xValues_from_pyarray), static_cast<const uint16_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'h')
PlotXY(color, static_cast<const int16_t *>(xValues_from_pyarray), static_cast<const int16_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'I')
PlotXY(color, static_cast<const uint32_t *>(xValues_from_pyarray), static_cast<const uint32_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'i')
PlotXY(color, static_cast<const int32_t *>(xValues_from_pyarray), static_cast<const int32_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'L')
PlotXY(color, static_cast<const np_uint_l *>(xValues_from_pyarray), static_cast<const np_uint_l *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'l')
PlotXY(color, static_cast<const np_int_l *>(xValues_from_pyarray), static_cast<const np_int_l *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'f')
PlotXY(color, static_cast<const float *>(xValues_from_pyarray), static_cast<const float *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'd')
PlotXY(color, static_cast<const double *>(xValues_from_pyarray), static_cast<const double *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'g')
PlotXY(color, static_cast<const long double *>(xValues_from_pyarray), static_cast<const long double *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
else if (yValues_type == 'q')
PlotXY(color, static_cast<const long long *>(xValues_from_pyarray), static_cast<const long long *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));
// If we reach this point, the array type is not supported!
else
throw std::runtime_error(std::string("Bad array type ('") + yValues_type + "') for param yValues");
};
PlotXY_adapt_c_buffers(color, xValues, yValues);
}, nb::arg("color"), nb::arg("x_values"), nb::arg("y_values"));
Vectorize functions#
See relevant portion of the pybind11 manual. This feature is not available with nanobind
Within litgen, you can set:
Which namespaces are candidates for vectorization (options.fn_namespace_vectorize__regex. Set it to
r".*"for all namespaces)Which function names are candidates for vectorization
Which optional suffix or prefix will be added to the vectorized functions
cpp_code = """
namespace MathFunctions
{
double fn1(double x, double y);
double fn2(double x);
}
"""
options = litgen.LitgenOptions()
options.fn_namespace_vectorize__regex = "^MathFunctions$"
options.fn_vectorize__regex = r".*"
options.fn_vectorize_suffix = "_v"
litgen_demo.demo(options, cpp_code)
namespace MathFunctions
{
double fn1(double x, double y);
double fn2(double x);
}
# <submodule math_functions>
class math_functions: # Proxy class that introduces typings for the *submodule* math_functions
pass # (This corresponds to a C++ namespace. All method are static!)
@staticmethod
def fn1(x: float, y: float) -> float:
pass
@staticmethod
def fn1_v(x: np.ndarray, y: np.ndarray) -> np.ndarray:
pass
@staticmethod
def fn2(x: float) -> float:
pass
@staticmethod
def fn2_v(x: np.ndarray) -> np.ndarray:
pass
# </submodule math_functions>
{ // <namespace MathFunctions>
py::module_ pyNsMathFunctions = m.def_submodule("math_functions", "");
pyNsMathFunctions.def("fn1",
MathFunctions::fn1, py::arg("x"), py::arg("y"));
pyNsMathFunctions.def("fn1_v",
py::vectorize(MathFunctions::fn1), py::arg("x"), py::arg("y"));
pyNsMathFunctions.def("fn2",
MathFunctions::fn2, py::arg("x"));
pyNsMathFunctions.def("fn2_v",
py::vectorize(MathFunctions::fn2), py::arg("x"));
} // </namespace MathFunctions>
{ // <namespace MathFunctions>
nb::module_ pyNsMathFunctions = m.def_submodule("math_functions", "");
pyNsMathFunctions.def("fn1",
MathFunctions::fn1, nb::arg("x"), nb::arg("y"));
pyNsMathFunctions.def("fn2",
MathFunctions::fn2, nb::arg("x"));
} // </namespace MathFunctions>
Accepting args and kwargs#
Relevant portion of the pybind11 manual and the nanobind manual
litgen will automatically detect signatures with params which look like (py::args args, const py::kwargs& kwargs) or (nb::args args, const nb::kwargs& kwargs) and adapt the python stub signature accordingly.
cpp_code = """
void generic_pybind(py::args args, const py::kwargs& kwargs)
{
/// .. do something with args
// if (kwargs)
/// .. do something with kwargs
}
void generic_nanobind(nb::args args, const nb::kwargs& kwargs)
{
/// .. do something with args
// if (kwargs)
/// .. do something with kwargs
}
"""
options = litgen.LitgenOptions()
litgen_demo.demo(options, cpp_code)
void generic_pybind(py::args args, const py::kwargs& kwargs)
{
/// .. do something with args
// if (kwargs)
/// .. do something with kwargs
}
void generic_nanobind(nb::args args, const nb::kwargs& kwargs)
{
/// .. do something with args
// if (kwargs)
/// .. do something with kwargs
}
def generic_pybind(*args, **kwargs) -> None:
pass
def generic_nanobind(*args, **kwargs) -> None:
pass
m.def("generic_pybind",
generic_pybind);
m.def("generic_nanobind",
generic_nanobind);
m.def("generic_pybind",
generic_pybind);
m.def("generic_nanobind",
generic_nanobind);
Force overload#
Relevant portion of the pybind11 manual and the nanobind manual
Automatic overload#
If litgen detect two overload, it will add a call to py::overload_cast automatically:
cpp_code = """
void foo(int x);
void foo(double x);
"""
options = litgen.LitgenOptions()
litgen_demo.demo(options, cpp_code)
void foo(int x);
void foo(double x);
@overload
def foo(x: int) -> None:
pass
@overload
def foo(x: float) -> None:
pass
m.def("foo",
py::overload_cast<int>(foo), py::arg("x"));
m.def("foo",
py::overload_cast<double>(foo), py::arg("x"));
m.def("foo",
nb::overload_cast<int>(foo), nb::arg("x"));
m.def("foo",
nb::overload_cast<double>(foo), nb::arg("x"));
Manual overload#
However, in some cases, you might want to add it manually: use options.fn_force_overload__regex
cpp_code = """
void foo2(int x);
"""
options = litgen.LitgenOptions()
options.fn_force_overload__regex += r"|^foo2$"
litgen_demo.demo(options, cpp_code, show_pydef=True)
void foo2(int x);
def foo2(x: int) -> None:
pass
m.def("foo2",
py::overload_cast<int>(foo2), py::arg("x"));
m.def("foo2",
nb::overload_cast<int>(foo2), nb::arg("x"));
Force usage of a lambda function#
In some rare cases, the usage of py::overload_cast might not be sufficient to discriminate the overload. In this case, you can tell litgen to disambiguate it via a lambda function. Look at the pybind C++ binding code below:
cpp_code = """
void foo3(int x);
"""
options = litgen.LitgenOptions()
options.fn_force_lambda__regex += r"|^foo3$"
litgen_demo.demo(options, cpp_code, show_pydef=True)
void foo3(int x);
def foo3(x: int) -> None:
pass
m.def("foo3",
[](int x)
{
auto foo3_adapt_force_lambda = [](int x)
{
foo3(x);
};
foo3_adapt_force_lambda(x);
}, py::arg("x"));
m.def("foo3",
[](int x)
{
auto foo3_adapt_force_lambda = [](int x)
{
foo3(x);
};
foo3_adapt_force_lambda(x);
}, nb::arg("x"));