{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Functions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are numerous generations options that can be set in order to change function bindings options.\n", "\n", "See [options.py](https://github.com/pthom/litgen/blob/main/src/litgen/options.py): all the function related options begin wth `fn_` or `fn_params` (when they deal with function parameters)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exclude functions and/or params" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Extract from options.py, showing the related options:\n", "\n", "```python\n", " ################################################################################\n", " # \n", " ################################################################################\n", "\n", " # Exclude certain functions and methods by a regex on their name\n", " fn_exclude_by_name__regex: str = \"\"\n", "\n", " # Exclude certain functions and methods by a regex on any of their parameter type and/or return type\n", " # (those should be decorated type)\n", " # For example:\n", " # options.fn_exclude_by_param_type__regex = \"^char\\s*$|^unsigned\\s+char$|Callback$\"\n", " # would exclude all functions having params of type \"char *\", \"unsigned char\", \"xxxCallback\"\n", " #\n", " # Note: this is distinct from `fn_params_exclude_types__regex` which removes params\n", " # from the function signature, but not the function itself.\n", " fn_exclude_by_param_type__regex: str = \"\"\n", "\n", " # ------------------------------------------------------------------------------\n", " # Exclude some params by name or type\n", " # ------------------------------------------------------------------------------\n", " # Remove some params from the python published interface. A param can only be removed if it has a default value\n", " # in the C++ signature\n", " fn_params_exclude_names__regex: str = \"\"\n", " fn_params_exclude_types__regex: str = \"\"\n", "```\n", "\n", "**As an example, let's consider the code below, where we would want to:**\n", "\n", "* exclude all functions beginning with \"priv_\"\n", "* exclude a function parameter if its type name starts with \"Private\"" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import litgen\n", "from litgen.demo import litgen_demo\n", "\n", "cpp_code = \"\"\"\n", "void priv_SetOptions(bool v);\n", "\n", "void SetOptions(const PublicOptions& options, const PrivateOptions& privateOptions = PrivateOptions());\n", "\"\"\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default the generated code will be:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
void priv_SetOptions(bool v);\n",
       "\n",
       "void SetOptions(const PublicOptions& options, const PrivateOptions& privateOptions = PrivateOptions());\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
def priv_set_options(v: bool) -> None:\n",
       "    pass\n",
       "\n",
       "def set_options(\n",
       "    options: PublicOptions,\n",
       "    private_options: PrivateOptions = PrivateOptions()\n",
       "    ) -> None:\n",
       "    pass\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("priv_set_options",\n",
       "    priv_SetOptions, py::arg("v"));\n",
       "\n",
       "m.def("set_options",\n",
       "    SetOptions, py::arg("options"), py::arg("private_options") = PrivateOptions());\n",
       "
\n", "\n", "
\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "options = litgen.LitgenOptions()\n", "litgen_demo.demo(options, cpp_code)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But we can set some options to change this.\n", "\n", "_In the generated code below, look closely at the C++ binding code: you will see that it takes steps to generate a default value for the parameter of type `PrivateOptions`_" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
void priv_SetOptions(bool v);\n",
       "\n",
       "void SetOptions(const PublicOptions& options, const PrivateOptions& privateOptions = PrivateOptions());\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
def set_options(options: PublicOptions) -> None:\n",
       "    pass\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("set_options",\n",
       "    [](const PublicOptions & options)\n",
       "    {\n",
       "        auto SetOptions_adapt_exclude_params = [](const PublicOptions & options)\n",
       "        {\n",
       "            SetOptions(options, PrivateOptions());\n",
       "        };\n",
       "\n",
       "        SetOptions_adapt_exclude_params(options);\n",
       "    },     py::arg("options"));\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "options = litgen.LitgenOptions()\n", "options.fn_exclude_by_name__regex = \"^priv_\" # Exclude functions whose name begin by \"priv_\"\n", "options.fn_params_exclude_types__regex = \"Private\" # Exclude functions params whose type name contains \"Private\"\n", "\n", "litgen_demo.demo(options, cpp_code, show_pydef=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Return policy\n", "\n", "See [relevant doc from pybind11](https://pybind11.readthedocs.io/en/stable/advanced/functions.html):\n", "\n", "> Python and C++ use fundamentally different ways of managing the memory and lifetime of objects managed by them. This can lead to issues when creating bindings for functions that return a non-trivial type. Just by looking at the type information, it is not clear whether Python should take charge of the returned value and eventually free its resources, or if this is handled on the C++ side. For this reason, pybind11 provides a several return value policy annotations that can be passed to the `module_::def()` and `class_::def()` functions. The default policy is `return_value_policy::automatic`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### return_value_policy::reference\n", "\n", "In the C++ code below, let's suppose that C++ is responsible for handling the destruction of the values returned by MakeWidget and MakeFoo: we do not want python to call the destructor automatically. " ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "cpp_code = \"\"\"\n", "Widget * MakeWidget();\n", "Foo& MakeFoo();\n", "\"\"\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In that case, we can set `options.fn_return_force_policy_reference_for_pointers__regex` and/or `options.fn_return_force_policy_reference_for_references__regex`, and the generated pydef binding code, will set the correct return value policy." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("make_widget",\n",
       "    MakeWidget, \n",
       "    "return_value_policy::take_ownership", \n",
       "    pybind11::return_value_policy::reference);\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "options = litgen.LitgenOptions()\n", "options.fn_return_force_policy_reference_for_pointers__regex = r\"^Make\"\n", "options.fn_return_force_policy_reference_for_references__regex = r\"^Make\"\n", "generated_code = litgen.generate_code(options, cpp_code)\n", "litgen_demo.show_cpp_code(generated_code.pydef_code)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Custom return value policy\n", "\n", "If you annotate the function declaration with `return_value_policy::...`, then the generator will use this information:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("make_widget",\n",
       "    MakeWidget, "return_value_policy::take_ownership");\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"\"\"\n", "Widget *MakeWidget(); // return_value_policy::take_ownership\n", "\"\"\"\n", "\n", "options = litgen.LitgenOptions()\n", "generated_code = litgen.generate_code(options, cpp_code)\n", "litgen_demo.show_cpp_code(generated_code.pydef_code)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Modifiable immutable function params\n", "\n", "Some C++ functions may use a modifiable input/output parameter, for which the corresponding type in python is **immutable** (e.g. its is a numeric type, or a string).\n", "\n", "For example, in the C++ code below, the param `inOutFlag` is modified by the function.\n", "```cpp\n", "void SwitchBool(bool* inOutFlag);\n", "``````\n", "\n", "In python, a function with the following signature **can not** change its parameter value, since bool is immutable:\n", "\n", "```python\n", "def switch_bool(in_out_v: bool) -> None:\n", " pass\n", "``````\n", "\n", "litgen offers two ways to handle those situations:\n", "* by using boxed types\n", "* by adding the modified value to the function output" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Using boxed types\n", "\n", "You can decide to replace this kind of parameters type by a \"Boxed\" type: this is a simple class that encapsulates the value, and makes it modifiable.\n", "\n", "Look at the example below where a `BoxedBool` is created: \n", "* its python signature is given in the stub\n", "* its C++ declaration is given in the glue code\n", "* the pybind11 C++ binding code handle the conversion between `bool *` and `BoxedBool`" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
void SwitchBool(bool* inOutFlag);\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
####################    <generated_from:BoxedTypes>    ####################\n",
       "class BoxedBool:\n",
       "    value: bool\n",
       "    def __init__(self, v: bool = False) -> None:\n",
       "        pass\n",
       "    def __repr__(self) -> str:\n",
       "        pass\n",
       "####################    </generated_from:BoxedTypes>    ####################\n",
       "\n",
       "\n",
       "def switch_bool(in_out_flag: BoxedBool) -> None:\n",
       "    pass\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
////////////////////    <generated_from:BoxedTypes>    ////////////////////\n",
       "auto pyClassBoxedBool =\n",
       "    py::class_<BoxedBool>\n",
       "        (m, "BoxedBool", "")\n",
       "    .def_readwrite("value", &BoxedBool::value, "")\n",
       "    .def(py::init<bool>(),\n",
       "        py::arg("v") = false)\n",
       "    .def("__repr__",\n",
       "        &BoxedBool::__repr__)\n",
       "    ;\n",
       "////////////////////    </generated_from:BoxedTypes>    ////////////////////\n",
       "\n",
       "\n",
       "m.def("switch_bool",\n",
       "    [](BoxedBool & inOutFlag)\n",
       "    {\n",
       "        auto SwitchBool_adapt_modifiable_immutable = [](BoxedBool & inOutFlag)\n",
       "        {\n",
       "            bool * inOutFlag_boxed_value = & (inOutFlag.value);\n",
       "\n",
       "            SwitchBool(inOutFlag_boxed_value);\n",
       "        };\n",
       "\n",
       "        SwitchBool_adapt_modifiable_immutable(inOutFlag);\n",
       "    },     py::arg("in_out_flag"));\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
struct BoxedBool\n",
       "{\n",
       "    bool value;\n",
       "    BoxedBool(bool v = false) : value(v) {}\n",
       "    std::string __repr__() const { return std::string("BoxedBool(") + std::to_string(value) + ")"; }\n",
       "};\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"void SwitchBool(bool* inOutFlag);\"\n", "options = litgen.LitgenOptions()\n", "options.fn_params_replace_modifiable_immutable_by_boxed__regex = r\".*\" # \"Box\" all modifiable immutable parameters\n", "litgen_demo.demo(options, cpp_code, show_pydef=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Adding the modified value to the function output\n", "\n", "Let's say that we have a C++ function that modifies a string, and returns a bool that indicates whether it was modified:\n", "\n", "```cpp\n", "bool UserInputString(std::string* inOutStr);\n", "```\n", "\n", "We can ask litgen to add the modified string to the output of the function. \n", "\n", "Look at the example below: \n", "\n", "* the python function returns a `Tuple[bool, str]`\n", "* the pybind11 binding adds a lambda that does the necessary transformation\n" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
bool UserInputString(std::string* inOutStr);\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
def user_input_string(in_out_str: str) -> Tuple[bool, str]:\n",
       "    pass\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("user_input_string",\n",
       "    [](std::string inOutStr) -> std::tuple<bool, std::string>\n",
       "    {\n",
       "        auto UserInputString_adapt_modifiable_immutable_to_return = [](std::string inOutStr) -> std::tuple<bool, std::string>\n",
       "        {\n",
       "            std::string * inOutStr_adapt_modifiable = & inOutStr;\n",
       "\n",
       "            bool r = UserInputString(inOutStr_adapt_modifiable);\n",
       "            return std::make_tuple(r, inOutStr);\n",
       "        };\n",
       "\n",
       "        return UserInputString_adapt_modifiable_immutable_to_return(inOutStr);\n",
       "    },     py::arg("in_out_str"));\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"bool UserInputString(std::string* inOutStr);\"\n", "options = litgen.LitgenOptions()\n", "options.fn_params_output_modifiable_immutable_to_return__regex = r\".*\"\n", "litgen_demo.demo(options, cpp_code, show_pydef=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## C style function params\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Immutable C array param\n", "\n", "If a function uses a param whose type is `const SomeType[N]`, then it will be translated automatically, and the pybind11 binding code will handle the necessary transformations." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
void foo(const int v[2]);\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
def foo(v: List[int]) -> None:\n",
       "    pass\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("foo",\n",
       "    [](const std::array<int, 2>& v)\n",
       "    {\n",
       "        auto foo_adapt_fixed_size_c_arrays = [](const std::array<int, 2>& v)\n",
       "        {\n",
       "            foo(v.data());\n",
       "        };\n",
       "\n",
       "        foo_adapt_fixed_size_c_arrays(v);\n",
       "    },     py::arg("v"));\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"void foo(const int v[2]);\"\n", "options = litgen.LitgenOptions()\n", "litgen_demo.demo(options, cpp_code, show_pydef=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Modifiable C array param\n", "\n", "If a function uses a param whose type is `SomeType[N] v`, then litgen will understand that any value inside `v` can be modified, and it will emit code where a C++ signature like this:\n", "\n", "```cpp\n", "void foo(int v[2]);\n", "```\n", "\n", "is transformed into python:\n", "\n", "```python\n", "def foo(v_0: BoxedInt, v_1: BoxedInt) -> None:\n", " pass\n", "```" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
void foo(int v[2]);\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
####################    <generated_from:BoxedTypes>    ####################\n",
       "class BoxedInt:\n",
       "    value: int\n",
       "    def __init__(self, v: int = 0) -> None:\n",
       "        pass\n",
       "    def __repr__(self) -> str:\n",
       "        pass\n",
       "####################    </generated_from:BoxedTypes>    ####################\n",
       "\n",
       "\n",
       "def foo(v_0: BoxedInt, v_1: BoxedInt) -> None:\n",
       "    pass\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
////////////////////    <generated_from:BoxedTypes>    ////////////////////\n",
       "auto pyClassBoxedInt =\n",
       "    py::class_<BoxedInt>\n",
       "        (m, "BoxedInt", "")\n",
       "    .def_readwrite("value", &BoxedInt::value, "")\n",
       "    .def(py::init<int>(),\n",
       "        py::arg("v") = 0)\n",
       "    .def("__repr__",\n",
       "        &BoxedInt::__repr__)\n",
       "    ;\n",
       "////////////////////    </generated_from:BoxedTypes>    ////////////////////\n",
       "\n",
       "\n",
       "m.def("foo",\n",
       "    [](BoxedInt & v_0, BoxedInt & v_1)\n",
       "    {\n",
       "        auto foo_adapt_fixed_size_c_arrays = [](BoxedInt & v_0, BoxedInt & v_1)\n",
       "        {\n",
       "            int v_raw[2];\n",
       "            v_raw[0] = v_0.value;\n",
       "            v_raw[1] = v_1.value;\n",
       "\n",
       "            foo(v_raw);\n",
       "\n",
       "            v_0.value = v_raw[0];\n",
       "            v_1.value = v_raw[1];\n",
       "        };\n",
       "\n",
       "        foo_adapt_fixed_size_c_arrays(v_0, v_1);\n",
       "    },     py::arg("v_0"), py::arg("v_1"));\n",
       "
\n", "\n", "
\n", " \n", " \n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
struct BoxedInt\n",
       "{\n",
       "    int value;\n",
       "    BoxedInt(int v = 0) : value(v) {}\n",
       "    std::string __repr__() const { return std::string("BoxedInt(") + std::to_string(value) + ")"; }\n",
       "};\n",
       "
\n", "\n", "
\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"void foo(int v[2]);\"\n", "options = litgen.LitgenOptions()\n", "litgen_demo.demo(options, cpp_code)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### C style string list\n", "\n", "If a pair of function params look like `const char * const items[], int item_count`, it will be transformed into a python `List[str]`:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
void PrintItems(const char * const items[], int item_count);\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
def print_items(items: List[str]) -> None:\n",
       "    pass\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("print_items",\n",
       "    [](const std::vector<std::string> & items)\n",
       "    {\n",
       "        auto PrintItems_adapt_c_string_list = [](const std::vector<std::string> & items)\n",
       "        {\n",
       "            std::vector<const char *> items_ptrs;\n",
       "            for (const auto& v: items)\n",
       "                items_ptrs.push_back(v.c_str());\n",
       "            int item_count = static_cast<int>(items.size());\n",
       "\n",
       "            PrintItems(items_ptrs.data(), item_count);\n",
       "        };\n",
       "\n",
       "        PrintItems_adapt_c_string_list(items);\n",
       "    },     py::arg("items"));\n",
       "
\n", "\n", "
\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"void PrintItems(const char * const items[], int item_count);\"\n", "options = litgen.LitgenOptions()\n", "options.fn_params_replace_c_string_list__regex = r\".*\" # apply to all function names (this is the default!)\n", "litgen_demo.demo(options, cpp_code)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### C style variadic string format\n", "\n", "If a function uses a pair of parameters like `char const* const format, ...`, then litgen will transform it into a simple python string." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
void Log(LogLevel level, char const* const format, ...);\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
def log(level: LogLevel, format: str) -> None:\n",
       "    pass\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("log",\n",
       "    [](LogLevel level, const char * const format)\n",
       "    {\n",
       "        auto Log_adapt_variadic_format = [](LogLevel level, const char * const format)\n",
       "        {\n",
       "            Log(level, "%s", format);\n",
       "        };\n",
       "\n",
       "        Log_adapt_variadic_format(level, format);\n",
       "    },     py::arg("level"), py::arg("format"));\n",
       "
\n", "\n", "
\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"void Log(LogLevel level, char const* const format, ...);\"\n", "options = litgen.LitgenOptions()\n", "litgen_demo.demo(options, cpp_code)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Passing numeric buffers to numpy\n", "\n", "### Simple numeric buffers\n", "\n", "If a function uses a pair (a more) of parameters which look like `(double *values, int count)`, or `(const float* values, int nb)` (etc.), then litgen can transform this parameter into a numpy array.\n", "\n", "Let's see an example with this function:\n", "```cpp\n", "void PlotXY(const float *xValues, const float *yValues, size_t how_many);\n", "```\n", "\n", "We would like it to be published as:\n", "\n", "```python\n", "def plot_xy(x_values: np.ndarray, y_values: np.ndarray) -> None:\n", " pass\n", "```\n", "\n", "We will need to tell litgen:\n", "* Which function are concerned (options.fn_params_replace_buffer_by_array__regex)\n", "* The name of the the \"count\" param if it is not a standard one (count, nb, etc)\n", "\n", "_Note: if you look at the pybind11 C++ binding code, you will see that litgen handles the transformation, and ensures that the types are correct._" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
void PlotXY(const float *xValues, const float *yValues, size_t how_many);\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
def plot_xy(x_values: np.ndarray, y_values: np.ndarray) -> None:\n",
       "    pass\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("plot_xy",\n",
       "    [](const py::array & xValues, const py::array & yValues)\n",
       "    {\n",
       "        auto PlotXY_adapt_c_buffers = [](const py::array & xValues, const py::array & yValues)\n",
       "        {\n",
       "            // convert py::array to C standard buffer (const)\n",
       "            const void * xValues_from_pyarray = xValues.data();\n",
       "            py::ssize_t xValues_count = xValues.shape()[0];\n",
       "            char xValues_type = xValues.dtype().char_();\n",
       "            if (xValues_type != 'f')\n",
       "                throw std::runtime_error(std::string(R"msg(\n",
       "                        Bad type!  Expected a numpy array of native type:\n",
       "                                    const float *\n",
       "                                Which is equivalent to\n",
       "                                    f\n",
       "                                (using py::array::dtype().char_() as an id)\n",
       "                    )msg"));\n",
       "\n",
       "            // convert py::array to C standard buffer (const)\n",
       "            const void * yValues_from_pyarray = yValues.data();\n",
       "            py::ssize_t yValues_count = yValues.shape()[0];\n",
       "            char yValues_type = yValues.dtype().char_();\n",
       "            if (yValues_type != 'f')\n",
       "                throw std::runtime_error(std::string(R"msg(\n",
       "                        Bad type!  Expected a numpy array of native type:\n",
       "                                    const float *\n",
       "                                Which is equivalent to\n",
       "                                    f\n",
       "                                (using py::array::dtype().char_() as an id)\n",
       "                    )msg"));\n",
       "\n",
       "            PlotXY(static_cast<const float *>(xValues_from_pyarray), static_cast<const float *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));\n",
       "        };\n",
       "\n",
       "        PlotXY_adapt_c_buffers(xValues, yValues);\n",
       "    },     py::arg("x_values"), py::arg("y_values"));\n",
       "
\n", "\n", "
\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"\"\"\n", "void PlotXY(const float *xValues, const float *yValues, size_t how_many);\n", "\"\"\"\n", "options = litgen.LitgenOptions()\n", "options.fn_params_replace_buffer_by_array__regex = r\"^Plot\"\n", "options.fn_params_buffer_size_names__regex += \"|how_many\"\n", "litgen_demo.demo(options, cpp_code)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Template numeric buffers\n", "\n", "If a template function uses a pair of parameters whose signature looks like `(const T* values, int count)`, then it can be transformed into a numpy array.\n", "\n", "In the example below, we would like the following C++ function:\n", "\n", "```cpp\n", "template void PlotXY(Color color, const NumberType *xValues, const NumberType *yValues, size_t count);\n", "````\n", "\n", "To be published as:\n", "\n", "```python\n", "def plot_xy(color: Color, x_values: np.ndarray, y_values: np.ndarray) -> None:\n", " pass\n", "```\n", "\n", "For this we need to:\n", "* Set which function names are concerned (options.fn_params_replace_buffer_by_array__regex)\n", "* Optionally, add the name of the template param (options.fn_params_buffer_template_types)\n", "\n", "\n", "_Note: if you look at the generated pybind11 C++ binding code, you will see that it handles all numeric types. This is a very efficient way to transmit numeric buffers of all types to python_ " ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
    template<typename NumberType> \n",
       "    void PlotXY(Color color, const NumberType *xValues, const NumberType *yValues, size_t count);\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
def plot_xy(color: Color, x_values: np.ndarray, y_values: np.ndarray) -> None:\n",
       "    pass\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("plot_xy",\n",
       "    [](Color color, const py::array & xValues, const py::array & yValues)\n",
       "    {\n",
       "        auto PlotXY_adapt_c_buffers = [](Color color, const py::array & xValues, const py::array & yValues)\n",
       "        {\n",
       "            // convert py::array to C standard buffer (const)\n",
       "            const void * xValues_from_pyarray = xValues.data();\n",
       "            py::ssize_t xValues_count = xValues.shape()[0];\n",
       "\n",
       "            // convert py::array to C standard buffer (const)\n",
       "            const void * yValues_from_pyarray = yValues.data();\n",
       "            py::ssize_t yValues_count = yValues.shape()[0];\n",
       "\n",
       "            #ifdef _WIN32\n",
       "            using np_uint_l = uint32_t;\n",
       "            using np_int_l = int32_t;\n",
       "            #else\n",
       "            using np_uint_l = uint64_t;\n",
       "            using np_int_l = int64_t;\n",
       "            #endif\n",
       "            // call the correct template version by casting\n",
       "            char yValues_type = yValues.dtype().char_();\n",
       "            if (yValues_type == 'B')\n",
       "                PlotXY(color, static_cast<const uint8_t *>(xValues_from_pyarray), static_cast<const uint8_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));\n",
       "            else if (yValues_type == 'b')\n",
       "                PlotXY(color, static_cast<const int8_t *>(xValues_from_pyarray), static_cast<const int8_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));\n",
       "            else if (yValues_type == 'H')\n",
       "                PlotXY(color, static_cast<const uint16_t *>(xValues_from_pyarray), static_cast<const uint16_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));\n",
       "            else if (yValues_type == 'h')\n",
       "                PlotXY(color, static_cast<const int16_t *>(xValues_from_pyarray), static_cast<const int16_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));\n",
       "            else if (yValues_type == 'I')\n",
       "                PlotXY(color, static_cast<const uint32_t *>(xValues_from_pyarray), static_cast<const uint32_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));\n",
       "            else if (yValues_type == 'i')\n",
       "                PlotXY(color, static_cast<const int32_t *>(xValues_from_pyarray), static_cast<const int32_t *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));\n",
       "            else if (yValues_type == 'L')\n",
       "                PlotXY(color, static_cast<const np_uint_l *>(xValues_from_pyarray), static_cast<const np_uint_l *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));\n",
       "            else if (yValues_type == 'l')\n",
       "                PlotXY(color, static_cast<const np_int_l *>(xValues_from_pyarray), static_cast<const np_int_l *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));\n",
       "            else if (yValues_type == 'f')\n",
       "                PlotXY(color, static_cast<const float *>(xValues_from_pyarray), static_cast<const float *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));\n",
       "            else if (yValues_type == 'd')\n",
       "                PlotXY(color, static_cast<const double *>(xValues_from_pyarray), static_cast<const double *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));\n",
       "            else if (yValues_type == 'g')\n",
       "                PlotXY(color, static_cast<const long double *>(xValues_from_pyarray), static_cast<const long double *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));\n",
       "            else if (yValues_type == 'q')\n",
       "                PlotXY(color, static_cast<const long long *>(xValues_from_pyarray), static_cast<const long long *>(yValues_from_pyarray), static_cast<size_t>(yValues_count));\n",
       "            // If we reach this point, the array type is not supported!\n",
       "            else\n",
       "                throw std::runtime_error(std::string("Bad array type ('") + yValues_type + "') for param yValues");\n",
       "        };\n",
       "\n",
       "        PlotXY_adapt_c_buffers(color, xValues, yValues);\n",
       "    },     py::arg("color"), py::arg("x_values"), py::arg("y_values"));\n",
       "
\n", "\n", "
\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"\"\"\n", " template \n", " void PlotXY(Color color, const NumberType *xValues, const NumberType *yValues, size_t count);\n", "\"\"\"\n", "options = litgen.LitgenOptions()\n", "options.fn_params_replace_buffer_by_array__regex = r\"^Plot\"\n", "options.fn_params_buffer_template_types += \"|NumberType\"\n", "litgen_demo.demo(options, cpp_code, height=80)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Vectorize functions\n", "\n", "See [relevant portion of the pybind11 manual](https://pybind11.readthedocs.io/en/stable/advanced/pycpp/numpy.html#vectorizing-functions).\n", "\n", "Within litgen, you can set:\n", "\n", "* Which namespaces are candidates for vectorization (options.fn_namespace_vectorize__regex. Set it to `r\".*\"` for all namespaces)\n", "* Which function names are candidates for vectorization\n", "* Which optional suffix or prefix will be added to the vectorized functions " ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
    namespace MathFunctions\n",
       "    {\n",
       "        double fn1(double x, double y);\n",
       "        double fn2(double x);\n",
       "    }\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
# <submodule math_functions>\n",
       "class math_functions:  # Proxy class that introduces typings for the *submodule* math_functions\n",
       "    pass  # (This corresponds to a C++ namespace. All method are static!)\n",
       "    @staticmethod\n",
       "    def fn1(x: float, y: float) -> float:\n",
       "        pass\n",
       "    @staticmethod\n",
       "    def fn1_v(x: np.ndarray, y: np.ndarray) -> np.ndarray:\n",
       "        pass\n",
       "    @staticmethod\n",
       "    def fn2(x: float) -> float:\n",
       "        pass\n",
       "    @staticmethod\n",
       "    def fn2_v(x: np.ndarray) -> np.ndarray:\n",
       "        pass\n",
       "\n",
       "# </submodule math_functions>\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
{ // <namespace MathFunctions>\n",
       "    py::module_ pyNsMathFunctions = m.def_submodule("math_functions", "");\n",
       "    pyNsMathFunctions.def("fn1",\n",
       "        MathFunctions::fn1, py::arg("x"), py::arg("y"));\n",
       "    pyNsMathFunctions.def("fn1_v",\n",
       "        py::vectorize(MathFunctions::fn1), py::arg("x"), py::arg("y"));\n",
       "\n",
       "    pyNsMathFunctions.def("fn2",\n",
       "        MathFunctions::fn2, py::arg("x"));\n",
       "    pyNsMathFunctions.def("fn2_v",\n",
       "        py::vectorize(MathFunctions::fn2), py::arg("x"));\n",
       "} // </namespace MathFunctions>\n",
       "
\n", "\n", "
\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"\"\"\n", " namespace MathFunctions\n", " {\n", " double fn1(double x, double y);\n", " double fn2(double x);\n", " }\n", "\"\"\"\n", "options = litgen.LitgenOptions()\n", "options.fn_namespace_vectorize__regex = \"^MathFunctions$\"\n", "options.fn_vectorize__regex = r\".*\"\n", "options.fn_vectorize_suffix = \"_v\"\n", "litgen_demo.demo(options, cpp_code)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Accepting args and kwargs\n", "\n", "[Relevant portion](https://pybind11.readthedocs.io/en/stable/advanced/functions.html#accepting-args-and-kwargs) of the pybind11 manual\n", "\n", "litgen will automatically detect signatures with params which look like `(py::args args, const py::kwargs& kwargs)` and adapt the python stub signature accordingly." ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
 void generic(py::args args, const py::kwargs& kwargs)\n",
       "    {\n",
       "        /// .. do something with args\n",
       "        // if (kwargs)\n",
       "            /// .. do something with kwargs\n",
       "    }\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
def generic(*args, **kwargs) -> None:\n",
       "    pass\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("generic",\n",
       "    generic);\n",
       "
\n", "\n", "
\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"\"\"\n", " void generic(py::args args, const py::kwargs& kwargs)\n", " {\n", " /// .. do something with args\n", " // if (kwargs)\n", " /// .. do something with kwargs\n", " }\n", "\"\"\"\n", "options = litgen.LitgenOptions()\n", "litgen_demo.demo(options, cpp_code)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Force overload\n", "\n", "[Relevant portion](https://pybind11.readthedocs.io/en/stable/classes.html?highlight=overload_cast#overloaded-methods) of the pybind11 manual.\n", "\n", "### Automatic overload\n", "\n", "If litgen detect two overload, it will add a call to `py::overload_cast` automatically:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", "\n", "\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
void foo(int x);\n",
       "void foo(double x);\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", " \n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
@overload\n",
       "def foo(x: int) -> None:\n",
       "    pass\n",
       "@overload\n",
       "def foo(x: float) -> None:\n",
       "    pass\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " \n", "
\n", "
\n", "
\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("foo",\n",
       "    py::overload_cast<int>(foo), py::arg("x"));\n",
       "\n",
       "m.def("foo",\n",
       "    py::overload_cast<double>(foo), py::arg("x"));\n",
       "
\n", "\n", "
\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"\"\"\n", "void foo(int x);\n", "void foo(double x);\n", "\"\"\"\n", "options = litgen.LitgenOptions()\n", "litgen_demo.demo(options, cpp_code)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Manual overload\n", "\n", "However, in some cases, you might want to add it manually: use `options.fn_force_overload__regex `" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("foo2",\n",
       "    py::overload_cast<int>(foo2), py::arg("x"));\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"\"\"\n", "void foo2(int x);\n", "\"\"\"\n", "options = litgen.LitgenOptions()\n", "options.fn_force_overload__regex += r\"|^foo2$\"\n", "generated_code = litgen.generate_code(options, cpp_code)\n", "litgen_demo.show_cpp_code(generated_code.pydef_code)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Force usage of a lambda function\n", "\n", "In some rare cases, the usage of `py::overload_cast` might not be sufficient to discriminate the overload. In this case, you can tell litgen to disambiguate it via a lambda function. Look at the pybind C++ binding code below:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", " \n", " \n", "
\n", "
\n", " \n", "
\n", " 
m.def("foo3",\n",
       "    [](int x)\n",
       "    {\n",
       "        auto foo3_adapt_force_lambda = [](int x)\n",
       "        {\n",
       "            foo3(x);\n",
       "        };\n",
       "\n",
       "        foo3_adapt_force_lambda(x);\n",
       "    },     py::arg("x"));\n",
       "
\n", "\n", "
\n", " \n", " \n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cpp_code = \"\"\"\n", "void foo3(int x);\n", "\"\"\"\n", "options = litgen.LitgenOptions()\n", "options.fn_force_lambda__regex += r\"|^foo3$\"\n", "generated_code = litgen.generate_code(options, cpp_code)\n", "litgen_demo.show_cpp_code(generated_code.pydef_code)" ] } ], "metadata": { "kernelspec": { "display_name": "venv311", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.0" } }, "nbformat": 4, "nbformat_minor": 2 }