Ctrl+K

litgen template

litgen template#

litgen_template is a template repository to build python bindings using litgen, pybind11 or nanobind and scikit-build.

This template is based on scikit_build_example.

Usage for final users#

Below are the instructions you would give to final users of your bindings. They are extremely short:

First, install the package from source

git clone https://github.com/pthom/litgen_template.git && cd litgen_template
pip install -v .

Then, use it from python

import daft_lib
daft_lib.add(1, 2)

(this template builds bindings for a C++ library called DaftLib, and publishes it as a python module called daft_lib)

Of course, you could also publish your bindings to PyPI, and tell your users to install them with pip install daft-lib. This template provides tooling to make the publishing process easier, via cibuildwheel

Autogenerate the binding code#

Install requirements#

Create a virtual environment

python3 -m venv venv
source venv/bin/activate

Install the requirements

pip install -r requirements-dev.txt

This will install litgen (the bindings generator), pybind11 and nanobind (libraries to create C++ to Python bindings), pytest (for the tests), black (a code formatter), and mypy (static type checker for python).

See requirements-dev.txt.

Generate bindings#

Optionally, change the C++ code

Optionally, switch to nanobind

By default, this template uses pybind11. If you want to switch to nanobind, you can do so with

export LITGEN_USE_NANOBIND=ON

Run the code generation via litgen

python tools/autogenerate_bindings.py

This will:

Tip: compare the python stubs with the C++ header file to see how close they are!

Customize bindings generation#

tools/autogenerate_bindings.py is an example of how to drive the bindings generation via a script.

It will set a number of options, demonstrating a subset of litgen options: see the litgen documentation for more details.

The options in this script are heavily documented, and correspond to the documentation you can find in DaftLib.h_

Adapt for your own library#

Names, names, names

In this template repository:

  • the C++ library is called DaftLib

  • the native python module generated by pybind11 is called _daft_lib

  • the python module which is imported by users is called daft_lib (it imports and optionally adapts _daft_lib)

  • the pip package that can optionally be published to PyPI is called daft-lib (as Pypi does not allow dashes in package names)

You can change these names by running change_lib_name.py in the tools/change_lib_name folder.

Structure of this template#

Bound C++ library#

The C++ library DaftLib is stored inside src/cpp_libraries/DaftLib/

src/
├── cpp_libraries/
     └── DaftLib/
         ├── CMakeLists.txt
         ├── DaftLib.h
         └── cpp/
             └── DaftLib.cpp

C++ binding code#

The C++ binding code is stored inside _pydef_pybind11/ (or _pydef_nanobind/ if you use nanobind).

_pydef_pybind11/
      ├─── module.cpp              # Main entry point of the python module
      └── pybind_DaftLib.cpp       # File with bindings *generated by litgen*

Python stubs#

The python stubs are stored inside _stubs/

_stubs/
└── daft_lib/
          ├── __init__.pyi         # Stubs *generated by litgen*
          ├── __init__.py          # The python module (daft_lib) main entry point
          │                        # (it imports and optionally adapts _daft_lib)
          └── py.typed             # An empty file that indicates that the python module is typed

Tooling for the bindings generation#

tools/
├── autogenerate_bindings.py
└── change_lib_name/
    └── change_lib_name.py

tools/autogenerate_bindings.py is the script that will generate the bindings using litgen.

tools/change_lib_name/change_lib_name.py is an optional utility that you can use once after cloning this template, in order to rename the libraries (e.g. from DaftLib to MyLib, daft_lib to my_lib, etc.)

Compilation#

├── CMakeLists.txt                 # CMakeLists (used also by pip, via skbuild)
├── litgen_cmake/                  # litgen_setup_module() is a cmake function that simplifies
│   └── litgen_setup_module.cmake  # the deployment of a python module with litgen
│
├── requirements-dev.txt           # Requirements for development (litgen, pybind11, pytest, black, mypy)

Deployment#

pyproject.toml is used by pip and skbuild to build and deploy the package. It defines the name of the package, the version, the dependencies, etc.

Continuous integration#

Several github workflows are defined in .github/workflows:

.github/
├── dependabot.yml   # Configuration for dependabot (automatically update CI dependencies)
├── workflows/
    ├── conda.yml    # Build the package with conda
    ├── pip.yml      # Build the package with pip
    └── wheels.yml   # Build the wheels with cibuildwheel, and publish them to PyPI
                     # (when a new release is created on github)

Note:

Tests#

├── tests/daft_lib_test.py    # This is a list of python tests that will check
└── pytest.ini

Those tests are run by cibuildwheel and by the pip CI workflow.

Editable development mode#

If you want to quickly iterate on the C++ code, and see the changes reflected in python without having to reinstall the package, you should use the python editable development mode.

Setup editable mode#

Step1: Install the package in editable mode

pip install -v -e .  # -e stands for --editable, and -v stands for --verbose

Step 2: Create a standard C++ build directory

mkdir build && cd build
cmake ..
make # rebuild when you change the C++ code, and the changes will be reflected in python!

Debug C++ bindings in editable mode#

The pybind_native_debug executable provided in this template is a simple C++ program that can be used to debug the bindings in editable mode.

src/pybind_native_debug/
        ├── CMakeLists.txt
        ├── pybind_native_debug.cpp
        └── pybind_native_debug.py

Simply edit the python file pybind_native_debug.py by adding calls to the C++ functions you want to debug. Then, place breakpoints in the C++ code, and debug the C++ program.

Development tooling#

This template is ready to be used with additional tools:

  • pre-commit

  • ruff

  • mypy

  • pyright

  • black

  • pytest

  • cibuildwheel

pre-commit#

pre-commit is a tool that allows you to run checks on your code before committing it. This template provides a default pre-commit configuration for it, but it is not active by default;

You can install pre-commit with:

pip install pre-commit

Then, you can activate the pre-commit hooks for your repository with:

pre-commit install

The pre-commit configuration file .pre-commit-config.yaml, is configured with the following hooks:

  • basic sanity checks: trailing-whitespace, end-of-file-fixer,check-yaml, check-added-large-files

  • black: uncompromising Python code formatter

  • ruff: fast Python linter and code formatter (only used for linting)

  • mypy: static type checker for python

You can find more interesting hooks on the pre-commit hooks repository, and for example add ruff, mypy, black, etc.

You may want to disable some checks in the .pre-commit-config.yaml file if you think this is too strict for your project.

ruff: python linter and code formatter#

ruff is a very fast python linter and code formatter. You can install it and run it with:

pip install ruff # install ruff (once)
ruff . # each time you want to check your python code

mypy and pyright: static type checkers for python#

mypy and pyright are static type checkers for python.

You can use either one of them, or both.

mypy#

pip install mypy  # install mypy (once)
mypy # each time you want to check your python code

mypy is configured via the mypy.ini file.

pyright#

pip install pyright # install pyright (once)
pyright # each time you want to check your python code

pyright is configured via the pyrightconfig.json file.

black: python code formatter#

black is a python code formatter.

pip install black # install black (once)
black . # each time you want to format your python code

pytest: python tests#

pytest is an easy-to-use python test framework.

pip install pytest # install pytest (once)
pytest # each time you want to run your python tests

It is configured via the pytest.ini file, and tests are stored in the tests folder.

cibuildwheel: build wheels for all platforms#

ci-buildwheel is a tool that allows you to build wheels for all platforms.

It is configured via the pyproject.toml file (see the [tool.cibuildwheel] section), and the github workflow file.

run_all_checks#

tools/run_all_checks.sh is a script you can run before committing or pushing. It will run a collection of checks (mypy, black, ruff, pytest).

Contents