Table of contents

• Packaging Compiled Projects
  • pyproject.toml: build-system
  • pyproject.toml: project table
    • Extras
    • Command line
  • Tool section in pyproject.toml
  • Backend specific files
  • Example compiled file
  • Package structure
  • Versioning
  • Including/excluding files in the SDist
  • Distributing
  • Special considerations
    • NumPy

Packaging Compiled Projects

There are a variety of ways to package compiled projects. In the past, the only way to do it was to use setuptools/distutils, which required using lots of fragile internals - distutils was intended primarily to compile CPython, and setuptools tried to stay away from changing the compile portions except as required. Now, however, we have several very nice options for compiled packages!

The most exciting developments have been new native build backends:

• scikit-build-core: Builds C/C++/Fortran using CMake.
• meson-python: Builds C/C++/Fortran using Meson.
• maturin: Builds Rust using Cargo. Written entirely in Rust!
• enscons: Builds C/C++ using SCONs. (Aging now, but this was the first native backend!)

You should be familiar with packing a pure Python project - the metadata configuration is the same.

There are also classic setuptools plugins:

• scikit-build: Builds C/C++/Fortran using CMake.
• setuptools-rust: Builds Rust using Cargo.

If you have a really complex build, the newer native build backends might not support your use case yet, but if that’s the case, ask - development is driven by community needs. The older, more fragile setuptools based plugins are still a bit more flexible if you really need that flexibility for a feature not yet implemented in the native backends.

pyproject.toml: build-system

PY001 Packages must have a pyproject.toml file PP001 that selects the backend:

[build-system]
requires = ["scikit-build-core"]
build-backend = "scikit_build_core.build"

[build-system]
requires = ["meson-python"]
build-backend = "mesonpy"

[build-system]
requires = ["maturin"]
build-backend = "maturin"

pyproject.toml: project table

The metadata is specified in a standards-based format:

[project]
name = "package"
description = "A great package."
readme = "README.md"
license.file = "LICENSE"
authors = [
  { name = "My Name", email = "me@email.com" },
]
maintainers = [
  { name = "My Organization", email = "myemail@email.com" },
]
requires-python = ">=3.8"

dependencies = [
  "typing_extensions",
]

classifiers = [
  "Development Status :: 4 - Beta",
  "License :: OSI Approved :: BSD License",
  "Programming Language :: Python :: 3 :: Only",
  "Programming Language :: Python :: 3.8",
  "Programming Language :: Python :: 3.9",
  "Programming Language :: Python :: 3.10",
  "Programming Language :: Python :: 3.11",
  "Programming Language :: Python :: 3.12",
  "Topic :: Scientific/Engineering :: Physics",
]

[project.urls]
Homepage = "https://github.com/organization/package"
Documentation = "https://package.readthedocs.io/"
"Bug Tracker" = "https://github.com/organization/package/issues"
Discussions = "https://github.com/organization/package/discussions"
Changelog = "https://package.readthedocs.io/en/latest/changelog.html"

You can read more about each field, and all allowed fields, in packaging.python.org, Flit or Whey. Note that “Homepage” is special, and replaces the old url setting.

Extras

It is recommended to use extras instead of or in addition to making requirement files. These extras a) correctly interact with install requires and other built-in tools, b) are available directly when installing via PyPI, and c) are allowed in requirements.txt, install_requires, pyproject.toml, and most other places requirements are passed.

Here is an example of a simple extras:

[project.optional-dependencies]
test = [
  "pytest >=6.0",
]
mpl = [
  "matplotlib >=2.0",
]

Self dependencies can be used by using the name of the package, such as dev = ["package[test,examples]"], but this requires Pip 21.2 or newer. We recommend providing at least test and docs.

Command line

If you want to ship an “app” that a user can run from the command line, you need to add a script entry point. The form is:

[project.scripts]
cliapp = "package.__main__:main"

The format is command line app name as the key, and the value is the path to the function, followed by a colon, then the function to call. If you use __main__.py as the file, then python -m followed by the module will also work to call the app (__name__ will be "__main__" in that case).

Tool section in pyproject.toml

These tools all read the project table. They also have extra configuration options in tool.* settings.

Backend specific files

Example CMakeLists.txt file (using pybind11, so include pybind11 in build-system.requires too):

cmake_minimum_required(VERSION 3.15...3.26)
project(${SKBUILD_PROJECT_NAME} LANGUAGES CXX)

set(PYBIND11_FINDPYTHON ON)
find_package(pybind11 CONFIG REQUIRED)

pybind11_add_module(_core MODULE src/main.cpp)
install(TARGETS _core DESTINATION ${SKBUILD_PROJECT_NAME})

Example meson.build file (using pybind11, so include pybind11 in build-system.requires too):

project(
    'package',
    'cpp',
    version: '0.1.0',
    license: 'BSD',
    meson_version: '>= 1.1.0',
    default_options: [
        'cpp_std=c++11',
    ],
)

py = import('python').find_installation(pure: false)
pybind11_dep = dependency('pybind11')

py.extension_module('_core',
    'src/main.cpp',
    subdir: 'package',
    install: true,
    dependencies : [pybind11_dep],
)

install_subdir('src/package', install_dir: py.get_install_dir() / 'package', strip_directory: true)

Example Cargo.toml file:

[package]
name = "package"
version = "0.1.0"
edition = "2018"

[lib]
name = "_core"
# "cdylib" is necessary to produce a shared library for Python to import from.
crate-type = ["cdylib"]

[dependencies]
rand = "0.8.3"

[dependencies.pyo3]
version = "0.19.1"
# "extension-module" tells pyo3 we want to build an extension module (skips linking against libpython.so)
# "abi3-py38" tells pyo3 (and maturin) to build using the stable ABI with minimum Python version 3.8
features = ["extension-module", "abi3-py38"]

Example compiled file

Example src/main.cpp file:

#include <pybind11/pybind11.h>

int add(int i, int j) { return i + j; }

namespace py = pybind11;

PYBIND11_MODULE(_core, m) {
  m.doc() = R"pbdoc(
      Pybind11 example plugin
      -----------------------
      .. currentmodule:: python_example
      .. autosummary::
         :toctree: _generate
         add
         subtract
  )pbdoc";

  m.def("add", &add, R"pbdoc(
      Add two numbers
      Some other explanation about the add function.
  )pbdoc");

  m.def("subtract", [](int i, int j) { return i - j; }, R"pbdoc(
      Subtract two numbers
      Some other explanation about the subtract function.
  )pbdoc");
}

Example src/main.cpp file:

#include <pybind11/pybind11.h>

int add(int i, int j) { return i + j; }

namespace py = pybind11;

PYBIND11_MODULE(_core, m) {
  m.doc() = R"pbdoc(
      Pybind11 example plugin
      -----------------------
      .. currentmodule:: python_example
      .. autosummary::
         :toctree: _generate
         add
         subtract
  )pbdoc";

  m.def("add", &add, R"pbdoc(
      Add two numbers
      Some other explanation about the add function.
  )pbdoc");

  m.def("subtract", [](int i, int j) { return i - j; }, R"pbdoc(
      Subtract two numbers
      Some other explanation about the subtract function.
  )pbdoc");
}

use pyo3::prelude::*;

#[pyfunction]
fn add(x: i64, y: i64) -> i64 {
    x + y
}

#[pyfunction]
fn subtract(x: i64, y: i64) -> i64 {
    x - y
}

/// A Python module implemented in Rust. The name of this function must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
#[pymodule]
fn _core(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(add, m)?)?;
    m.add_function(wrap_pyfunction!(subtract, m)?)?;
    m.add("__version__", env!("CARGO_PKG_VERSION"))?;

    Ok(())
}

Package structure

The recommendation (followed above) is to have source code in /src, and the Python package files in /src/<package>.

Versioning

Check the documentation for the tools above to see what forms of dynamic versioning the tool supports.

Including/excluding files in the SDist

Each tool uses a different mechanism to include or remove files from the SDist, though the defaults are reasonable.

Distributing

Unlike pure Python, you’ll need to build redistributable wheels for each platform and supported Python version if you want to avoid compilation on the user’s system. See the CI page on wheels for a suggested workflow.

Special considerations

NumPy

Modern versions of NumPy (1.25+) allow you to target older versions when building, which is highly recommended, and this will become required in NumPy 2.0. Now you add:

#define NPY_TARGET_VERSION NPY_1_22_API_VERSION

(Where that number is whatever version you support as a minimum) then make sure you build with NumPy 1.25+ (or 2.0+ when it comes out). Before 1.25, it was necessary to actually pin the oldest NumPy you supported (the oldest-supported-numpy package is the easiest method). If you support Python < 3.9, you’ll have to use the old method for those versions.

If using pybind11, you don’t need NumPy at build-time in the first place.