PyO3 Multiphase Initialization

Starting from PyO3 v0.28, all Rust #[pymodule]s will automatically start using the modern multi-phase initialization machinery added in cpython 3.4, making Rust Python extensions similar to native py modules, with (overridable!) __new__ and __init__ stages.

See the pull-request here: github.com/PyO3/pyo3#5525.

What does this mean to…

#

Authors

#

Mostly this is a transparent change. Under the hood, your module is no longer a singleton, global python object, meaning that the Python interpreter now can manage your module(s) lifecycle – for example, being unloaded from sys.modules will now actually deallocate your module, and if/when it is reimported, it will be reinitialized, the same as a normal .py module.

This makes your module a bit less surprising, and makes it easier to test the lifecycle functions the way python normally does.

Consumers

#

Nothing really changes for module consumers – CPython, Pypy and GraalPy should all just work as usual.

Future work

#

This is hopefully just the beginning, in the coming months I’m looking to expose per-module state to module authors, which should make state management on the Rust side much easier to deal with (less PyOnceLocks, statics and thread_local!s!), plus various improvements to the pyo3 machinery itself, so we can get to natively supported subinterpreters for all Rust pymodules.

If you’re interested in updates around this, subscribe to github.com/PyO3/pyo3#3451.

Personal thoughts

#

Special thanks to @Aequitosh, as him and @davidhewitt are responsible for the majority of this work, but lacked the time to push it over the finish line.

This is a pretty interesting area of work, as it requires me to roughly understand:

1. Rust
2. C++
3. CPython APIs
4. Python (the language)

In order to implement features. This is a pretty wide set of technologies, that must be understood well enough to be confident about working between them and across FFI boundaries, which is always complicated to do right.

I personally need this, because my self hosted Ceph cluster has suffered from the lack of subinterpreter support for nearly 3 years now¹, something even Debian, Ubuntu, and even Proxmox are now suffering from.

So hopefully, my continued work here can finally resolve the root cause of the trouble. Time will tell.