5 Years Running Ceph

I have been a Ceph administrator for 5 years, and the Archlinux maintainer of Ceph for 3. With the release of Ceph 20 (tentacle) I made my first contributions to the upstream project, and I want to reflect on my journey so far.

Primer: what is Ceph?

#

Ceph is a storage solution that “provides a unified storage service with object, block, and file interfaces from a single cluster built from commodity hardware components.”

It provides distributed block (like your hard drive), file (like NFS), and object (like AWS S3) storage interfaces for Linux, OSX and Windows – plus more exotic datacenter-centric platforms.

It is also the underlying technology of Red Hat Storage, which is sold by many providers for 7 and 8 figure deals, and stores >2 exabytes¹ of data around the world.

Primer: why Ceph?

#

Bluntly put, because distributed storage is a super power.

Ceph has been the storage medium for all of my VMs, containers and PVCs for a little over 5 years now, operating on hard drives that are on average 6 years old, across multiple complete node refreshes and has never lost or corrupted a single byte of data in the entire period.

Every cloud provider uses network storage as their primary offering because it frees your state from the underlying physicalities.

• No more hard drive failures²
• No more IO limitations because of bad PCI buses³
• Reliable, consistent storage of 1s and 0s (no more evil firmware lying to your OS)⁴

But those are just the table stakes. Here’s a couple of the crazier things I’ve done with Ceph over the years:

1. Mount a single block device as single-writer, multi-reader across ~20 nodes and 3 datacenters
   • I’m not even sure a cloud provider has a product that is equivalent to this functionality
2. Export a S3 bucket as an NFS filesystem without paying bills that would bankrupt a medium size org
   • looking at you AWS FileGateway

v13, the early years

#

If I recall correctly, it was mid 2019 that I first bootstrapped what would become my production cluster.

It was Ceph 13 back then, and I used the (at the time) only supported deployment tool for Archlinux: ceph-deploy. This, as it turns out, was a mistake in hindsight. But we’ll get there, it worked for the initial setup.

After some testing and a nasty MTU bug which I only figured out after trawling the mailing lists and coming across a (now deleted?) post by grin, finally achieved a 3 healthy node cluster.

retrieved from my documentation:

This is possibly some network issue, but I cannot see the indicator about what to see. mon0 usually stands in quorum alone, and other mons cannot join. They get the monmap, they intend to join, but it just never happens, mons get from synchronising to probing, forever. Raising log level doesn’t reveal anything to me.

Spoiler alert: it was an MTU problem.

A crappy switch (cisco nexus, by courtesy of Satan) actually wasn’t able to handle its designated MTU size so the cluster members pushed larger-than-switch-reality-of-mtu packets, some of which obviously have failed to deliver. After lowering the MTU on the nodes to the level of the crappy switch everything came alive.

I just want to sidenote that I was absolutely unable to see that there were messages lost, and from where to where and what. I have seen no indicator on the initiator side that a message has been sent but never acknowledged(?) or acted upon; and I have no obvious sign that the destination have expected a message which was missing.

This maybe has been helped in the later versions, but if it wasn’t, maybe it ought to be.

I initially added a couple OSDs and just let it sit for a while. Then two months later, I started migrating my existing Incus (then LXD) workloads to use the Ceph storage driver.

The v14 nautilus upgrade

#

At some point Thore (previous maintainer for Ceph on Archlinux) pushed the v14 packages to packages/community, and I promptly fragged one of the mon deployments by upgrading prematurely. I tried to fix it, but ceph-deploy ran into many and varied Python related troubles (narrator: this is foreshadowing) that left me worse off than before.

…However, Ceph continued to function normally throughout this entire kerfuffle. Even down a monitor and partially upgraded!

I eventually ripped out all of the ceph-deploy machinery and replaced it with a couple pretty straight forward bash scripts that just directly configured the systemd services + config files.

This was one of the best decisions I made. What had previously been ~200 lines of Python became ~10 lines of bash that I could directly modify and understand.

This experience shaped the way I operate Ceph all the way to today.

The long sleep for v17

#

The v15 upgrade came and went peacefully for my cluster, and in early 2021 I started preparing for v16 which was slated to release in Q1. It released, and I waited for the Arch package to drop, which had normally happened within a couple weeks. It did not come. Months went by and eventually in early 2022 Ceph started breaking again because of incompatibilities between Python 3.10 and the version Ceph was compiled against (v3.7 IIRC?).

Then Archlinux pushed Python v3.10 which completely broke the Python parts of Ceph, making upgrades a bit awkward.

I eventually opened a bug report with Archlinux, and in the background started to build Ceph myself so I could keep doing system upgrades.

The months dragged on, and Ceph v17 was released but still Archlinux remained on the now EOL v15.

Eventually, myself and a few others that use Ceph on Archlinux managed to contact the then maintainer of Ceph, and have him hand over the package to the AUR, where I adopted it.⁵

Maintaining a >1 million LOC, mission critical C++ project

#

After about 3 eventful months I had managed to build, test and upgrade to Ceph versions 16 and 17, getting me back to a supported version and giving me strong opinions about C++, Python, CMake, Make and build systems generally – but that is a story for another time.

While this was not my first contribution to the open source community, it was definitely the largest and taught me a lot about how to deal with consumers of the software I write maintain, which has stuck with me to this day.

v18, or: is the Python subinterpreter in the room with us now?

#

During the update to v18, my mgr services starting dying with a fateful error:

ImportError: PyO3 modules may only be initialized once per interpreter process

Now, this is an interesting error for a couple reasons:

1. The mgr service is written in C++, so how are we getting Python ImportErrors?
2. What is an “interpreter process”?
3. What is a PyO3 module?

Without getting to much into the weeds:

• CPython allows C API users to create >1 Python interpreters per single OS process
• What happens if you “share” Python objects between interpreters is “unspecified” and/or “undefined”
• Virtually no-one embeds >1 interpreters in a single process so this undefined-ness largely went unnoticed, minus a segfault here or there
• PyO3 is a library for building rust extension modules for Python and because of the Rust community’s emphasis on writing safe software added a check to ensure that PyO3 modules were never “initialized” more than once
• Hence the error message above

This is not a small problem. It largely affects the entire C API extension ecosystem around Python, the Python 3.12 changes that cause it are now starting to make their way into “stable” distros like Rocky Linux and OpenSUSE which I’m very sure is going to be painful for everyone in involved – speaking from personal experience.

This issue is the reason I’m working with PyO3 to implement the changes needed to support subinterpreters (and free threaded Python) which is going to become a pretty big issue for the ecosystem as more and more people start using Python >=3.12.

Especially given how much of the Python ecosystem now uses Rust as the extension language:

v20 and my first upstream contributions

#

During my v19 upgrade, I came across a truly awful C++ / Boost metaprogramming issue during the build, and unlikely many of the previous issues, this one wasn’t google-able.

Even now if you search for BOOST_PP_ITERATION_02 my GitHub issue comes up first.

So, naturally my first contribution to Ceph includes the ranty-sweary⁶ comments make in the heat of the moment 😅.

I wouldn’t have it any other way.

v21 and beyond

#

I expect to contribute more to Ceph in 2026, likely indirectly at first via enabling to PyO3 to support subinterpreters and then helping the important Python module consumers to upgrade to the relevant PyO3 version, and also perhaps directly via upstreaming some of the patches I maintain for Ceph on Archlinux.

I may also start packaging Ceph for Debian, as I need it in Trixie and Ceph v19 is notably absent.

All in all, I’ve really enjoyed using and participating in the Ceph ecosystem. Here’s to another 5 years! 🥂