Why use a mainline kernel
For the past year or so, I’ve increasingly been using mainline Linux kernels on my various servers and eventually laptop and desktop machines too.
That was transitioning from Ubuntu’s generic kernel which I feel has sadly decreased in quality over time. The Ubuntu kernel includes a lot of backported fixes and occasionally, those backports go bad, resulting in missing commits, introducing bugs and regressions. Unfortunately the way the Ubuntu kernel is built, tested and published comes with a lot of delays, making fixing such regressions often take weeks if not months (depending on whether security updates show up in between).
So I started taking the latest stable bugfix release of the mainline kernel, generate a configuration that’s very close to an Ubuntu generic kernel, cherry-pick a few small changes that aren’t upstream yet and then build that and push it to my machines.
That’s been working surprisingly well so far! Those kernels haven’t been perfect, I did catch a couple of regressions, but as I’m now working with a mainline kernel, performing a bisect, identifying the offending commit and getting it resolved upstream is very easy, with a revert taking an hour or so at most and a fix taking just a few days to hit mainline.
Making them available to everyone
Up until now, I’ve been manually building those kernels from an internal git repository, building them directly on a couple of servers (amd64 and arm64) and then transferring the resulting .debs directly to my other machines.
That works, but it’s not a particularly clean build environment and installing kernels that way doesn’t really scale!
That’s why I’ve now spent a few days moving it all to Github and a proper package repository.
The kernel tree is now available here: https://github.com/zabbly/linux
For building, I’m using some self-hosted Github runners on my local Incus cluster so I can have access to beefy Debian and Ubuntu builders on both amd64 and arm64.
The result is a repository that contains both amd64 and arm64 builds for Ubuntu 20.04 LTS, Ubuntu 22.04 LTS and Debian 12. This is all automatically built and automatically imported into the repository with the only manual step being to update the “linux-zabbly” meta-package after testing the new kernel on some test systems.
Installation instructions can be found here: https://github.com/zabbly/linux#installation
Just keep in mind that you’ll most likely have to disable UEFI SecureBoot as those kernel builds aren’t signed unlike those that come directly from your distribution.
The kernel will be updated once a week unless something major happens requiring an intermediate update. It will roll from one kernel version to the next after it has received its first bugfix release which has so far been a good way to avoid some of those initial regressions!
I use ZFS quite extensively to store local containers and VMs on Incus.
The Ubuntu kernel ships with a built-in version of ZFS but to keep the Zabbly kernel clean, I opted not to do that.
Instead, I maintain a separate ZFS repository at: https://github.com/zabbly/zfs
This currently contains ZFS 2.2rc3 and will be updated with new release candidates and eventually the 2.2 stable release. The decision to ship 2.2 rather than stick to 2.1 is motivated by ZFS 2.2 properly handling VFS idmap shift, a critical feature for Incus.
That repository includes both openzfs-zfs-dkms, the package providing the kernel driver as well as the usual set of tools used to manage zfs, openzfs-zfsutils.