Why use a mainline kernel
For the past year or so, I’ve increasingly been using mainline Linux kernels on my various servers and eventually laptop and desktop machines too.
That was transitioning from Ubuntu’s generic kernel which I feel has sadly decreased in quality over time. The Ubuntu kernel includes a lot of backported fixes and occasionally, those backports go bad, resulting in missing commits, introducing bugs and regressions. Unfortunately the way the Ubuntu kernel is built, tested and published comes with a lot of delays, making fixing such regressions often take weeks if not months (depending on whether security updates show up in between).
So I started taking the latest stable bugfix release of the mainline kernel, generate a configuration that’s very close to an Ubuntu generic kernel, cherry-pick a few small changes that aren’t upstream yet and then build that and push it to my machines.
That’s been working surprisingly well so far! Those kernels haven’t been perfect, I did catch a couple of regressions, but as I’m now working with a mainline kernel, performing a bisect, identifying the offending commit and getting it resolved upstream is very easy, with a revert taking an hour or so at most and a fix taking just a few days to hit mainline.
Making them available to everyone
Up until now, I’ve been manually building those kernels from an internal git repository, building them directly on a couple of servers (amd64 and arm64) and then transferring the resulting .debs directly to my other machines.
That works, but it’s not a particularly clean build environment and installing kernels that way doesn’t really scale!
That’s why I’ve now spent a few days moving it all to Github and a proper package repository.
The kernel tree is now available here: https://github.com/zabbly/linux
For building, I’m using some self-hosted Github runners on my local Incus cluster so I can have access to beefy Debian and Ubuntu builders on both amd64 and arm64.
The result is a repository that contains both amd64 and arm64 builds for Ubuntu 20.04 LTS, Ubuntu 22.04 LTS and Debian 12. This is all automatically built and automatically imported into the repository with the only manual step being to update the “linux-zabbly” meta-package after testing the new kernel on some test systems.
Using them
Installation instructions can be found here: https://github.com/zabbly/linux#installation
Just keep in mind that you’ll most likely have to disable UEFI SecureBoot as those kernel builds aren’t signed unlike those that come directly from your distribution.
The kernel will be updated once a week unless something major happens requiring an intermediate update. It will roll from one kernel version to the next after it has received its first bugfix release which has so far been a good way to avoid some of those initial regressions!
ZFS
I use ZFS quite extensively to store local containers and VMs on Incus.
The Ubuntu kernel ships with a built-in version of ZFS but to keep the Zabbly kernel clean, I opted not to do that.
Instead, I maintain a separate ZFS repository at: https://github.com/zabbly/zfs
This currently contains ZFS 2.2rc3 and will be updated with new release candidates and eventually the 2.2 stable release. The decision to ship 2.2 rather than stick to 2.1 is motivated by ZFS 2.2 properly handling VFS idmap shift, a critical feature for Incus.
That repository includes both openzfs-zfs-dkms, the package providing the kernel driver as well as the usual set of tools used to manage zfs, openzfs-zfsutils.
Github
Twitter
LinkedIn
Mastodon
RSS feed
Do you have the same issues with the debian 12 kernels?
I do not, though to be fair, I’ve only recently started using Debian again here and there after an almost 20 years long break from the project 🙂
The main benefit from having the Debian builds is that I now have an absolutely identical kernel on all systems regardless of distribution, making transitioning a fair bit easier.
What instabilities have you experienced?
Off the top of my head for the past year or so, the main ones I’ve hit are:
– Regression in KVM support causing a variety of systems to trigger a kernel log loop when starting LXD VMs, very rapidly filling the entire disk with log causing further breakage. (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1966499)
– Regression in ZFS+shiftfs handling breaking LXD (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1990849)
– Regression in io_uring handling allowing for unprivileged users to deadlock or panic the kernel (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2020901)
Those are the ones I remember the most as I was involved in either reporting them or having someone directly fix them.
I was lucky to have a kernel engineer working directly on the LXD team that I could temporarily redirect to fixing those issues.
The Ubuntu kernel is a massive beast of patches, tracking down what was missed or badly merged is much harder than just doing a normal bisect and fix on a regular mainline kernel.
Even with that though, the time to resolution hasn’t been great, a “quick” fix usually would still take 6-8 weeks, with some of the more tricky ones like that ZFS one taking several months, all while leaving users broken or running old unsafe kernels…
Thanx for your great work, it’s working on my bookworm 🥳
Is it possible to install kernel 6.5 in Debian 10 ?
I currently only support the past two Ubuntu LTS and past two Debian releases, going further back starts becoming a bit more trouble than it’s worth when considering dealing with older compilers, having to run more Github Actions runners and dealing with all the space usage from those builds.
That said, you could try using the Debian 11 repo on Debian 10, this may work as the kernel is usually pretty self-contained. The main thing that could break is DKMS which may or may not be a problem for you.
Ok I understand.
I installed kernel 6.5 on Debian 11. It works great !!!
Thank You!!!
Great work 👍
Was wondering, would it also be possible to add Kernel version 6.1 to your Repository?
The reason I am asking is that 6.1.x has been marked by Linus / Kernel.org as an LTK (Long-Term [Support] Kernel – not to be confused with Ubuntu’s LTS for their Distro), which is feature-frozen (except perhaps for the Device Driver and Firmware Buildchains) and will be maintained for a number of years going forward.
I was downloading the updated 6.1.x Auto-Build Deb Packages from the Ubuntu Mainline Kernel PPA, but that has either fallen over, or has been intentionally discontinued by Ubuntu.
Thank You and Best Regards!
.. .. K. X. Adams
Hey there,
I’m not planning to do it right now as I don’t really have a use for the LTS kernels myself.
Now, should someone want to sponsor such a repository, I could certainly set one up though.
It wouldn’t be part of the “stable” kernel repository but likely be an “lts” repository alongside it, tracking the latest LTS mainline kernel.
Stéphane
What about dependency on newer libc versions? These mainline kernel builds do not include it, meaning that if you upgrade the kernel keeping the old libc, then the system will not boot, correct?
That’s incorrect.
Unlike other operating systems like some of the BSDs, Linux has a stable syscall API, meaning that newer kernels will remain compatible with older user space including the C library.
You can perfectly run an old version of CentOS or Slackware from decades ago using the latest kernel and without having to make any alteration to userspace.
Earlier this week Ubuntu issued an updated version of the Nvidia drivers (at least for 535) which is now version nvidia-535.161.07. Ever since then my Zabbly+ kernels cannot connect to graphics. I have removed and reinstalled the kernels several times but X-windows always fails. I also tried several other possible drivers but the behavior was unchanged. I have loaded the Ubuntu 22.04 version of 6.5.0-21-generic and this works ok with the new drivers. Is there a way to make the Zabbly drivers work with the new Nvidia drivers, or do I need to await new software.
Thanks for any help.
It’s most likely some kind of regression on NVIDIA’s end with the more recent Linux 6.7 kernels.
I did push the update to the latest 6.7.10 a few days ago, I don’t know if that’s going to help in your case.
Later this week, my kernels will move over to 6.8.x but again, I have no idea what impact that may have on 3rd party drivers like NVIDIA’s.