For the past year or so, I’ve increasingly been using mainline Linux kernels on my various servers and eventually laptop and desktop machines too.
That was transitioning from Ubuntu’s generic kernel which I feel has sadly decreased in quality over time. The Ubuntu kernel includes a lot of backported fixes and occasionally, those backports go bad, resulting in missing commits, introducing bugs and regressions. Unfortunately the way the Ubuntu kernel is built, tested and published comes with a lot of delays, making fixing such regressions often take weeks if not months (depending on whether security updates show up in between).
So I started taking the latest stable bugfix release of the mainline kernel, generate a configuration that’s very close to an Ubuntu generic kernel, cherry-pick a few small changes that aren’t upstream yet and then build that and push it to my machines.
That’s been working surprisingly well so far! Those kernels haven’t been perfect, I did catch a couple of regressions, but as I’m now working with a mainline kernel, performing a bisect, identifying the offending commit and getting it resolved upstream is very easy, with a revert taking an hour or so at most and a fix taking just a few days to hit mainline.
Making them available to everyone
Up until now, I’ve been manually building those kernels from an internal git repository, building them directly on a couple of servers (amd64 and arm64) and then transferring the resulting .debs directly to my other machines.
That works, but it’s not a particularly clean build environment and installing kernels that way doesn’t really scale!
That’s why I’ve now spent a few days moving it all to Github and a proper package repository.
For building, I’m using some self-hosted Github runners on my local Incus cluster so I can have access to beefy Debian and Ubuntu builders on both amd64 and arm64.
The result is a repository that contains both amd64 and arm64 builds for Ubuntu 20.04 LTS, Ubuntu 22.04 LTS and Debian 12. This is all automatically built and automatically imported into the repository with the only manual step being to update the “linux-zabbly” meta-package after testing the new kernel on some test systems.
Using them
Installation instructions can be found here: https://github.com/zabbly/linux#installation Just keep in mind that you’ll most likely have to disable UEFI SecureBoot as those kernel builds aren’t signed unlike those that come directly from your distribution.
The kernel will be updated once a week unless something major happens requiring an intermediate update. It will roll from one kernel version to the next after it has received its first bugfix release which has so far been a good way to avoid some of those initial regressions!
ZFS
I use ZFS quite extensively to store local containers and VMs on Incus. The Ubuntu kernel ships with a built-in version of ZFS but to keep the Zabbly kernel clean, I opted not to do that.
This currently contains ZFS 2.2rc3 and will be updated with new release candidates and eventually the 2.2 stable release. The decision to ship 2.2 rather than stick to 2.1 is motivated by ZFS 2.2 properly handling VFS idmap shift, a critical feature for Incus.
That repository includes both openzfs-zfs-dkms, the package providing the kernel driver as well as the usual set of tools used to manage zfs, openzfs-zfsutils.
It’s now been a whole month since I left Canonical and started working as an independent!
This has been quite the month, both professionally and personally! In no particular order, this included, setting up a new business, dealing with a somewhat last minute datacenter move (thankfully just one floor down), doing some initial sponsored work, helping out with a LXD fork, selling a house and caring for a sick cat (now all back to normal).
Given everything that’s been happening, I thought I’d use the opportunity to write down some details on the most relevant things I’ve been doing and what to expect moving forward.
Zabbly
Zabbly is the name of the business I’ve registered here in Canada.
I didn’t really like the idea of doing all business moving forward just under my own name as I may want to sub-contract some aspects of it or even have employees down the line. Having the business part of my life have its own name will make that a fair bit cleaner.
For now, the main things that have been moved over to Zabbly are my organization and IP allocations with ARIN, membership on the Montreal Internet Exchange (QIX) and a number of associated contracts related to AS399760 (my BGP ASN). As part of that, Zabbly is also now listed as the sponsor for all the Linux Containers infrastructure.
Allowing to more clearly separate personal and work-related expenses is going to be another benefit of this move even if legally and from a tax point of view, it’s still all me.
ZFS delegation
An initial bit of sponsored work I got to do this month has been adding support for ZFS delegation to LXD. This makes use of a ZFS 2.2 feature which allows for a dataset to be delegated to a particular user namespace. The ZFS tools can then be used from within that container to create nested datasets or manage snapshots.
This is very exciting as it was the one feature that btrfs had which ZFS offered no equivalent for. It should allow for things like running Docker with the ZFS backend inside of LXD containers, having VPS users be able to create their own datasets, handled their own snapshots and be able to send and receive datasets.
This was quite an exciting development and the LXC team spent quite a bit of time over the past couple weeks chatting with Aleksa and seeing where things were headed.
On my end, I initially helped out trying to make the thing actually pass the testsuite, quite a bit harder than it may sound when dealing with a pretty big codebase and everything having been renamed! I also contributed some ideas of what such a fork may want to change compared to stock LXD.
It’s not often that you get a second chance at designing something like LXD/Incus. While having a working upgrade path and good backward compatibility is obviously still very important, the fact that anyone migrating will need to deal with some amount of manual work also makes it possible to do away with past mistakes and remove some bits that are seldom used.
I expect I’ll be spending a bunch of my time over the next couple of months helping get Incus into a releasable state. Continuing with the current cleanups, getting the documentation back into shape, putting CI and publishing infrastructure back online (basically re-using what I was once providing to LXD).
The biggest task yet to come is to write tooling and processes to monitor changes happening in Canonical’s LXD and then cherry-pick those into Incus. Again, the hard fork, name and path changes and variety of other changes is going to make that a bit of a challenge but once done, it should make it quite easy to do weekly syncs and reviews of changes.
What’s next
As mentioned, I expect to spend a fair bit of my time over the next few weeks/months helping out with Incus, getting it into shape for an initial release.
I’m all set up for contract work and sponsorship now, so if there’s anything you think I can do for you, feel free to reach out at info@zabbly.com.
I’ve also been added to the Github Sponsors program, so if you’d just like to help out with my work on those various projects, that’s available too: https://github.com/sponsors/stgraber
After a bit over 12 years working for Canonical, Friday 7th of July was my last day.
It’s a bit of a bittersweet moment leaving a company after you’ve invested so much of your time into it, but I believe that now was the right time for me. As I’ve told colleagues and upper management, Canonical isn’t the company I excitedly joined back in 2011 and it’s not a company that I would want to join today, therefore it shouldn’t be a company that I keep working for either.
I’ll most miss working with the LXD team. Canonical is truly lucky to have such a great team of engineers going above and beyond to support a project like LXD. It’s quite unique to have a small team with such a wide variety of skills ranging from kernel development, to distributed systems, to web frontends and documentation, all working together to make a project like LXD possible.
LXD
Following the announcement of my resignation, Canonical decided to pull LXD out of the Linux Containers projects and relocate it to a full in-house project. That’s the news which we announced last week.
I obviously wish that this particular change hadn’t happened, I strongly see value in having a project like LXD be run in a more open, community environment where everyone’s opinion is valued and everyone’s contribution, no matter the size, is welcome. Having the “LXD community experiment” be labeled a failure within Canonical seems unfair to me and to everyone who contributed over the years.
As for my particular involvement in Canonical’s LXD moving forward, I will definitely remain an active user of LXD and will likely still be filing issues and the occasional fix. However, I don’t intend to ever sign Canonical’s CLA, so should that become a barrier to contribution for the project, I will have to stop contributing to it.
Ubuntu
On the Ubuntu front, I’m currently a mostly inactive member of the Ubuntu Release team, Ubuntu Archive team and Ubuntu SRU team. I will be stepping down from all of those as I struggled to find any time to help them out while working for Canonical full time and don’t expect things to improve now.
I will remain an Ubuntu Core Developer and may contribute the occasional bugfix, package updates or new packages here and there. I don’t have any plans to move away from Ubuntu for my own systems.
Future
As for what I’ll be doing next. One thing I can share immediately is that I’m not joining another company nor do I have any intention to join another company at this stage.
I’m going to start by working on a number of pet projects that I’ve either neglected or been unable to even start so far. Some of those could lead to a source of revenue, some others will just be for the community’s benefit.
I’m also getting setup for freelance work, so will be able to accept the occasional consultancy or training contract where those make sense for me.
Conclusion
It’s a bit of an end of an era for me, a lot has changed over those 12 years both personally and in the industry, so I’m looking forward to have some time to reset and figure out what’s next!
Over the past few posts, I covered the hardware I picked up to setup a small LXD cluster and get it all setup at a co-location site near home. I’ve then gone silent for about 6 months, not because anything went wrong but just because of not quite finding the time to come back and complete this story!
So let’s pick things up where I left them with the last post and cover the last few bits of the network setup and then go over what happened over the past 6 months.
Routing in a HA environment
You may recall that the 3 servers are both connected to a top of the rack switch (bonded dual-gigabit) as well as connected to each other (bonded dual-10-gigabit). The netplan config in the previous post would allow each of the servers to talk to the others directly and establish a few VLANs on the link to the top of the rack switch.
Those are for:
WAN-HIVE: Peering VLAN with my provider containing their core routers and mine
INFRA-UPLINK: OVN uplink network (where all the OVN virtual routers get their external addresses)
INFRA-HOSTS: VLAN used for external communication with the servers
INFRA-BMC: VLAN used for the management ports of the servers (BMCs) and switch, isolated from the internet
Simply put, the servers have their main global address and default gateway on INFRA-HOSTS, the BMCs and switch have their management addresses in INFRA-BMC, INFRA-UPLINK is consumed by OVN and WAN-HIVE is how I access the internet.
In my setup, I then run three containers, one on each server which each gets direct access to all those VLANs and act as a router using FRR. FRR is configured to establish BGP sessions with both of my provider’s core routers, getting routing to the internet that way and announcing my IPv4 and IPv6 subnets that way too.
On the internal side of things, I’m using VRRP to provide a virtual router internally. Typically this means that frr01 is the default gateway for all egress traffic while ingress traffic is somewhat spread across all 3 thanks to them having the same BGP weight (so my provider’s routers distribute the connections across all active peers).
With that in place, so long as one of the FRR instances are running, connectivity is maintained. This makes doing maintenance quite easy as there is effectively no SPOF.
Enter LXD networks with OVN
Now for where things get a bit trickier. As I’m using OVN to provide virtual networks inside of LXD, each of those networks will typically need some amount of public addressing. For IPv6, I don’t do NAT so each of my networks get a public /64 subnet. For IPv4, I have a limited number of those, so I just assign them one by one (/32) directly to specific instances.
Whenever such a network is created, it will grab an IPv4 and IPv6 address from the subnet configured on INFRA-UPLINK. That part is all good and the OVN gateway becomes immediately reachable.
The issue is with the public IPv6 subnet used by each network and with any additional addresses (IPv4 or IPv6) which are routed directly to its instances. For that to work, I need my routers to send the traffic headed for those subnets to the correct OVN gateway.
But how do you do that? Well, there are pretty much three options here:
You use LXD’s default mode of performing NDP proxying. Effectively, LXD will configure OVN to directly respond to ARP/NDP on the INFRA-UPLINK VLAN as if the gateway itself was holding the address being reached. This is a nice trick which works well at pretty small scale. But it relies on LXD configuring a static entry for every single address in the subnet. So that’s fine for a few addresses but not so much when you’re talking a /64 IPv6 subnet.
You add static routing rules to your routers. Basically you run lxc network show some-name and look for the IPv4 and IPv6 addresses that the network got assigned, then you go on your routers and you configure static routes for all the addresses that need to be sent to that OVN gateway. It works, but it’s pretty manual and effectively prevents you from delegating network creation to anyone who’s not the network admin too.
You use dynamic routing to have all public subnets and addresses configured on LXD to be advertised to the routers with the correct next-hop address. With this, there is no need to configure anything manually, keeping the OVN config very simple and allowing any user of the cluster to create their own networks and get connectivity.
Naturally I went with the last one. At the time, there was no way to do that through LXD, so I made my own by writing lxd-bgp. This is a pretty simple piece of software which uses the LXD API to inspect its networks, determine all OVN networks tied to a particular uplink network (INFRA-UPLINK in my case) and then inspect all instances running on that network.
It then sends announcements both for the subnets backing each OVN networks as well as for specific routes/addresses that are routed on top of that to specific instances running on the local system.
The result is that when an instance with a static IPv4 and IPv6 starts, the lxd-bgp instance running on that particular system will send an announcement for those addresses and traffic will start flowing.
Now deploy the same service on 3 servers, put them into 3 different LXD networks and set the exact same static IPv4 and IPv6 addresses on them and you now have a working anycast service. When one of the containers or its host go down for some reason, that route announcement goes away and the traffic now heads to the remaining instances. That does a good job at some simplistic load-balancing and provides pretty solid service availability!
The past 6 months
Now that we’ve covered the network setup I’m running, let’s spend a bit of time going over what happened over the past 6 months!
In short, well, not a whole lot. Things have pretty much just been working. The servers were installed in the datacenter on the 21st of December. I’ve then been busy migrating services from my old server at OVH over to the new cluster, finalizing that migration at the end of April.
I’ve gotten into the habit of doing a full reboot of the entire cluster every week and developed a bit of tooling for this called lxd-evacuate. This makes it easy to relocate any instance which isn’t already highly available, emptying a specific machine and then letting me reboot it. By and large this has been working great and it’s always nice to have confidence that should something happen, you know all the machines will boot up properly!
These days, I’m running 63 instances across 9 projects and a dozen networks. I spent a bit of time building up a Grafana dashboard which tracks and alerts on my network consumption (WAN port, uplink to servers and mesh), monitors the health of my servers (fan speeds, temperature, …), tracks CEPH consumption and performance, monitors the CPU, RAM and load of each of the servers and also track performance on my top services (NSD, unbound and HAProxy).
LXD also rolled out support for network ACLs somewhat recently, allowing for proper stateful firewalling directly through LXD and implemented in OVN. It took some time to setup all those ACLs for all instances and networks but that’s now all done and makes me feel a whole lot better about service security!
What’s next
On the LXD front, I’m excited about a few things we’re doing over the next few months which will make environments like mine just that much nicer:
Native BGP support (no more lxd-bgp)
Native cluster server evacuation (no more lxd-evacuate)
Built-in DNS server for instance forward/reverse records as well as DNS zones tied to networks
Built-in metrics (prometheus) endpoint exposing CPU/memory/disk/network usage of all local instances
This will let me deprecate some of those side projects I had to start as part of this work, will reduce the amount of manual labor involved in setting up all the DNS records and will give me much better insight on what’s consuming resources on the cluster.
I’m also in the process of securing my own ASN and address space through ARIN, mostly because that seemed like a fun thing to do and will give me a tiny bit more flexibility too (not to mention let me consolidate a whole bunch of subnets). So soon enough, I expect to have to deal with quite a bit of re-addressing, but I’m sure it will be a fun and interesting experience!
The previous post went over the planned redundancy aspect of this setup at the storage, networking and control plane level. Now let’s see how to get those systems installed and configured for this setup.
Firmware updates and configuration
First thing first, whether its systems coming from an eBay seller or straight from the factory, the first step is always to update all firmware to the latest available.
In my case, that meant updating the SuperMicro BMC firmware and then the BIOS/UEFI firmware too. Once done, perform a factory reset of both the BMC and UEFI config and then go through the configuration to get something that suits your needs.
The main things I had to tweak other than the usual network settings and accounts were:
Switch the firmware to UEFI only and enable Secure Boot This involves flipping all option ROMs to EFI, disabling CSM and enabling Secure Boot using the default keys.
Enable SR-IOV/IOMMU support Useful if you ever want to use SR-IOV or PCI device passthrough.
Disable unused devices In my case, the only storage backplane is connected to a SAS controller with nothing plugged into the SATA controller, so I disabled it.
Tweak storage drive classification The firmware allows configuring if a drive is HDD or SSD, presumably to control spin up on boot.
Base OS install
With that done, I grabbed the Ubuntu 20.04.1 LTS server ISO, dumped it onto a USB stick and booted the servers from it.
I had all servers and their BMCs connected to my existing lab network to make things easy for the initial setup, it’s easier to do complex network configuration after the initial installation.
The main thing to get right at this step is the basic partitioning for your OS drive. My original plan was to carve off some space from the NVME drive for the OS, unfortunately after an initial installation done that way, I realized that my motherboard doesn’t support NVME booting so ended up reinstalling, this time carving out some space from the SATA SSD instead.
In my case, I ended up creating a 35GB root partition (ext4) and 4GB swap partition, leaving the rest of the 2TB drive unpartitioned for later use by Ceph.
With the install done, make sure you can SSH into the system, also check that you can access the console through the BMC both through VGA and through the IPMI text console. That last part can be done by dumping a file in /etc/default/grub.d/ that looks like:
Finally you’ll want to make sure you apply any pending updates and reboot, then check dmesg for anything suspicious coming from the kernel. Better catch compatibility and hardware issues early on.
Networking setup
On the networking front you may remember I’ve gotten configs with 6 NICs, two gigabit ports and four 10gbit ports. The gigabit NICs are bonded together and go to the switch, the 10gbit ports are used to create a mesh with each server using a two ports bond to the others.
Combined with the dedicated BMC ports, this ends up looking like this:
Here we can see the switch receiving its uplink over LC fiber, each server has its BMC plugged into a separate switch port and VLAN (green cables), each server is also connected to the switch with a two port bond (black cables) and each server is connected to the other two using a two port bond (blue cables).
Ubuntu uses Netplan for its network configuration these days, the configuration on those servers looks something like this:
That’s the part which is common to all servers, then on top of that, each server needs its own tiny bit of config to setup the right routes to its other two peers, this looks like this:
network:
version: 2
bonds:
# server 2
bond-mesh01:
addresses:
- 2602:XXXX:Y:ZZZ::101/64
routes:
- to: 2602:XXXX:Y:ZZZ::100/128
via: fe80::ec7c:7eff:fe69:55fa
# server 3
bond-mesh02:
addresses:
- 2602:XXXX:Y:ZZZ::101/64
routes:
- to: 2602:XXXX:Y:ZZZ::102/128
via: fe80::8cd6:b3ff:fe53:7cc
bridges:
br-hosts:
addresses:
- 2602:XXXX:Y:ZZZ::101/64
My setup is pretty much entirely IPv6 except for a tiny bit of IPv4 for some specific services so that’s why everything above very much relies on IPv6 addressing, but the same could certainly be done using IPv4 instead.
With this setup, I have a 2Gbit/s bond to the top of the rack switch configured to use static addressing but using the gateway provided through IPv6 router advertisements. I then have a first 20Gbit/s bond to the second server with a static route for its IP and then another identical bond to the third server.
This allows all three servers to communicate at 20Gbit/s and then at 2Gbit/s to the outside world. The fast links will almost exclusively be carrying Ceph, OVN and LXD internal traffic, the kind of traffic that’s using a lot of bandwidth and requires good latency.
To complete the network setup, OVN is installed using the ovn-central and ovn-host packages from Ubuntu and then configured to communicate using the internal mesh subnet.
This part is done by editing /etc/default/ovn-central on all 3 systems and updating OVN_CTL_OPTS to pass a number of additional parameters:
--db-nb-addr to the local address
--db-sb-addr to the local address
--db-nb-cluster-local-addr to the local address
--db-sb-cluster-local-addr to the local address
--db-nb-cluster-remote-addr to the first server’s address
--db-sb-cluster-remote-addr to the first server’s address
--ovn-northd-nb-db to all the addresses (port 6641)
--ovn-northd-sb-db to all the addresses (port 6642)
The first server shouldn’t have the remote-addr ones set as it’s the bootstrap server, the others will then join that initial server and join the cluster at which point that startup argument isn’t needed anymore (but it doesn’t really hurt to keep it in the config).
If OVN was running unclustered, you’ll want to reset it by wiping /var/lib/ovn and restarting ovn-central.service.
Storage setup
On the storage side, I won’t go over how to get a three nodes Ceph cluster, there are many different ways to achieve that using just about every deployment/configuration management tool in existence as well as upstream’s own ceph-deploy tool.
In short, the first step is to deploy a Ceph monitor (ceph-mon) per server, followed by a Ceph manager (ceph-mgr) and a Ceph metadata server (ceph-mds). With that done, one Ceph OSD (ceph-osd) per drive needs to be setup. In my case, both the HDDs and the NVME SSD are consumed in full for this while for the SATA SSD I created a partition using the remaining space from the installation and put that into Ceph.
At that stage, you may want to learn about Ceph crush maps and do any tweaking that you want based on your storage setup.
In my case, I have two custom crush rules, one which targets exclusively HDDs and one which targets exclusively SSDs. I’ve also made sure that each drive has the proper device class and I’ve tweaked the affinity a bit such that the faster drives will be prioritized for the first replica.
I’ve also created an initial ceph fs filesystem for use by LXD with:
ceph osd pool create lxd-cephfs_metadata 32 32 replicated replicated_rule_ssd
ceph osd pool create lxd-cephfs_data 32 32 replicated replicated_rule_hdd
ceph fs new lxd-cephfs lxd-cephfs_metadata lxd-cephfs_data
ceph fs set lxd-cephfs allow_new_snaps true
This makes use of those custom rules, putting the metadata on SSD with the actual data on HDD.
The cluster should then look something a bit like that:
root@langara:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 34.02979 root default
-3 11.34326 host abydos
4 hdd 3.63869 osd.4 up 1.00000 0.12500
7 hdd 5.45799 osd.7 up 1.00000 0.25000
0 ssd 0.46579 osd.0 up 1.00000 1.00000
10 ssd 1.78079 osd.10 up 1.00000 0.75000
-5 11.34326 host langara
5 hdd 3.63869 osd.5 up 1.00000 0.12500
8 hdd 5.45799 osd.8 up 1.00000 0.25000
1 ssd 0.46579 osd.1 up 1.00000 1.00000
11 ssd 1.78079 osd.11 up 1.00000 0.75000
-7 11.34326 host orilla
3 hdd 3.63869 osd.3 up 1.00000 0.12500
6 hdd 5.45799 osd.6 up 1.00000 0.25000
2 ssd 0.46579 osd.2 up 1.00000 1.00000
9 ssd 1.78079 osd.9 up 1.00000 0.75000
LXD setup
The last piece is building up a LXD cluster which will then be configured to consume both the OVN networking and Ceph storage.
For OVN support, using an LTS branch of LXD won’t work as 4.0 LTS predates OVN support, so instead I’ll be using the latest stable release.
Installation is as simple as: snap install lxd --channel=latest/stable
Then on run lxd init on the first server, answer yes to the clustering question, make sure the hostname is correct and that the address used is that on the mesh subnet, then create the new cluster setting an initial password and skipping over all the storage and network questions, it’s easier to configure those by hand later on.
After that, run lxd init on the remaining two servers, this time pointing them to the first server to join the existing cluster.
In my case, I’ve also setup a lxd-hdd pool, resulting in a final setup of:
root@langara:~# lxc storage list
+--------+-------------+--------+---------+---------+
| NAME | DESCRIPTION | DRIVER | STATE | USED BY |
+--------+-------------+--------+---------+---------+
| hdd | | ceph | CREATED | 1 |
+--------+-------------+--------+---------+---------+
| shared | | cephfs | CREATED | 0 |
+--------+-------------+--------+---------+---------+
| ssd | | ceph | CREATED | 16 |
+--------+-------------+--------+---------+---------+
Up next
The next post is likely to be quite network heavy, going into why I’m using dynamic routing and how I’ve got it all setup. This is the missing piece of the puzzle in what I’ve shown so far as without it, you’d need an external router with a bunch of static routes to send traffic to the OVN networks.
Github
Twitter
LinkedIn
Mastodon
RSS feed