Some of the Incus maintainers will be present at FOSDEM 2025, helping run both the containers and kernel devrooms. For those arriving in town early, there will be a “Friends of Incus” gathering sponsored by FuturFusion on Thursday evening (January 30th), you can find the details of that here.
And as always, my company is offering commercial support on Incus, ranging from by-the-hour support contracts to one-off services on things like initial migration from LXD, review of your deployment to squeeze the most out of Incus or even feature sponsorship. You’ll find all details of that here: https://zabbly.com/incus
Donations towards my work on this and other open source projects is also always appreciated, you can find me on Github Sponsors, Patreon and Ko-fi.
A lot has happened in 2024 for the Incus project, so I thought it’d be interesting to see where we started, what we did and where we ended up after that very busy year, then look forward to what’s next in 2025!
Where we started
We began 2024 right on the heels of the Incus 0.4 release at the end of December 2023.
This means that effectively everything that made it into Incus in 2024 originated directly from the Incus community. There is one small exception to that as LXD 5.0 LTS still saw some activity and as that’s still under Apache 2.0, we were able to import a few commits (83 to be exact) from that branch.
Incus 6.0 LTS was released at the beginning of April, alongside LXC and LXCFS 6.0 LTS. All of which get 5 years of security support.
That was a huge milestone for Incus as it now allowed production users who don’t feel like going through an update cycle every month to switch over to Incus 6.0 LTS and have a stable production release for the years to come.
It also provides a much easier packaging target for Linux distributions as the monthly releases can be tricky to follow, especially when they introduce new dependencies.
Today, Incus 6.0 LTS represents around 50% of the Incus user base.
Notable feature additions
It’s difficult to come up with a list of the most notable new features because so much happened all over the place and deciding what’s notable ends up being very personal and subjective, depending on one’s usage of Incus, but here are a few!
Application container support (OCI), gives us the ability to natively run Docker containers on Incus
Clustered LVM storage backend, adds support for iSCSI/NVMEoTCP/FC storage in clusters
Network integrations (OVN inter-connect), allows for cross-cluster overlay networking
Automatic cluster re-balancing, simplifies operation of large clusters
Performance improvements
As more and more users run very large Incus systems, a number of performance issues were noticed and have been fixed.
An early one was related to how Incus handled OVN. The old implementation relied on the OVN command line tools to drive OVN database changes. This is incredibly inefficient as each call to those tools would require new TLS handshakes with all database servers, tracking down the leader, fetching a new copy of the database, performing a trivial operation and exiting. The new implementation uses a native database client directly in Incus which maintains a constant connection with the database, gets notified of changes and can instantly perform any needed configuration changes.
Then there were 2-3 different cases of database performance issues. Two of them were caused by our auto-generated database helpers which weren’t very smart about handling of profiles, effectively causing a situation where performance would get exponentially worse as more profiles would be present in the database. Addressing this issue resulted in dramatic performance improvement for users operating with hundreds or even thousands of profiles.
Another was related to loading of instances on Incus startup, specifically loading the device definitions to check whether anything needed to be done on startup. This logic was always hitting configuration validation which can be costly, in this case, so costly that Incus would fail to startup during the allotted time by the init system (10 minutes). After some fixes to that logic, the affected system, running over 2000 virtual machines (on a single server) at the time, is now able to process all running VMs in just 10-15s.
On top of those issues, special attention was also put in optimizing resource usage on large systems, especially systems with multiple NUMA nodes, supporting basic NUMA balancing of virtual machines as well as selecting the best GPU devices based on NUMA cost.
Distribution integration
Back at the beginning of 2024, Incus was only available through my own packages for Debian or Ubuntu, or through native packages on Gentoo and NixOS.
This has changed considerably through 2024 with Incus now being readily available on:
Alpine Linux
Arch Linux
Chimera Linux
Debian
Fedora
Gentoo
NixOS
openSUSE
Rocky Linux
Ubuntu
Void Linux
Additionally, it’s also available as a Docker container to run on most any other platforms as well as available on MacOS through Colima. The client tool itself is available everywhere that Go supports.
Deployment tooling
Terraform/OpenTofu provider
The Incus Terraform/OpenTofu provider has seen quite a lot of activity this year.
We’re slowly headed towards a 1.0 release for it, basically ensuring that it can drive every single Incus feature and that its resources are defined in a clear and consistent way.
There is only one issue left in the 1.0 release milestone and there is an open pull request for it, so we are very close to where we want as far as feature coverage and with a few more bugfixes here and there, we should have that 1.0 release out in the coming weeks/month!
incus-deploy
incus-deploy was introduced in February and is basically a collection of Ansible and Terraform that allows for easy deployment of Incus, whether standalone or clustered and whether for testing/development or production.
This is commonly used by the Incus team to quickly deploy test clusters, complete with Ceph, OVN, clustered LVM, … all in a very reproducible way.
Incus OS
While incus-deploy provides an automated way to deploy Incus on top of traditional Linux servers, Incus OS is working on providing a solution for those who don’t want to have to deal with maintaining traditional Linux servers.
This is a fully immutable OS image, kept as minimal as possible and solely focused on running Incus.
It heavily relies on systemd tooling to provide a secure environment, starting from SecureBoot signing, to having every step of the boot be TPM measured, to having storage encrypted using that TPM state and the entire read-only disk image being verified through dm-verity.
The end result is an extremely secure and locked down environment which is designed for just one thing, running Incus!
We’re getting close to having something ready for early adopters with automated builds and update logic now working, but it will be a few more weeks before it’s safe/useful to install on a server.
Where we ended up
Over that year, Incus really turned into a full fledged Open Source project and community.
We have kept on with our release cadence, pushing out a new feature release every month while very actively backporting bugfixes and smaller improvements to our LTS release.
Distributions have done a great job at getting Incus packaged, making it natively available just about everywhere (we’re still waiting on solid EPEL packaging).
Our supporting projects like terraform-provider-incus, incus-deploy and incus-os are making it easier than ever to deploy and operate large scale Incus clusters as well as providing a simpler, more repeatable way of running Incus.
2024 was a very very good year for Incus!
What’s coming in 2025
Looking ahead, 2025 has the potential to be and even better year for us!
On the Incus front, there are no single huge feature to be looking forward to, but just the continual improvement, whether it be for containers, VMs, networking or clustering. We have a lot of small new features and polishing in mind which will help fill in some of the current gaps and provide a nice and consistent experience.
But it’s on the supporting projects that a lot of the potential now rests.
This will hopefully be the year of Incus OS, making installing Incus as easy as writing a file to a USB stick, booting a machine from it and accessing it over the network. Want to make a cluster, no problem, just boot a few more machines onto Incus OS and join them together as a cluster!
But we’re also going to be expanding incus-deploy. It’s currently doing a good job at deploying Incus on Ubuntu servers with Ansible but we want to expand that to also cover Debian and some of the RHEL derivatives so we can cover the majority of our current production users with it. On top of that, we want to also have incus-deploy handle setting up the common support services used by Incus clusters, typically OpenFGA, Keycloak, Grafana, Prometheus and Loki.
We also want to improve our testing and development lab, add more systems, add the ability to test on more architectures and easily test more complex features, whether it’s 100Gb/s+ networking with full hardware offload or confidential computing features like AMD SEV.
Sovereign Tech Fund
Thankfully a lot of that is going to be made a whole lot easier thanks to funding by the Sovereign Tech Fund who’s going to be supporting a variety of Incus related projects, especially focusing on the kind of work that’s not particularly exciting but is very much critical to the proper running of a project like ours.
This includes a big refresh of our testing and development lab, work on our LTS releases, new security features through the stack, improved support for other Linux distributions and OSes across our projects and more!
The Linux Containers project maintains Long Term Support (LTS) releases for its core projects. Those come with 5 years of support from upstream with the first two years including bugfixes, minor improvements and security fixes and the remaining 3 years getting only security fixes.
This is now the third round of bugfix releases for LXC, LXCFS and Incus 6.0 LTS.
LXC
LXC is the oldest Linux Containers project and the basis for almost every other one of our projects. This low-level container runtime and library was first released in August 2008, led to the creation of projects like Docker and today is still actively used directly or indirectly on millions of systems.
Reworked testsuite including support for 64-bit Arm
LXCFS
LXCFS is a FUSE filesystem used to workaround some shortcomings of the Linux kernel when it comes to reporting available system resources to processes running in containers. The project started in late 2014 and is still actively used by Incus today as well as by some Docker and Kubernetes users.
Reworked testsuite including support for 64-bit Arm
Incus
Incus is our most actively developed project. This virtualization platform is just over a year old but has already seen over 3500 commits by over 120 individual contributors. Its first LTS release made it usable in production environments and significantly boosted its user base.
Configurable columns, formatting and refresh time in incus top
Support for DHCP ranges in OVN (ipv4.dhcp.ranges)
Support for changing the backing interface of a managed physical network
Extended QEMU scriptlet (additional functions)
New log file for QEMU QMP traffic (qemu.qmp.log)
New get_instances_count function available in placement scriptlet
Support for --format in incus admin sql
Storage live migration for virtual machines
New authorization scriptlet as an alternative to OpenFGA
API to retrieve console screenshots
Configurable initial owner for custom storage volumes (initial.uid, initial.gid, initial.mode)
Image alias reuse on import (incus image import --reuse --alias)
New incus-simplestreams prune command
Console access locking (incus console --force to override)
What’s next?
We’re expecting another LTS bugfix release for the 6.0 branches in the first quarter of 2025. We’re also actively working on a new stable release (non-LTS) for LXCFS. Incus will keep going with its usual monthly feature release cadence.
Thanks
This LTS release update was made possible thanks to funding provided by the Sovereign Tech Fund (now part of the Sovereign Tech Agency).
The Sovereign Tech Fund supports the development, improvement, and maintenance of open digital infrastructure. Its goal is to sustainably strengthen the open source ecosystem, focusing on security, resilience, technological diversity, and the people behind the code.
The Incus team is pleased to announce the release of Incus 6.8!
This is the last release for 2024 but it still packs a punch with a bunch of VM related improvements, including the ability to move a running VM between storage pools, a new authorization backend, improvements to volume handling for application containers and more.
The highlights for this release are:
Storage live migration for VMs
Authorization scriptlet
Console screenshots for VMs
Initial owner and mode for custom storage volumes
Small updates to the OpenFGA model
Image alias reuse on import
New incus-simplestreams prune command
Console access locking
The full announcement and changelog can be found here. And for those who prefer videos, here’s the release overview video:
Some of the Incus maintainers will be present at FOSDEM 2025, helping run both the containers and kernel devrooms. For those arriving in town early, there will be a “Friends of Incus” gathering sponsored by FuturFusion on Thursday evening (January 30th), you can find the details of that here.
And as always, my company is offering commercial support on Incus, ranging from by-the-hour support contracts to one-off services on things like initial migration from LXD, review of your deployment to squeeze the most out of Incus or even feature sponsorship. You’ll find all details of that here: https://zabbly.com/incus
Donations towards my work on this and other open source projects is also always appreciated, you can find me on Github Sponsors, Patreon and Ko-fi.
The Incus team is pleased to announce the release of Incus 6.7!
This is another one of those pretty well rounded releases with new features and improvements for everyone from standalone users to those running a small homelab all the way to large scale cluster users, there’s something for everyone!
The highlights for this release are:
Easy access to the Incus web interface
Automatic cluster re-balancing
DHCP renewal for OCI containers
Partial instance/volume refresh
Configurable columns, formatting and refresh time for incus top
Support for DHCP address ranges in OVN networks
Changing of parent device for physical networks
Aditional QMP helpers in QEMU scriptlet
Additional QEMU log file for QMP commands
New get_instances_count command for placement scriptlet
Support of formatting in incus admin sql command
The full announcement and changelog can be found here. And for those who prefer videos, here’s the release overview video:
And as always, my company is offering commercial support on Incus, ranging from by-the-hour support contracts to one-off services on things like initial migration from LXD, review of your deployment to squeeze the most out of Incus or even feature sponsorship. You’ll find all details of that here: https://zabbly.com/incus
Donations towards my work on this and other open source projects is also always appreciated, you can find me on Github Sponsors, Patreon and Ko-fi.
Github
Twitter
LinkedIn
Mastodon
RSS feed