Category Archives: LXCFS

LXC/LXCFS/Incus 6.0.3 LTS release

Posted on 2024/12/20 by Stéphane Graber

Introduction

The Linux Containers project maintains Long Term Support (LTS) releases for its core projects. Those come with 5 years of support from upstream with the first two years including bugfixes, minor improvements and security fixes and the remaining 3 years getting only security fixes.

This is now the third round of bugfix releases for LXC, LXCFS and Incus 6.0 LTS.

LXC

LXC is the oldest Linux Containers project and the basis for almost every other one of our projects. This low-level container runtime and library was first released in August 2008, led to the creation of projects like Docker and today is still actively used directly or indirectly on millions of systems.

Announcement: https://discuss.linuxcontainers.org/t/lxc-6-0-3-lts-has-been-released/22402

Highlights of this point release:

  • Added support for PuzzleFS images in lxc-oci
  • SIGHUP is now propagated through lxc.init
  • Reworked testsuite including support for 64-bit Arm

LXCFS

LXCFS is a FUSE filesystem used to workaround some shortcomings of the Linux kernel when it comes to reporting available system resources to processes running in containers. The project started in late 2014 and is still actively used by Incus today as well as by some Docker and Kubernetes users.

Announcement: https://discuss.linuxcontainers.org/t/lxcfs-6-0-3-lts-has-been-released/22401

Highlights of this point release:

  • Better detection of swap accounting support
  • Reworked testsuite including support for 64-bit Arm

Incus

Incus is our most actively developed project. This virtualization platform is just over a year old but has already seen over 3500 commits by over 120 individual contributors. Its first LTS release made it usable in production environments and significantly boosted its user base.

Announcement: https://discuss.linuxcontainers.org/t/incus-6-0-3-lts-has-been-released/22403

Highlights of this point release:

  • OS info for virtual machines (incus info)
  • Console history for virtual machines (incus console --show-log)
  • Ability to create clustered LVM pools directly through Incus
  • QCOW2 and VMDK support in incus-migrate
  • Configurable macvlan mode (bridge, vepa, passthru or private)
  • Load-balancer health information (incus network load-balancer info)
  • External interfaces in OVN networks (support for bridge.external_interfaces)
  • Parallel cluster evacuation/restore (on systems with large number of CPUs)
  • Introduction of incus webui as a quick way to access the web interface
  • Automatic cluster re-balancing
  • Partial instance/volume refresh (incus copy --refresh-exclude-older --refresh)
  • Configurable columns, formatting and refresh time in incus top
  • Support for DHCP ranges in OVN (ipv4.dhcp.ranges)
  • Support for changing the backing interface of a managed physical network
  • Extended QEMU scriptlet (additional functions)
  • New log file for QEMU QMP traffic (qemu.qmp.log)
  • New get_instances_count function available in placement scriptlet
  • Support for --format in incus admin sql
  • Storage live migration for virtual machines
  • New authorization scriptlet as an alternative to OpenFGA
  • API to retrieve console screenshots
  • Configurable initial owner for custom storage volumes (initial.uid, initial.gid, initial.mode)
  • Image alias reuse on import (incus image import --reuse --alias)
  • New incus-simplestreams prune command
  • Console access locking (incus console --force to override)

What’s next?

We’re expecting another LTS bugfix release for the 6.0 branches in the first quarter of 2025.
We’re also actively working on a new stable release (non-LTS) for LXCFS.
Incus will keep going with its usual monthly feature release cadence.

Thanks

This LTS release update was made possible thanks to funding provided by the Sovereign Tech Fund (now part of the Sovereign Tech Agency).

The Sovereign Tech Fund supports the development, improvement, and maintenance of open digital infrastructure. Its goal is to sustainably strengthen the open source ecosystem, focusing on security, resilience, technological diversity, and the people behind the code.

Find out more at: https://www.sovereign.tech

Posted in Incus, LXC, LXCFS, Planet Ubuntu | Leave a comment

LXC/LXCFS/Incus 6.0.2 LTS release

Posted on 2024/09/17 by Stéphane Graber

Introduction

The Linux Containers project maintains Long Term Support (LTS) releases for its core projects.
Those come with 5 years of support from upstream with the first two years including bugfixes, minor improvements and security fixes and the remaining 3 years getting only security fixes.

This is now the second round of bugfix releases for LXC, LXCFS and Incus 6.0 LTS.

LXC

LXC is the oldest Linux Containers project and the basis for almost every other one of our projects.
This low-level container runtime and library was first released in August 2008, led to the creation of projects like Docker and today is still actively used directly or indirectly on millions of systems.

Announcement: https://discuss.linuxcontainers.org/t/lxc-6-0-2-lts-has-been-released/21632

Highlights of this point release:

  • Reduced log level on some common messages
  • Fix compilation error on aarch64

LXCFS

LXCFS is a FUSE filesystem used to workaround some shortcomings of the Linux kernel when it comes to reporting available system resources to processes running in containers.
The project started in late 2014 and is still actively used by Incus today as well as by some Docker and Kubernetes users.

Announcement: https://discuss.linuxcontainers.org/t/lxcfs-6-0-2-lts-has-been-released/21631

Highlights of this point release:

  • Fix building of LXCFS on musl systems (missing include)

Incus

Incus is our most actively developed project. This virtualization platform is just over a year old but has already seen over 3500 commits by over 120 individual contributors. Its first LTS release made it usable in production environments and significantly boosted its user base.

Announcement: https://discuss.linuxcontainers.org/t/incus-6-0-2-lts-has-been-released/21633

Highlights of this point release:

  • Completion of transition to native OVSDB for OVS/OVN
  • Baseline CPU defintiion for clustered users
  • Filesystem support for io.bus and io.cache
  • CPU flags in server resources
  • Unified image support in incus-simplestreams
  • Completion of libovsdb transition
  • Using a sub-path of a volume as a disk
  • Per storage pool projects limits
  • Isolated OVN networks (no uplink)
  • Per-instance LXCFS
  • Support for environment file at create/launch time
  • Instance auto-restart
  • Column selection in all list commands
  • QMP command hooks and scriptlet
  • Live disk resize support in virtual machines
  • PCI devices hotplug
  • OVN load-balancer health checks
  • Promiscuous mode for OVN NICs
  • Ability to run off IP allocation on OVN NICs
  • Customizable OIDC scope request
  • Configurable LVM PV metadata size
  • Configurable OVS socket path

What’s next?

We’re expecting another LTS bugfix release for the 6.0 branches later this year.

On top of that, Q4 of 2024 will also feature non-LTS feature releases of both LXC and LXCFS as we’re trying to push out new releases of those two projects every 6 months now.

Incus will keep going with its usual monthly feature release cadence.

Posted in Incus, LXC, LXCFS, Planet Ubuntu | 1 Comment

One year of freelancing

Posted on 2024/07/07 by Stéphane Graber

Introduction

It was exactly one year ago today that I left my day job as Engineering Manager of LXD at Canonical and went freelance. It’s been quite a busy year but things turned out better than I had hoped and I’m excited about year two!

Zabbly

Zabbly is the company I created for my freelance work. Over the year, a number of my personal projects were transferred over to being part of Zabbly, including the operation of my ASN (as399760.net), my datacenter co-location infrastructure and more.

Through Zabbly I offer a mix of by-the-hour consultation with varying prices depending on the urgency of the work (basic consultation, support, emergency support) as well as fixed-cost services, mostly related to Incus (infrastructure review, migration from LXD, remote or on-site trainings, …).

Other than Incus, Zabbly also provides up to date mainline kernel packages for Debian and Ubuntu and associated up to date ZFS packages. This is something that came out as needed for a number of projects I work on, from being able to test Incus on recent Linux kernels to avoiding Ubuntu kernel bugs on my own and NorthSec’s servers.

Zabbly is also the legal entity for donations related to my open source work, currently supporting:

And lastly, Zabbly also runs a Youtube channel covering the various projects I’m involved with.
A lot of it is currently about Incus, but there is also the occasional content on NorthSec or other side projects. The channel grew to a bit over 800 subscribers in the past 10 months or so.

Now, how well is all of that doing? Well enough that I could stop relying on my savings just a few months in and turn a profit by the end of 2023. Zabbly currently has around a dozen active customers from 7 countries and across 3 continents with size ranging from individuals to large governmental agencies.

2024 has also been very good so far and while I’m not back to the level of income I had while at Canonical, I also don’t have to go through 4-5 hours of meetings a day and get to actually contribute to open source again, so I’ll gladly take the (likely temporary) pay cut!

Incus

A lot of my time in the past year has been dedicated to Incus.

This wasn’t exactly what I had planned when leaving Canonical.
I was expecting LXD to keep on going as a proper Open Source project as part of the Linux Containers community. But Canonical had other plans and so things changed a fair bit over the few months following my departure.

For those not aware, the rough timeline of what happened is:

So rather than contributing to LXD while working on some other new projects, a lot of my time has instead gone into setting up the Incus project for success.

And I think I’ve been pretty successful at that as we’re seeing a monthly user base growth (based on image server interactions) of around 25%. Incus is now natively available in most Linux distributions (Alpine, Arch Linux, Debian, Gentoo, Nix, Ubuntu and Void) with more coming soon (Fedora and EPEL).

Incus has 6 maintainers, most of whom were the original LXD maintainers.
We’ve seen over 100 individual contributors since Incus was forked from LXD including around 20 students from the University of Texas in Austin who contributed to Incus as part of their virtualization class.

I’ve been acting as the release manager for Incus, also running all the infrastructure behind the project, mentoring new contributors and reviewing a number of changes while also contributing a number of new features myself, some sponsored by my customers, some just based on my personal interests.

A big milestone for Incus was its 6.0 LTS release as that made it suitable for production users.
Today we’re seeing around 40% of our users running the LTS release while the rest run the monthly releases.

On top of Incus itself, I’ve also gotten to contribute to both create the Incus Deploy project, which is a collection of Ansible playbooks and Terraform modules to make it easy to deploy Incus clusters and contribute to both the Ansible Incus connection plugin and our Incus Terraform/OpenTofu provider.

The other Linux Containers projects

As mentioned in my recent post about the 6.0.1 LTS releases, the Linux Containers project tries to do coordinated LTS releases on our core projects. This currently includes LXC, LXCFS and Incus.

I didn’t have to do too much work myself on LXC and LXCFS, thanks to Aleksandr Mikhalitsyn from the Canonical LXD team who’s been dealing with most of the review and issues in both LXC and LXCFS alongside other long time maintainers, Serge Hallyn and Christian Brauner.

NorthSec

NorthSec is a yearly cybersecurity conference, CTF and training provider, usually happening in late May in Montreal, Canada. It’s been operating since 2013 and is now one of the largest on-site CTF events in the world along with having a pretty sizable conference too.

I’m the current VP of Infrastructure for the event and have been involved with it from the beginning, designing and running its infrastructure, first on a bunch of old donated hardware and then slowly modernizing that to the environment we have now with proper production hardware both at our datacenter and on-site during the event.

This year, other than transitioning everything from LXD to Incus, the main focus has been on upgrading the OS on our 6 physical servers and dozens of infrastructure containers and VMs from Ubuntu 20.04 LTS to Ubuntu 24.04 LTS.

At the same time, also significantly reducing the complexity of our infrastructure by operating a single unified Incus cluster, switching to OpenID Connect and OpenFGA for access control and automating even more of our yearly infrastructure with Ansible and Terraform.

Automation is really key with NorthSec as it’s a non-profit organization with a lot of staffing changes every year, around 100 year long contributors and then an additional 50 or so on-site volunteers!

I went over the NorthSec infrastructure in a couple of YouTube videos:

Conferences

I’ve cut down and focused my conference attendance a fair bit over this past year.
Part of it for budgetary reasons, part of it because of having so many things going on that fitting another couple of weeks of cross-country travel was difficult.

I decided to keep attending two main events. The Linux Plumbers Conference where I co-organizer the Containers and Checkpoint-Restore Micro-Conference and FOSDEM where I co-organize both the Containers and the Kernel devrooms.

With one event usually in September/October and the other in February, this provides two good opportunities to catch up with other developers and users, get to chat a bunch and make plans for the year.

I’m looking forward to catching up with folks at the upcoming Linux Plumbers Conference in Vienna, Austria!

What’s next

I’ve got quite a lot going on, so the remaining half of 2024 and first half of 2025 are going to be quite busy and exciting!

On the Incus front, we’ve got some exciting new features coming in, like the native OCI container support, more storage options, more virtual networking features, improved deployment tooling, full coverage of Incus features in Terraform/OpenTofu and even a small immutable OS image!

NorthSec is currently wrapping up a few last items related to its 2024 edition and then it will be time to set up the development infrastructure and get started on organizing 2025!

For conferences, as mentioned above, I’ll be in Vienna, Austria in September for Linux Plumbers and expect to be in Brussels again for FOSDEM in February.

There’s also more that I’m not quite ready to talk about, but expect some great Incus related news to come out in the next few months!

Posted in Conferences, Incus, LXC, LXCFS, Planet Ubuntu, Zabbly | 2 Comments

LXC/LXCFS/Incus 6.0.1 LTS release

Posted on 2024/07/01 by Stéphane Graber

Introduction

The Linux Containers project maintains Long Term Support (LTS) releases for its core projects.
Those come with 5 years of support from upstream with the first two years including bugfixes, minor improvements and security fixes and the remaining 3 years getting only security fixes.

Our current LTS release, 6.0, is as the name implies the 6th time we’ve released an LTS release of our projects, starting over 10 years ago, in February 2014.

At the time of writing, we have three currently supported LTS releases:

  • 4.0 (supported until June 2025, security-only)
  • 5.0 (supported until June 2027, security-only)
  • 6.0 (supported until June 2029).

The 6.0 LTS release begun in April 2024 and was the first to include Incus.

LXC

LXC is the oldest Linux Containers project and the basis for almost every other one of our projects.
This low-level container runtime and library was first released in August 2008, led to the creation of projects like Docker and today is still actively used directly or indirectly on millions of systems.

Announcement: https://discuss.linuxcontainers.org/t/lxc-6-0-1-lts-has-been-released/20283

Highlights of this point release:

  • Fixed some build tooling issues
  • Fixed startup failures on system without IPv6 support
  • Updated AppArmor rules to avoid potential warnings

LXCFS

LXCFS is a FUSE filesystem used to workaround some shortcomings of the Linux kernel when it comes to reporting available system resources to processes running in containers.
The project started in late 2014 and is still actively used by Incus today as well as by some Docker and Kubernetes users.

Unfortunately the LXCFS approach is starting to run into issues due to tools relying more and more on system call interfaces or other methods to obtain resource information these days requiring more complex solution such as Incus’ system call interception support (using the Seccomp Notifier).

Because of that development, we’ve been slowly discussing better ways to provide reliable resource information to userspace without having to rely on filesystem tricks or costly system call interception, but as with anything that requires widespread userspace adoption, it will take a while until such a solution is in place and so LXCFS isn’t going anywhere any time soon!

Announcement: https://discuss.linuxcontainers.org/t/lxcfs-6-0-1-lts-has-been-released/20277

Highlights of this point release:

  • Support for running multiple instances of LXCFS (--runtime-dir)
  • Detect systems that has a Yama policy preventing reading process personalities

Incus

Incus is our most actively developed project. This virtualization platform is less than a year old but has already seen over 3000 commits by over 100 individual contributors. Its first LTS release made it usable in production environments and significantly boosted its user base.

Announcement: https://discuss.linuxcontainers.org/t/incus-6-0-1-lts-has-been-released/20297

Highlights of this point release:

  • Extended source syntax for ZFS pools (allows mirror & raidz1/raidz2)
  • Cross-project listing on all objects (instances, profiles, images, storage volumes/buckets, networks, …)
  • Additional functions exposed to instance placement scriptlet
  • All create sub-commands in the CLI now accept YAML input
  • All list sub-commands in the CLI now accept customizable columns
  • The migration.stateful config key was expanded to containers too
  • Stateless network ACLs are now supported on OVN
  • New timestamp exposed for instance uptime
  • New incus top command (uses existing metric API)
  • System load information in incus info --resources
  • PCI devices information in incus info --resources
  • Ability to query who has access to a given project or instance
  • Forceful deletion of projects
  • Improved alias handling in incus-simplestreams

What’s next?

We’re going to keep backporting all relevant fixes and minor improvements to our LTS branches and will likely be releasing another LTS point release of those 3 projects later this year.

There is no set schedule on LTS point releases as we instead prefer to wait until we feel there are significant enough fixes to warrant one, then make sure that all three projects are properly tested and ready for a release.

This year we’ve also decided to start releasing non-LTS releases of both LXC and LXCFS.
It’s something we used to do some years ago but then stopped, mostly due to lack of time.
So you can look forward to LXC and LXCFS 6.1 in Q4 of 2024!

Posted in Incus, LXC, LXCFS, Planet Ubuntu | Leave a comment

A month later

Posted on 2023/08/10 by Stéphane Graber

It’s now been a whole month since I left Canonical and started working as an independent!

This has been quite the month, both professionally and personally!
In no particular order, this included, setting up a new business, dealing with a somewhat last minute datacenter move (thankfully just one floor down), doing some initial sponsored work, helping out with a LXD fork, selling a house and caring for a sick cat (now all back to normal).

Given everything that’s been happening, I thought I’d use the opportunity to write down some details on the most relevant things I’ve been doing and what to expect moving forward.

Zabbly

Zabbly is the name of the business I’ve registered here in Canada.

I didn’t really like the idea of doing all business moving forward just under my own name as I may want to sub-contract some aspects of it or even have employees down the line.
Having the business part of my life have its own name will make that a fair bit cleaner.

For now, the main things that have been moved over to Zabbly are my organization and IP allocations with ARIN, membership on the Montreal Internet Exchange (QIX) and a number of associated contracts related to AS399760 (my BGP ASN). As part of that, Zabbly is also now listed as the sponsor for all the Linux Containers infrastructure.

Allowing to more clearly separate personal and work-related expenses is going to be another benefit of this move even if legally and from a tax point of view, it’s still all me.

ZFS delegation

An initial bit of sponsored work I got to do this month has been adding support for ZFS delegation to LXD. This makes use of a ZFS 2.2 feature which allows for a dataset to be delegated to a particular user namespace. The ZFS tools can then be used from within that container to create nested datasets or manage snapshots.

This is very exciting as it was the one feature that btrfs had which ZFS offered no equivalent for. It should allow for things like running Docker with the ZFS backend inside of LXD containers, having VPS users be able to create their own datasets, handled their own snapshots and be able to send and receive datasets.

The pull request can be found here: https://github.com/canonical/lxd/pull/12056

Incus

Some of you may have seen the announcement of a new LXD fork called Incus and its subsequent inclusion into the Linux Containers project.

This was quite an exciting development and the LXC team spent quite a bit of time over the past couple weeks chatting with Aleksa and seeing where things were headed.

On my end, I initially helped out trying to make the thing actually pass the testsuite, quite a bit harder than it may sound when dealing with a pretty big codebase and everything having been renamed! I also contributed some ideas of what such a fork may want to change compared to stock LXD.

It’s not often that you get a second chance at designing something like LXD/Incus.
While having a working upgrade path and good backward compatibility is obviously still very important, the fact that anyone migrating will need to deal with some amount of manual work also makes it possible to do away with past mistakes and remove some bits that are seldom used.

I expect I’ll be spending a bunch of my time over the next couple of months helping get Incus into a releasable state. Continuing with the current cleanups, getting the documentation back into shape, putting CI and publishing infrastructure back online (basically re-using what I was once providing to LXD).

The biggest task yet to come is to write tooling and processes to monitor changes happening in Canonical’s LXD and then cherry-pick those into Incus. Again, the hard fork, name and path changes and variety of other changes is going to make that a bit of a challenge but once done, it should make it quite easy to do weekly syncs and reviews of changes.

What’s next

As mentioned, I expect to spend a fair bit of my time over the next few weeks/months helping out with Incus, getting it into shape for an initial release.

For those who enjoyed the LXD YouTube channel, I’m also setting up a new channel that will primarily cover Incus but also some other of my projects: https://www.youtube.com/@TheZabbly.

I’m all set up for contract work and sponsorship now, so if there’s anything you think I can do for you, feel free to reach out at info@zabbly.com.

I’ve also been added to the Github Sponsors program, so if you’d just like to help out with my work on those various projects, that’s available too: https://github.com/sponsors/stgraber

Posted in Incus, LXC, LXCFS, Planet Ubuntu, Zabbly | 4 Comments