LXC in Ubuntu 12.04 LTS

Quite a few people have been asking for a status update of LXC in Ubuntu as of Ubuntu 12.04 LTS. This post is meant as an overview of the work we did over the past 6 months and pointers to more detailed blog posts for some of the new features.

What’s LXC?

LXC is a userspace tool controlling the kernel namespaces and cgroup features to create system or application containers.

To give you an idea:

  • Feels like somewhere between a chroot and a VM
  • Can run a full distro using the “host” kernel
  • Processes running in a container are visible from the outside
  • Doesn’t require any specific hardware, works on all supported architectures

A libvirt driver for LXC exists (libvirt-lxc), however it doesn’t use the “lxc” userspace tool even though it uses the same kernel features.

Making LXC easier

One of the main focus for 12.04 LTS was to make LXC dead easy to use, to achieve this, we’ve been working on a few different fronts fixing known bugs and improving LXC’s default configuration.

Creating a basic container and starting it on Ubuntu 12.04 LTS is now down to:

sudo apt-get install lxc
sudo lxc-create -t ubuntu -n my-container
sudo lxc-start -n my-container

This will default to using the same version and architecture as your machine, additional option are obviously available (–help will list them). Login/Password are ubuntu/ubuntu.

Another thing we worked on to make LXC easier to work with is reducing the number of hacks required to turn a regular system into a container down to zero.
Starting with 12.04, we don’t do any modification to a standard Ubuntu system to get it running in a container.
It’s now even possible to take a raw VM image and have it boot in a container!

The ubuntu-cloud template also lets you get one of our EC2/cloud images and have it start as a container instead of a cloud instance:

sudo apt-get install lxc cloud-utils
sudo lxc-create -t ubuntu-cloud -n my-cloud-container
sudo lxc-start -n my-cloud-container

And finally, if you want to test the new cool stuff, you can also use juju with LXC:

[ ! -f ~/.ssh/id_rsa.pub ] && ssh-keygen -t rsa
sudo apt-get install juju apt-cacher-ng zookeeper lxc libvirt-bin --no-install-recommends
sudo adduser $USER libvirtd
juju bootstrap
sed -i "s/ec2/local/" ~/.juju/environments.yaml
echo " data-dir: /tmp/juju" >> ~/.juju/environments.yaml
juju bootstrap
juju deploy mysql
juju deploy wordpress
juju add-relation wordpress mysql
juju expose wordpress

# To tail the logs
juju debug-log

# To get the IPs and status
juju status

Making LXC safer

Another main focus for LXC in Ubuntu 12.04 was to make it safe. John Johansen did an amazing work of extending apparmor to let us implement per-container apparmor profiles and prevent most known dangerous behaviours from happening in a container.

NOTE: Until we have user namespaces implemented in the kernel and used by the LXC we will NOT say that LXC is root safe, however the default apparmor profile as shipped in Ubuntu 12.04 LTS is blocking any armful action that we are aware of.

This mostly means that write access to /proc and /sys are heavily restricted, mounting filesystems is also restricted, only allowing known-safe filesystems to be mounted by default. Capabilities are also restricted in the default LXC profile to prevent a container from loading kernel modules or control apparmor.

More details on this are available here:

Other cool new stuff

Emulated architecture containers

It’s now possible to use qemu-user-static with LXC to run containers of non-native architectures, for example:

sudo apt-get install lxc qemu-user-static
sudo lxc-create -n my-armhf-container -t ubuntu -- -a armhf
sudo lxc-start -n my-armhf-container

Ephemeral containers

Quite a bit of work also went into lxc-start-ephemeral, the tool letting you start a copy of an existing container using an overlay filesystem, discarding any change you make on shutdown:

sudo apt-get install lxc
sudo lxc-create -n my-container -t ubuntu
sudo lxc-start-ephemeral -o my-container

Container nesting

You can now start a container inside a container!
For that to work, you first need to create a new apparmor profile as the default one doesn’t allow this for security reason.
I already did that for you, so the few commands below will download it and install it in /etc/apparmor.d/lxc/lxc-with-nesting. This profile (or something close to it) will ship in Ubuntu 12.10 as an example of alternate apparmor profile for container.

sudo apt-get install lxc
sudo lxc-create -t ubuntu -n my-host-container
sudo wget https://www.stgraber.org/download/lxc-with-nesting -O /etc/apparmor.d/lxc/lxc-with-nesting
sudo /etc/init.d/apparmor reload
sudo sed -i "s/#lxc.aa_profile = unconfined/lxc.aa_profile = lxc-container-with-nesting/" /var/lib/lxc/my-host-container/config
sudo lxc-start -n my-host-container
(in my-host-container) sudo apt-get install lxc
(in my-host-container) sudo stop lxc
(in my-host-container) sudo sed -i "s/10.0.3/10.0.4/g" /etc/default/lxc
(in my-host-container) sudo start lxc
(in my-host-container) sudo lxc-create -n my-sub-container -t ubuntu
(in my-host-container) sudo lxc-start -n my-sub-container

Documentation

Outside of the existing manpages and blog posts I mentioned throughout this post, Serge Hallyn did a very good job at creating a whole section dedicated to LXC in the Ubuntu Server Guide.
You can read it here: https://help.ubuntu.com/12.04/serverguide/lxc.html

Next steps

Next week we have the Ubuntu Developer Summit in Oakland, CA. There we’ll be working on the plans for LXC in Ubuntu 12.10. We currently have two sessions scheduled:

If you want to make sure the changes you want will be in Ubuntu 12.10, please make sure to join these two sessions. It’s possible to participate remotely to the Ubuntu Developer Summit, through IRC and audio streaming.

My personal hope for LXC in Ubuntu 12.10 is to have a clean liblxc library that can be used to create bindings and be used in languages like python. Working towards that goal should make it easier to do automated testing of LXC and cleanup our current tools.

I hope this post made you want to try LXC or for existing users, made you discover some of the new features that appeared in Ubuntu 12.04. We’re actively working on improving LXC both upstream and in Ubuntu, so do not hesitate to report bugs (preferably with “ubuntu-bug lxc”).

Posted in Canonical voices, Conferences, LXC, Planet Ubuntu | Tagged | 64 Comments

Booting an Ubuntu 12.04 virtual machine in an LXC container

One thing that we’ve been working on for LXC in 12.04 is getting rid of any remaining LXC specific hack in our templates. This means that you can now run a perfectly clean Ubuntu system in a container without any change.

To better illustrate that, here’s a guide on how to boot a standard Ubuntu VM in a container.

First, you’ll need an Ubuntu VM image in raw disk format. The next few steps also assume a default partitioning where the first primary partition is the root device. Make sure you have the lxc package installed and up to date and lxcbr0 enabled (the default with recent LXC).

Then run kpartx -a vm.img this will create loop devices in /dev/mapper for your VM partitions, in the following configuration I’m assuming /dev/mapper/loop0p1 is the root partition.

Now write a new LXC configuration file (myvm.conf in my case) containing:

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = lxcbr0
lxc.utsname = myvminlxc

lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = /dev/mapper/loop0p1
lxc.arch = amd64
lxc.cap.drop = sys_module mac_admin

lxc.cgroup.devices.deny = a
# Allow any mknod (but not using the node)
lxc.cgroup.devices.allow = c *:* m
lxc.cgroup.devices.allow = b *:* m
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
#lxc.cgroup.devices.allow = c 4:0 rwm
#lxc.cgroup.devices.allow = c 4:1 rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
# rtc
lxc.cgroup.devices.allow = c 254:0 rwm
#fuse
lxc.cgroup.devices.allow = c 10:229 rwm
#tun
lxc.cgroup.devices.allow = c 10:200 rwm
#full
lxc.cgroup.devices.allow = c 1:7 rwm
#hpet
lxc.cgroup.devices.allow = c 10:228 rwm
#kvm
lxc.cgroup.devices.allow = c 10:232 rwm

The bits in bold may need updating if you’re not using the same architecture, partition scheme or bridges as I’m.

Then finally, run: lxc-start -n myvminlxc -f myvm.conf

And watch your VM boot in an LXC container.

I did this test with a desktop VM using network manager so it didn’t mind LXC’s random MAC address, server VMs might get stuck for a minute at boot time because of that though.
In such case, either clean /etc/udev/rules.d/70-persistent-net.rules or set “lxc.network.hwaddr” to the same mac address as your VM.

Once done, run kpartx -d vm.img to remove the loop devices.

Posted in Canonical voices, LXC, Planet Ubuntu | Tagged | 28 Comments

DNS in Ubuntu 12.04

Anyone who’s been using 12.04 over the past month or so may have noticed some pretty significant changes in the way we do DNS resolving in Ubuntu.

This is the result of the implementation of: foundations-p-dns-resolving

Here is a description of the two big changes that happened:

Switch to resolvconf for /etc/resolv.conf management

resolvconf is a set of script and hooks managing DNS resolution. The most notable difference for the user is that any change manually done to /etc/resolv.conf will be lost as it gets overwritten next time something triggers resolvconf. Instead, resolvconf uses DHCP client hooks, a Network Manager plugin and /etc/network/interfaces to generate a list of nameservers and domain to put in /etc/resolv.conf.

For more details, I’d highly encourage you to read resolvconf’s manpage but here are a few answers to common questions:

  • I use static IP configuration, where should I put my DNS configuration?
    The DNS configuration for a static interface should go as “dns-nameservers”, “dns-search” and “dns-domain” entries added to the interface in /etc/network/interfaces
  • How can I override resolvconf’s configuration or append some entries to it?
    Resolvconf has a /etc/resolvconf/resolv.conf.d/ directory that can contain “base”, “head”, “original” and “tail” files. All in resolv.conf format.
    • base: Used when no other data can be found
    • head: Used for the header of resolv.conf, can be used to ensure a DNS server is always the first one in the list
    • original: Just a backup of your resolv.conf at the time of resolvconf installation
    • tail: Any entry in tail is appended at the end of the resulting resolv.conf. In some cases, upgrading from a previous Ubuntu release, will make tail a symlink to original (when we think you manually modified resolv.conf in the past)
  • I really don’t want resolvconf, how can I disable it?
    I certainly wouldn’t recommend disabling resolvconf but you can do it by making /etc/resolv.conf a regular file instead of a symlink.
    Though please note that you may then be getting inconsistent /etc/resolv.conf when multiple software are fighting to change it.

This change affects all Ubuntu installs except for Ubuntu core.

Using dnsmasq as local resolver by default on desktop installations

That’s the second big change of this release. On a desktop install, your DNS server is going to be “127.0.0.1” which points to a NetworkManager-managed dnsmasq server.

This was done to better support split DNS for VPN users and to better handle DNS failures and fallbacks. This dnsmasq server isn’t a caching server for security reason to avoid risks related to local cache poisoning and users eavesdropping on other’s DNS queries on a multi-user system.

The big advantage is that if you connect to a VPN, instead of having all your DNS traffic be routed through the VPN like in the past, you’ll instead only send DNS queries related to the subnet and domains announced by that VPN. This is especially interesting for high latency VPN links where everything would be slowed down in the past.

As for dealing with DNS failures, dnsmasq often sends the DNS queries to more than one DNS servers (if you received multiple when establishing your connection) and will detect bogus/dead ones and simply ignore them until they start returning sensible information again. This is to compare against the libc’s way of doing DNS resolving where the state of the DNS servers can’t be saved (as it’s just a library) and so every single application has to go through the same, trying the first DNS, waiting for it to timeout, using the next one.

Now for the most common questions:

  • How to know what DNS servers I’m using (since I can’t just “cat /etc/resolv.conf”)?
    “nm-tool” can be used to get information about your existing connections in Network Manager. It’s roughly the same data you’d get in the GUI “connection information”.
    Alternatively, you can also read dnsmasq’s configuration from /run/nm-dns-dnsmasq.conf
  • I really don’t want a local resolver, how can I turn it off?
    To turn off dnsmasq in Network Manager, you need to edit /etc/NetworkManager/NetworkManager.conf and comment the “dns=dnsmasq” line (put a # in front of it) then do a “sudo restart network-manager”.

Bugs and feedback

Although we’ve been doing these changes more than a month ago and we’ve been looking pretty closely at bug reports, there may be some we haven’t found yet.

Issues related to resolvconf should be reported with:
ubuntu-bug resolvconf

Issues related to the dnsmasq configuration should be reported with:
ubuntu-bug network-manager

And finally, actual dnsmasq bugs and crashed should be reported with:
ubuntu-bug dnsmasq

In all cases, please try to include the following information:

  • How was your system installed (desktop, alternate, netinstall, …)?
  • Whether it’s a clean install or an upgrade?
  • Tarball of /etc/resolvconf and /run/resolvconf
  • Content of /run/nm-dns-dnsmasq.conf
  • Your /var/log/syslog
  • Your /etc/network/interfaces
  • And obviously a detailed description of your problem
Posted in Canonical voices, Planet Ubuntu | 243 Comments

Upcoming pastebinit 1.3 release

Every year, I try to set a few hours aside to work on one of my upstream projects, pastebinit.

This is one of these projects which mostly “just works” with quite a lot of users and quite a few of them sending merge proposals and fixes in bug reports.

I’m planning on uploading pastebinit 1.3 right before Feature Freeze, either on Wednesday or early Thursday, delaying the release as much as possible to get a few last translations in.

If you speak any language other than English, please go to:
https://translations.launchpad.net/pastebinit

Any help getting this as well translated as possible would be appreciated, for Ubuntu users, you’ll have to deal with it for the next 5 years, so it’s kind of important 🙂

Now the changes, they’re pretty minimal but still will make some people happy I’m sure:

  • Finally merged pbget/pbput/pbputs from Dustin Kirkland, these 3 tools let you securely push and retrieve files using a pastebin. It’s using a mix of base64, tar and gpg as well as some wget and parsing to retrieve the data.
    These are nice scripts to use with pastebinit, though please don’t send huge files to the pastebins, they really aren’t meant for that 😉
  • Removed stikked.com from the supported pastebins as it’s apparently dead.
  • Now the new pastebins:
  • paste.debian.net should now work fine with the ‘-f’ (format) option, thanks for their work on making their form pastebinit-friendly.
  • pastebinit should now load pastebin definition files properly from multiple locations.
    Starting with /usr/share/pastebin.d, then going through /etc/pastebin.d, /usr/local/etc/pastebin.d, ~/.pastebin.d and finally <wherever pastebinit is>/.pastebin.d
  • A few other minor improvements and fixes merged by Rolf Leggewie over the last year or so, thanks again for taking care of these!

Testing of the current trunk before release would also be greatly appreciate, you can get the code with: bzr branch lp:pastebinit

Bug reports are welcome at: https://launchpad.net/pastebinit/+filebug

Posted in Canonical voices, pastebinit, Planet Ubuntu | 4 Comments

Ever wanted an armhf container on your x86 machine? It’s now possible with LXC in Ubuntu Precise

It took a while to get some apt resolver bugs fixed, a few packages marked for multi-arch and some changes in the Ubuntu LXC template, but since yesterday, you can now run (using up to date Precise):

  • sudo apt-get install lxc qemu-user-static
  • sudo lxc-create -n armhf01 -t ubuntu — -a armhf -r precise
  • sudo lxc-start -n armhf01
  • Then login with root as both login and password

And enjoy an armhf system running on your good old x86 machine.

Now, obviously it’s pretty far from what you’d get on real ARM hardware.
It’s using qemu’s user space CPU emulation (qemu-user-static), so won’t be particularly fast, will likely use a lot of CPU and may give results pretty different from what you’d expect on real hardware.

Also, because of limitations in qemu-user-static, a few packages from the “host” architecture are installed in the container. These are mostly anything that requires the use of ptrace (upstart) or the use of netlink (mountall, iproute and isc-dhcp-client).
This is the bare minimum I needed to install to get the rest of the container to work using armhf binaries. I obviously didn’t test everything and I’m sure quite a few other packages will fail in such environment.

This feature should be used as an improvement on top of a regular armhf chroot using qemu-user-static and not as a replacement for actual ARM hardware (obviously), but it’s cool to have around and nice to show what LXC can do.

I confirmed it to work for armhf and armel, powerpc should also work, though it didn’t succeed to debootstrap when I tried it earlier today.

Enjoy!

Posted in Canonical voices, LXC, Planet Ubuntu | Tagged | 12 Comments