Networking in Ubuntu 12.04 LTS

Posted on 2012/01/04 by Stéphane Graber

One of my focus for this cycle is to get Ubuntu’s support for complex networking working in a predictable way. The idea was to review exactly what’s happening at boot time, get a list of possible scenario that are used on servers in corporate environment and make sure these always work.

Bonding

Bonding basically means aggregating multiple physical link into one virtual link for high availability and load balancing. There are different ways of setting up such a link though the industry standard is 802.3ad (LACP – Link Aggregation Control Protocol). In that mode your server will negotiate with your switch to establish an aggregate link, then send monitoring packets to detect failure. LACP also does load balancing (MAC, IP and protocol based depending on hardware support).

One problem we had since at least Ubuntu 10.04 LTS is that Ubuntu’s boot sequence is event based, including bringing up network interfaces. The good old “ifup -a” is only done at the end of the boot sequence to try and fix anything that wasn’t brought up through events.

Unfortunately that meant that if your server takes a long time to detect the hardware, your bond would be initialised before the network cards have been detected, giving you a bond0 without a MAC address, making DHCP queries fail in pretty weird ways and making bridging or tagging fail with “Operation not permitted”.
As that all depends on hardware detection timing, it was racy, giving you random results at boot time.

Thankfully that should now be all fixed in 12.04, the new ifenslave-2.6 I uploaded a few weeks ago now initialises the bond whenever the first slave appears. If no slave appeared by the time we get to the catch-all “ifup -a”, it’ll simply wait for up to an additional minute for a slave to appear before giving up and continuing the boot sequence.
To avoid another common race condition where a bridge is brought up with a bond as one of its members before the bond is ready, ifenslave will now detect a bond is part of a bridge and add it only once ready.

Tagging

Another pretty common thing on corporate networks is the use of VLANs (802.1q), letting you create up to 4096 virtual networks on one link.
In the past, Ubuntu would rely on the catch all “ifup -a” to create any required vlan interface, once again, that’s a problem when an interface that depends on that vlan interface is initialised before the vlan interface is created.

To fix that, Ubuntu 12.04’s vlan package now ships with a udev rule that triggers the creation of the vlan interface whenever its parent interface is created.

Bridging

Bridging on Linux can be seen as creating a virtual switch on your system (including STP support).

Bridges have been working pretty well for a while on Ubuntu as we’ve been shipping a udev rule similar to the one for vlans for a few releases already. Members are simply added to the bridge as they appear on the system. The changes to ifenslave and the vlan package make sure that even bond interfaces with VLANs get properly added to bridges.

Complex network configuration example

My current test setup for networking on Ubuntu 12.04 is actually something I’ve been using on my network for years.

As you may know, I’m also working on LXC (Linux Containers), so my servers usually run somewhere between 15 and 80 containers, each of these container has a virtual ethernet interface that’s bridged.
I have one bridge per network zone, each of these network zone being a separate VLAN. These VLANs are created on top of a two gigabit link bond.

At boot time, the following happens (roughly):

  1. One of the two network interfaces appear
  2. The bond is initialised and the first interface is enslaved
  3. This triggers the creation of all the VLAN interfaces
  4. Creating the VLAN interfaces triggers the creation of all the bridges
  5. All the VLAN interfaces are added to their respective bridge
  6. The other network interface appear and gets added to the bond

My /etc/network/interfaces can be found here:
http://www.stgraber.org/download/complex-interfaces

This contains the very strict minimum needed for LACP to work. One thing worth noting is that the two physical interfaces are listed before bond0, this is to ensure that even if the events don’t work and we have to rely on the fallback “ifup -a”, the interfaces will be initialised in the right order avoiding the 60s delay.

Please note that this example will only reliably work with Ubuntu Precise (to become 12.04 LTS). It’s still a correct configuration for previous releases but race conditions may give you a random result.

I’ll be trying to push these changes to Ubuntu 11.10 as they are pretty easy to backport there, however it’d be much harder and very likely dangerous to backport these to even older releases.
For these, the only recommendation I can give is to add some “pre-up sleep 5” or similar to your bridges and vlan interfaces to make sure whatever interface they depend on exists and is ready by the time the “ifup -a” call is reached.

IPv6

Another interesting topic for 12.04 is IPv6, as a release that’ll be supported for 5 years on both servers and desktops, it’s important to get IPv6 right.

Ubuntu 12.04 LTS will be the first Ubuntu release shipping with IPv6 private extensions turned on by default. Ubuntu 11.10 already brought most of what’s needed for IPv6 support on the desktop and in the installer, supporting SLAAC (stateless autoconfiguration), stateless DHCPv6 and stateful DHCPv6.

Once we get a new ifupdown in Ubuntu Precise, we’ll have full support for IPv6 also for people that aren’t using Network Manager (mostly servers) which should at this point give us support for any IPv6 setup you may find.

The userspace has been working pretty well with IPv6 for years. I recently made my whole network dual-stack and now have all my servers and services defaulting to IPv6 for a total of 40% of my network traffic (roughly 1.5TB a month of IPv6 traffic). The only user space related problem I noticed is the lack of IPv6 support in Nagios’ nrpe plugin, meaning I can’t start converting servers to single stack IPv6 as I’d loose monitoring …

I also wrote a python script using pypcap to give me the percentage of ipv6 and ipv4 traffic going through a given interface, the script can be found here: http://www.stgraber.org/download/v6stats.py (start with: python v6stats.py eth0)

What now ?

At this point, I think Ubuntu Precise is pretty much ready as far as networking is concerned. The only remaining change is the new ifupdown and the small installer change that comes with it for DHCPv6 support.

If you have a spare machine or VM, now is the time to grab a recent build of Ubuntu Precise and make sure whatever network configuration you use in production works reliably for you and that any hack you needed in the past can now be dropped.
If your configuration doesn’t work at all or fails randomly, please file a bug so we can have a look at your configuration and fix any bug (or documentation).

Posted in Canonical voices, IPv6, LXC, Planet Ubuntu | 45 Comments

Using Arkose for development and packaging

Posted on 2011/09/25 by Stéphane Graber

Since I last reinstalled my laptop, I try to keep my usually insanely long list of installed packages to a bare minimum. I’d usually have hundreds if not thousands of libraries and development packages as these are required by a bunch of packages I maintain or code I work on.

To achieve this and still be as productive as before (if not more), I’m using arkose quite a lot to generate temporary dev/build environment that are wiped as soon as I close the shell.
This helps maintain the number of extra libraries to a minimum, avoiding situations where something mysteriously works fine on my laptop but not on another machine and avoids the maintenance needed when dealing with chroots.

Arkose used to install libdbus-1-dev

An example of this is when I’m working on ubiquity (the Ubuntu graphical installer).
Ubiquity depends on quite a few libraries and development packages that are required even if you just want to build its source package.

So having arkose installed on my system, I usually start working on a bug with:

sudo arkose -n -h -c "cd $PWD; $SHELL"

You can make that an alias if you use it quite often. At this point, you’ll see your shell showing a different hostname, like “arkose-tmpaF9yqa”, that’s how you know you’re in a container.
The command above creates a new container using copy-on-write for all the file system but your home directory and lets the container access the network without any restriction.

I then install all the packages I’ll need to work

sudo apt-get build-dep ubiquity

Then work as usual in that container, run debuild, dput, … everything should work as usual as it has direct access to my home directory.

Once I’m done and I don’t need all these packages anymore, I just exit that shell and all the changes done outside of /home will be lost.

Posted in Arkose, Canonical voices, LXC, Planet Ubuntu | Tagged | 17 Comments

Install multiple version of the same package and avoid filesystem conflicts

Posted on 2011/09/08 by Stéphane Graber

One common problem with installing out-of-distro packages is that you can’t really be sure they won’t mess with your existing system. Most common problems include having them replace existing files or add plugins that you don’t really want or that will end up crashing other applications.

There’s also the case where you may want to get the all new and shiny version of a software but still keep the old version around as some other users may prefer it or other software may depend on it. The current way of doing that is by spending quite a bit of time packaging the version to avoid any filesystem conflicts, this works but is usually quite painful to do.

So a few weeks ago I quickly hacked together a debhelper script called dh_ubuntuarb that runs at the end of the package build process and moves all the content to /opt/extras.ubuntu.com/<package name> except for .desktop entries that are renamed to be arb-<package name>-<original name> and altered to run using “arb-wrapper”.

arb-wrapper uses an overlay fuse filesystem to create a view where the content of the package is stacked on top your root filesystem, it then uses fakechroot to make the app believe it’s running in /.

The software and the user shouldn’t notice any difference and doing that solves the whole file system conflicts and allows you to run as many version of the package as you want (it just needs to have a different name). This is particularly handy for automated packaging where you don’t want to spend too much time reviewing the packages and for cases where you want to run multiple version of the same package.

This code is very much proof of concept and might be extended and uploaded to the archive for Ubuntu 12.04. For now it can be tested in a PPA: https://launchpad.net/~stgraber/+archive/arb-testing

Below is an example using “lightyears” from the arb-testing PPA:
# Build lightyear with dh_ubuntuarb and installed it on my system
# then created an empty /tmp/test directory
stgraber@castiana:~$ ls /tmp/test
stgraber@castiana:~$ which lightyears

# Now starting the overlay of my / and /opt/extras.ubuntu.com/lightyears/
stgraber@castiana:~$ arb-wrapper lightyears bash

# Checking that lightyears is indeed accessible now
stgraber@castiana:/$ which lightyears
/usr/games/lightyears

# Creating some data
stgraber@castiana:/$ touch /tmp/test/me

# Exiting the environment
stgraber@castiana:/$ exit

# And the file is still there
stgraber@castiana:~$ ls /tmp/test/me
/tmp/test/me

Installing the “lightyears” package from that PPA will add a desktop entry called arb-lightyears_lightyears.desktop in /usr/share/applications and that’s the only change that will happen outside of /opt/extras.ubuntu.com/lightyears. This software will appear in your menu as usual and when you click on it will use the arb-wrapper to start just as you’d expect it to.

I’m very much interested in hearing ideas of possible use cases for that kind of things in Ubuntu and hope to be able to discuss it more in Orlando at the next UDS.
I think we should be able to get some pretty exciting things going using that kind of tools and containers like arkose provides.

UPDATE: The “arb” in the script names stands for the Ubuntu App Review Board which was the initial reason for doing that work as an easy way of getting a package to match our packaging requirements.

Posted in Canonical voices, Planet Ubuntu | 3 Comments

Help translate Edubuntu 11.10

Posted on 2011/09/06 by Stéphane Graber

Now that the release of (Edu|Ku|Lu|Myth|Xu|U)buntu is just a month or so away and all the UIs aren’t supposed to change till release, it’s time to help making sure everything is translated.

One of my goals for Edubuntu 11.10 was to have all our custom components (installer steps, LTSP, Arkose, …) supporting translations as early as possible so people can just boot our DVD and get a fully translated environment right out of the box (yes, we ship all the langpacks).

While that goal has been relatively easily reached, Edubuntu still needs some help getting everything translated. Here are a few links to some parts of Edubuntu that need some love:

All of the above weren’t translated at all in Edubuntu 11.04 so they definitely need help to get as good as the rest of the distro.

More information on helping translating Ubuntu (not only Edubuntu) can be found on the wiki: https://wiki.ubuntu.com/Translations

Thanks!

Posted in Canonical voices, Edubuntu, Planet Ubuntu | 1 Comment

Arkose 1.3.1 released

Posted on 2011/08/18 by Stéphane Graber

Last week I was in Austin, TX where a bunch of people with interest in getting containers working on Linux were meeting for the Oneiric Container Sprint.
We all had a very productive week with a lot of work being done on LXC, the kernel namespaces and Arkose.

Right before the Ubuntu Feature Freeze last Thursday, I released Arkose 1.3 brining most of the features I wanted for Ubuntu 11.10.

Here’s a brief list of the new stuff Arkose can do:

  • All the UIs and CLIs now support translation with an initial (rough) french translation already available.
  • DBUS filtering is now included in Arkose and available through the wrapper. The gedit example profile is using it.
  • It’s now possible to temporarily modify a wrapper profile before starting it.
  • Device support has been changed to no longer be limited to /dev/video* devices.

Some bugs have also been fixed, most of them in Arkose 1.3.1 (released yesterday):

  • Make the Global Menu integration (dbusmenu) work with Ubuntu Oneiric
  • Update the test suite
  • Fix arkose-cli’s help to be a lot more accurate
  • Restrict LXC’s configuration to the bare minimal
  • Use point-to-point network configuration in filtered mode (rather than a /30 per container)
  • Make sure everything in the container gets properly killed on exit
  • Fix Arkose to handle command line parameters properly (instead of just ignoring them)

That’s all available in current Ubuntu Oneiric as well as in the arkose stable PPA for Ubuntu 10.10 and Ubuntu 11.04.

Sadly one feature didn’t make it in time for Feature Freeze, that’s the advanced firewalling in filtered network mode. I’ll probably be working on it on the side and push it to a 1.4 branch that’ll be used for Oneiric+1.

I’ll now mostly be focusing on bugfixes for the remaining of the cycle and polishing some of the existing features. So please, test it and file bugs!

If you want to help with the translation effort, you can go translate Arkose on Launchpad or just send me a .po and I’ll do it for you.

For these who want to run the current upstream code, get the bzr branch:
bzr branch lp:arkose

Posted in Arkose, Canonical voices, Conferences, LXC, Planet Ubuntu | Leave a comment