Category Archives: Canonical voices

Dual-stack 3G connections

A few months ago, I received two test SIM cards for Orange Poland’s new IPv6 network.

The interesting thing about this network is that it’s running IPv6 in a fairly unusual configuration and it was interesting to see how to get that work on Ubuntu.

This network uses two separate APNs, one for IPv4 (internet) and one for IPv6 (internetipv6).
Using two separate APNs is certainly easier on the carrier’s infrastructure side as they can get IPv6 online without actually changing anything on the IPv4 equipement, however that means that any client wanting to use both protocols at once needs to use multiple PDP contexts.

I’m now going to detail how to manually configure ppp to connect to such a network:
/etc/ppp/peers/orange

noauth
connect "/usr/sbin/chat -e -f /etc/ppp/peers/orange-connect"
/dev/ttyACM0
460800
+ipv6

/etc/ppp/peers/orange-connect

TIMEOUT 5
ABORT BUSY
ABORT 'NO CARRIER'
ABORT VOICE
ABORT 'NO DIALTONE'
ABORT 'NO ANSWER'
ABORT DELAYED
ABORT ERROR
'' \nAT
TIMEOUT 12
OK ATH
OK ATE1
OK 'AT+CGDCONT=1,"IP","internet"'
OK 'AT+CGDCONT=2,"IPV6","internetipv6"'
OK ATD*99#
CONNECT ""

Then all that’s needed is a good old:

pon orange

And a few seconds later, I’m getting the following on ppp0:

ppp0      Link encap:Point-to-Point Protocol  
          inet addr:87.96.119.169  P-t-P:10.6.6.6  Mask:255.255.255.255
          inet6 addr: 2a00:f40:2100:ac9:8c1e:da60:93e2:c234/64 Scope:Global
          inet6 addr: 2a00:f40:2100:ac9::1/64 Scope:Global
          inet6 addr: fe80::1/10 Scope:Link
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:13 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3 
          RX bytes:354 (354.0 B)  TX bytes:767 (767.0 B

This config should work for any mobile network using a similar setup (likely to become more and more popular as the various RIRs are running out of IPv4).

Sadly ModemManager/NetworkManager don’t support mutliple PDP contexts yet, though it’s being discussed upstream, so we can hope to see something land soon.

Apparently multiple PDP contexts support is also dependant on hardware. In my case, I’ve been using an “old” Nokia E51 over USB as I didn’t have any luck getting that to work with an Android phone. My Nokia N900 also worked but required a custom kernel to be installed first to properly handle IPv6.

Posted in Canonical voices, IPv6, Planet Ubuntu | 4 Comments

Easily ssh to your containers and VMs on Ubuntu 12.04

With the DNS changes in Ubuntu 12.04, most development machines running with libvirt and lxc end up running quite a few DNS servers.

These DNS servers work fine when queried from a system on their network, but aren’t integrated with the main dnsmasq instance and so won’t let you resolve your VM and containers from outside of their respective networks.

One way to solve that is to install yet another DNS resolver and use it to redirect between the various dnsmasq instances. That can quickly become tricky to setup and doesn’t integrate too well with resolvconf and NetworkManager.

Seeing a lot of people wondering how to solve that problem, I took a few minutes yesterday to come up with an ssh configuration that’d allow one to access their containers and VM using their name.

The result is the following, to add to your ~/.ssh/config file:

Host *.lxc
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  ProxyCommand nc $(host $(echo %h | sed "s/\.lxc//g") 10.0.3.1 | tail -1 | awk '{print $NF}') %p

Host *.libvirt
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  ProxyCommand nc $(host $(echo %h | sed "s/\.libvirt//g") 192.168.122.1 | tail -1 | awk '{print $NF}') %p

After that, things like:

  • ssh user@myvm.libvirtu
  • ssh ubuntu@mycontainer.lxc

Will just work.

For LXC, you may also want to add a “User ubuntu” line to that config as it’s the default user for LXC containers on Ubuntu.
If you configured your bridges with a non-default subnet, you’ll also need to update the IPs or add more sections to the config.

These also turn off StrictHostKeyChecking and UserKnownHostsFile as my VMs and containers are local to my machine (reducing risk of MITM attacks) and tend to exist only for a few hours, to then be replaced by a completely different one with a different SSH host key. Depending on your setup, you may want to remove these lines.

Posted in Canonical voices, LXC, Planet Ubuntu | Tagged | 15 Comments

LXC in Ubuntu 12.04 LTS

Quite a few people have been asking for a status update of LXC in Ubuntu as of Ubuntu 12.04 LTS. This post is meant as an overview of the work we did over the past 6 months and pointers to more detailed blog posts for some of the new features.

What’s LXC?

LXC is a userspace tool controlling the kernel namespaces and cgroup features to create system or application containers.

To give you an idea:

  • Feels like somewhere between a chroot and a VM
  • Can run a full distro using the “host” kernel
  • Processes running in a container are visible from the outside
  • Doesn’t require any specific hardware, works on all supported architectures

A libvirt driver for LXC exists (libvirt-lxc), however it doesn’t use the “lxc” userspace tool even though it uses the same kernel features.

Making LXC easier

One of the main focus for 12.04 LTS was to make LXC dead easy to use, to achieve this, we’ve been working on a few different fronts fixing known bugs and improving LXC’s default configuration.

Creating a basic container and starting it on Ubuntu 12.04 LTS is now down to:

sudo apt-get install lxc
sudo lxc-create -t ubuntu -n my-container
sudo lxc-start -n my-container

This will default to using the same version and architecture as your machine, additional option are obviously available (–help will list them). Login/Password are ubuntu/ubuntu.

Another thing we worked on to make LXC easier to work with is reducing the number of hacks required to turn a regular system into a container down to zero.
Starting with 12.04, we don’t do any modification to a standard Ubuntu system to get it running in a container.
It’s now even possible to take a raw VM image and have it boot in a container!

The ubuntu-cloud template also lets you get one of our EC2/cloud images and have it start as a container instead of a cloud instance:

sudo apt-get install lxc cloud-utils
sudo lxc-create -t ubuntu-cloud -n my-cloud-container
sudo lxc-start -n my-cloud-container

And finally, if you want to test the new cool stuff, you can also use juju with LXC:

[ ! -f ~/.ssh/id_rsa.pub ] && ssh-keygen -t rsa
sudo apt-get install juju apt-cacher-ng zookeeper lxc libvirt-bin --no-install-recommends
sudo adduser $USER libvirtd
juju bootstrap
sed -i "s/ec2/local/" ~/.juju/environments.yaml
echo " data-dir: /tmp/juju" >> ~/.juju/environments.yaml
juju bootstrap
juju deploy mysql
juju deploy wordpress
juju add-relation wordpress mysql
juju expose wordpress

# To tail the logs
juju debug-log

# To get the IPs and status
juju status

Making LXC safer

Another main focus for LXC in Ubuntu 12.04 was to make it safe. John Johansen did an amazing work of extending apparmor to let us implement per-container apparmor profiles and prevent most known dangerous behaviours from happening in a container.

NOTE: Until we have user namespaces implemented in the kernel and used by the LXC we will NOT say that LXC is root safe, however the default apparmor profile as shipped in Ubuntu 12.04 LTS is blocking any armful action that we are aware of.

This mostly means that write access to /proc and /sys are heavily restricted, mounting filesystems is also restricted, only allowing known-safe filesystems to be mounted by default. Capabilities are also restricted in the default LXC profile to prevent a container from loading kernel modules or control apparmor.

More details on this are available here:

Other cool new stuff

Emulated architecture containers

It’s now possible to use qemu-user-static with LXC to run containers of non-native architectures, for example:

sudo apt-get install lxc qemu-user-static
sudo lxc-create -n my-armhf-container -t ubuntu -- -a armhf
sudo lxc-start -n my-armhf-container

Ephemeral containers

Quite a bit of work also went into lxc-start-ephemeral, the tool letting you start a copy of an existing container using an overlay filesystem, discarding any change you make on shutdown:

sudo apt-get install lxc
sudo lxc-create -n my-container -t ubuntu
sudo lxc-start-ephemeral -o my-container

Container nesting

You can now start a container inside a container!
For that to work, you first need to create a new apparmor profile as the default one doesn’t allow this for security reason.
I already did that for you, so the few commands below will download it and install it in /etc/apparmor.d/lxc/lxc-with-nesting. This profile (or something close to it) will ship in Ubuntu 12.10 as an example of alternate apparmor profile for container.

sudo apt-get install lxc
sudo lxc-create -t ubuntu -n my-host-container
sudo wget https://www.stgraber.org/download/lxc-with-nesting -O /etc/apparmor.d/lxc/lxc-with-nesting
sudo /etc/init.d/apparmor reload
sudo sed -i "s/#lxc.aa_profile = unconfined/lxc.aa_profile = lxc-container-with-nesting/" /var/lib/lxc/my-host-container/config
sudo lxc-start -n my-host-container
(in my-host-container) sudo apt-get install lxc
(in my-host-container) sudo stop lxc
(in my-host-container) sudo sed -i "s/10.0.3/10.0.4/g" /etc/default/lxc
(in my-host-container) sudo start lxc
(in my-host-container) sudo lxc-create -n my-sub-container -t ubuntu
(in my-host-container) sudo lxc-start -n my-sub-container

Documentation

Outside of the existing manpages and blog posts I mentioned throughout this post, Serge Hallyn did a very good job at creating a whole section dedicated to LXC in the Ubuntu Server Guide.
You can read it here: https://help.ubuntu.com/12.04/serverguide/lxc.html

Next steps

Next week we have the Ubuntu Developer Summit in Oakland, CA. There we’ll be working on the plans for LXC in Ubuntu 12.10. We currently have two sessions scheduled:

If you want to make sure the changes you want will be in Ubuntu 12.10, please make sure to join these two sessions. It’s possible to participate remotely to the Ubuntu Developer Summit, through IRC and audio streaming.

My personal hope for LXC in Ubuntu 12.10 is to have a clean liblxc library that can be used to create bindings and be used in languages like python. Working towards that goal should make it easier to do automated testing of LXC and cleanup our current tools.

I hope this post made you want to try LXC or for existing users, made you discover some of the new features that appeared in Ubuntu 12.04. We’re actively working on improving LXC both upstream and in Ubuntu, so do not hesitate to report bugs (preferably with “ubuntu-bug lxc”).

Posted in Canonical voices, Conferences, LXC, Planet Ubuntu | Tagged | 64 Comments

Booting an Ubuntu 12.04 virtual machine in an LXC container

One thing that we’ve been working on for LXC in 12.04 is getting rid of any remaining LXC specific hack in our templates. This means that you can now run a perfectly clean Ubuntu system in a container without any change.

To better illustrate that, here’s a guide on how to boot a standard Ubuntu VM in a container.

First, you’ll need an Ubuntu VM image in raw disk format. The next few steps also assume a default partitioning where the first primary partition is the root device. Make sure you have the lxc package installed and up to date and lxcbr0 enabled (the default with recent LXC).

Then run kpartx -a vm.img this will create loop devices in /dev/mapper for your VM partitions, in the following configuration I’m assuming /dev/mapper/loop0p1 is the root partition.

Now write a new LXC configuration file (myvm.conf in my case) containing:

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = lxcbr0
lxc.utsname = myvminlxc

lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = /dev/mapper/loop0p1
lxc.arch = amd64
lxc.cap.drop = sys_module mac_admin

lxc.cgroup.devices.deny = a
# Allow any mknod (but not using the node)
lxc.cgroup.devices.allow = c *:* m
lxc.cgroup.devices.allow = b *:* m
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
#lxc.cgroup.devices.allow = c 4:0 rwm
#lxc.cgroup.devices.allow = c 4:1 rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
# rtc
lxc.cgroup.devices.allow = c 254:0 rwm
#fuse
lxc.cgroup.devices.allow = c 10:229 rwm
#tun
lxc.cgroup.devices.allow = c 10:200 rwm
#full
lxc.cgroup.devices.allow = c 1:7 rwm
#hpet
lxc.cgroup.devices.allow = c 10:228 rwm
#kvm
lxc.cgroup.devices.allow = c 10:232 rwm

The bits in bold may need updating if you’re not using the same architecture, partition scheme or bridges as I’m.

Then finally, run: lxc-start -n myvminlxc -f myvm.conf

And watch your VM boot in an LXC container.

I did this test with a desktop VM using network manager so it didn’t mind LXC’s random MAC address, server VMs might get stuck for a minute at boot time because of that though.
In such case, either clean /etc/udev/rules.d/70-persistent-net.rules or set “lxc.network.hwaddr” to the same mac address as your VM.

Once done, run kpartx -d vm.img to remove the loop devices.

Posted in Canonical voices, LXC, Planet Ubuntu | Tagged | 28 Comments

DNS in Ubuntu 12.04

Anyone who’s been using 12.04 over the past month or so may have noticed some pretty significant changes in the way we do DNS resolving in Ubuntu.

This is the result of the implementation of: foundations-p-dns-resolving

Here is a description of the two big changes that happened:

Switch to resolvconf for /etc/resolv.conf management

resolvconf is a set of script and hooks managing DNS resolution. The most notable difference for the user is that any change manually done to /etc/resolv.conf will be lost as it gets overwritten next time something triggers resolvconf. Instead, resolvconf uses DHCP client hooks, a Network Manager plugin and /etc/network/interfaces to generate a list of nameservers and domain to put in /etc/resolv.conf.

For more details, I’d highly encourage you to read resolvconf’s manpage but here are a few answers to common questions:

  • I use static IP configuration, where should I put my DNS configuration?
    The DNS configuration for a static interface should go as “dns-nameservers”, “dns-search” and “dns-domain” entries added to the interface in /etc/network/interfaces
  • How can I override resolvconf’s configuration or append some entries to it?
    Resolvconf has a /etc/resolvconf/resolv.conf.d/ directory that can contain “base”, “head”, “original” and “tail” files. All in resolv.conf format.
    • base: Used when no other data can be found
    • head: Used for the header of resolv.conf, can be used to ensure a DNS server is always the first one in the list
    • original: Just a backup of your resolv.conf at the time of resolvconf installation
    • tail: Any entry in tail is appended at the end of the resulting resolv.conf. In some cases, upgrading from a previous Ubuntu release, will make tail a symlink to original (when we think you manually modified resolv.conf in the past)
  • I really don’t want resolvconf, how can I disable it?
    I certainly wouldn’t recommend disabling resolvconf but you can do it by making /etc/resolv.conf a regular file instead of a symlink.
    Though please note that you may then be getting inconsistent /etc/resolv.conf when multiple software are fighting to change it.

This change affects all Ubuntu installs except for Ubuntu core.

Using dnsmasq as local resolver by default on desktop installations

That’s the second big change of this release. On a desktop install, your DNS server is going to be “127.0.0.1” which points to a NetworkManager-managed dnsmasq server.

This was done to better support split DNS for VPN users and to better handle DNS failures and fallbacks. This dnsmasq server isn’t a caching server for security reason to avoid risks related to local cache poisoning and users eavesdropping on other’s DNS queries on a multi-user system.

The big advantage is that if you connect to a VPN, instead of having all your DNS traffic be routed through the VPN like in the past, you’ll instead only send DNS queries related to the subnet and domains announced by that VPN. This is especially interesting for high latency VPN links where everything would be slowed down in the past.

As for dealing with DNS failures, dnsmasq often sends the DNS queries to more than one DNS servers (if you received multiple when establishing your connection) and will detect bogus/dead ones and simply ignore them until they start returning sensible information again. This is to compare against the libc’s way of doing DNS resolving where the state of the DNS servers can’t be saved (as it’s just a library) and so every single application has to go through the same, trying the first DNS, waiting for it to timeout, using the next one.

Now for the most common questions:

  • How to know what DNS servers I’m using (since I can’t just “cat /etc/resolv.conf”)?
    “nm-tool” can be used to get information about your existing connections in Network Manager. It’s roughly the same data you’d get in the GUI “connection information”.
    Alternatively, you can also read dnsmasq’s configuration from /run/nm-dns-dnsmasq.conf
  • I really don’t want a local resolver, how can I turn it off?
    To turn off dnsmasq in Network Manager, you need to edit /etc/NetworkManager/NetworkManager.conf and comment the “dns=dnsmasq” line (put a # in front of it) then do a “sudo restart network-manager”.

Bugs and feedback

Although we’ve been doing these changes more than a month ago and we’ve been looking pretty closely at bug reports, there may be some we haven’t found yet.

Issues related to resolvconf should be reported with:
ubuntu-bug resolvconf

Issues related to the dnsmasq configuration should be reported with:
ubuntu-bug network-manager

And finally, actual dnsmasq bugs and crashed should be reported with:
ubuntu-bug dnsmasq

In all cases, please try to include the following information:

  • How was your system installed (desktop, alternate, netinstall, …)?
  • Whether it’s a clean install or an upgrade?
  • Tarball of /etc/resolvconf and /run/resolvconf
  • Content of /run/nm-dns-dnsmasq.conf
  • Your /var/log/syslog
  • Your /etc/network/interfaces
  • And obviously a detailed description of your problem
Posted in Canonical voices, Planet Ubuntu | 243 Comments