Quite a few people have been asking for a status update of LXC in Ubuntu as of Ubuntu 12.04 LTS. This post is meant as an overview of the work we did over the past 6 months and pointers to more detailed blog posts for some of the new features.
LXC is a userspace tool controlling the kernel namespaces and cgroup features to create system or application containers.
To give you an idea:
- Feels like somewhere between a chroot and a VM
- Can run a full distro using the “host” kernel
- Processes running in a container are visible from the outside
- Doesn’t require any specific hardware, works on all supported architectures
A libvirt driver for LXC exists (libvirt-lxc), however it doesn’t use the “lxc” userspace tool even though it uses the same kernel features.
Making LXC easier
One of the main focus for 12.04 LTS was to make LXC dead easy to use, to achieve this, we’ve been working on a few different fronts fixing known bugs and improving LXC’s default configuration.
Creating a basic container and starting it on Ubuntu 12.04 LTS is now down to:
sudo apt-get install lxc sudo lxc-create -t ubuntu -n my-container sudo lxc-start -n my-container
This will default to using the same version and architecture as your machine, additional option are obviously available (–help will list them). Login/Password are ubuntu/ubuntu.
Another thing we worked on to make LXC easier to work with is reducing the number of hacks required to turn a regular system into a container down to zero.
Starting with 12.04, we don’t do any modification to a standard Ubuntu system to get it running in a container.
It’s now even possible to take a raw VM image and have it boot in a container!
The ubuntu-cloud template also lets you get one of our EC2/cloud images and have it start as a container instead of a cloud instance:
sudo apt-get install lxc cloud-utils sudo lxc-create -t ubuntu-cloud -n my-cloud-container sudo lxc-start -n my-cloud-container
And finally, if you want to test the new cool stuff, you can also use juju with LXC:
[ ! -f ~/.ssh/id_rsa.pub ] && ssh-keygen -t rsa sudo apt-get install juju apt-cacher-ng zookeeper lxc libvirt-bin --no-install-recommends sudo adduser $USER libvirtd juju bootstrap sed -i "s/ec2/local/" ~/.juju/environments.yaml echo " data-dir: /tmp/juju" >> ~/.juju/environments.yaml juju bootstrap juju deploy mysql juju deploy wordpress juju add-relation wordpress mysql juju expose wordpress # To tail the logs juju debug-log # To get the IPs and status juju status
Making LXC safer
Another main focus for LXC in Ubuntu 12.04 was to make it safe. John Johansen did an amazing work of extending apparmor to let us implement per-container apparmor profiles and prevent most known dangerous behaviours from happening in a container.
NOTE: Until we have user namespaces implemented in the kernel and used by the LXC we will NOT say that LXC is root safe, however the default apparmor profile as shipped in Ubuntu 12.04 LTS is blocking any armful action that we are aware of.
This mostly means that write access to /proc and /sys are heavily restricted, mounting filesystems is also restricted, only allowing known-safe filesystems to be mounted by default. Capabilities are also restricted in the default LXC profile to prevent a container from loading kernel modules or control apparmor.
More details on this are available here:
Other cool new stuff
Emulated architecture containers
It’s now possible to use qemu-user-static with LXC to run containers of non-native architectures, for example:
sudo apt-get install lxc qemu-user-static sudo lxc-create -n my-armhf-container -t ubuntu -- -a armhf sudo lxc-start -n my-armhf-container
Quite a bit of work also went into lxc-start-ephemeral, the tool letting you start a copy of an existing container using an overlay filesystem, discarding any change you make on shutdown:
sudo apt-get install lxc sudo lxc-create -n my-container -t ubuntu sudo lxc-start-ephemeral -o my-container
You can now start a container inside a container!
For that to work, you first need to create a new apparmor profile as the default one doesn’t allow this for security reason.
I already did that for you, so the few commands below will download it and install it in /etc/apparmor.d/lxc/lxc-with-nesting. This profile (or something close to it) will ship in Ubuntu 12.10 as an example of alternate apparmor profile for container.
sudo apt-get install lxc sudo lxc-create -t ubuntu -n my-host-container sudo wget https://www.stgraber.org/download/lxc-with-nesting -O /etc/apparmor.d/lxc/lxc-with-nesting sudo /etc/init.d/apparmor reload sudo sed -i "s/#lxc.aa_profile = unconfined/lxc.aa_profile = lxc-container-with-nesting/" /var/lib/lxc/my-host-container/config sudo lxc-start -n my-host-container (in my-host-container) sudo apt-get install lxc (in my-host-container) sudo stop lxc (in my-host-container) sudo sed -i "s/10.0.3/10.0.4/g" /etc/default/lxc (in my-host-container) sudo start lxc (in my-host-container) sudo lxc-create -n my-sub-container -t ubuntu (in my-host-container) sudo lxc-start -n my-sub-container
Outside of the existing manpages and blog posts I mentioned throughout this post, Serge Hallyn did a very good job at creating a whole section dedicated to LXC in the Ubuntu Server Guide.
You can read it here: https://help.ubuntu.com/12.04/serverguide/lxc.html
Next week we have the Ubuntu Developer Summit in Oakland, CA. There we’ll be working on the plans for LXC in Ubuntu 12.10. We currently have two sessions scheduled:
If you want to make sure the changes you want will be in Ubuntu 12.10, please make sure to join these two sessions. It’s possible to participate remotely to the Ubuntu Developer Summit, through IRC and audio streaming.
My personal hope for LXC in Ubuntu 12.10 is to have a clean liblxc library that can be used to create bindings and be used in languages like python. Working towards that goal should make it easier to do automated testing of LXC and cleanup our current tools.
I hope this post made you want to try LXC or for existing users, made you discover some of the new features that appeared in Ubuntu 12.04. We’re actively working on improving LXC both upstream and in Ubuntu, so do not hesitate to report bugs (preferably with “ubuntu-bug lxc”).
Liked the article and looking forward to future developments of LXC and support in Ubuntu.
I have been trying something for a while, the idea is to have containers on my laptop to which I can just switch graphically into by a combination of CTRL-ALT + F4 as an example. (The idea come from watching a demo of Citrix XenClient)
While running containers wasn’t really an issue on the command-line, I did struggle a lot to get the graphical switching working.
Any ideas or suggestions on the matter?
If you just want the container to bind to tty4 as a text console, you’ll need to allow the container access to /dev/tty4 in the container config, modify the apparmor profile to allow the container to mount devpts from the host, then mount devpts and make the /dev/tty4 node in the thin client reference the one outside of it. You’ll also need to stop the existing getty on tty4 to avoid both the container’s and the host’s fighting over it.
I never tried it so I can’t give more detailed instructions on how to do that.
If what you’re after instead is getting a graphical session showing software running in the container, the easiest would be to spawn another X server on a free vt from outside the container, then bind mount /tmp/.X11-unix inside the container (“/tmp/.X11-unix/ tmp/.X11-unix none bind” in /var/lib/lxc/
/fstab) and run “export DISPLAY=:1” (assuming you start it with “X :1”).
You can then start a window manager and any other software from the container that’ll show up on that X server.
thanks! I’ll try that
ow by the way, it’s
“sudo lxc-start -n my-container” instead of -t
Good catch, fixed.
Did it work?
I succesfully created a LXC container and an additional empty X session on Ctrl+Alt+F8, but the part on binding the two together isn’t that clear at all.
Maybe the tutorial could be expanded to cover that? It would be a really fantastic option imo 🙂
Excellent and very informative LXC articles!
I get very excited about the possibilities LXC offers, as a long time user (and fan) of Solaris zones I’d love to see LXC grow and establish itself in a similar way.
Aside from the kernel namespaces (which we know is coming) I still think LXC (compared to Solaris zones) is overly complex to administer, but your articles go a long way to simplifying the process.
In the line:
sudo lxc-create -n ubuntu-cloud my-cloud-container
shouldn’t we read:
sudo lxc-create -t ubuntu-cloud -n my-cloud-container
You’re correct, fixed.
Thanks for all of your LXC work. I am attending UDS but I was unfortunately not able to attend the LXC demo presentation.
Would it be possible for you to record a demo video of roughly what was covered in the demo presentation and post it on a place like YouTube?
“You need to ‘sudo apt-get install cloud-utils’ to provide the
Oh right, that was fixed in quantal but not in precise. Updating the post. Thanks
This look promising. We want to switch for use container approach (instead of Xen) for our web hosting clients, we would want to know if improvement on resources control, escpecially with disk IO, CPU and network.
Resource control in LXC is done through the cgroup subsystem.
This allows fine grained resource control of CPU time and scheduling, as well as restriction on memory usage and swap usage.
For I/O, the blkio cgroup lets you restrict the per-device throughput as well as put a restriction on the maximum I/Ops (Input/Ouput per second).
There isn’t a networking cgroup, though as the trafic is bridged in a standard Linux bridge, any of the usual firewalling and trafic shapping tools will work.
When we type this command : sudo lxc-create -t ubuntu -n my-container
Is it really downloading from Internet?
I have slow Internet connection. And I just wondering if I could skip downloading from Internet and then using local file (e.g. Ubuntu ISO file) for installation. Is it possible?
Yes, it’s downloading the individual packages from the internet, it’s probably not impossible to generate a container from a media but it’s not supported at this point.
Though, LXC does caching, so only the initial lxc-create will download the packages from the internet, any further call will simply copy the cached container.
This may be a silly question, but if I start firefox from arkose-wrapper-gui, with COW access only, are other processes started by firefox also restricted to the LXC container? I want to use LXC for accessing things like facebook & game websites without having to worry with whether there is a weakness in firefox or Flashplayer or that allows malware to be installed on my computer.
Also, how should I set the arkose-wrapper-gui firefox profile for optimal security?
Any child process will also be in the container.
The most secure profile for firefox would be something like (not tested):
Though in real day use, you’ll probably at least want pulseaudio=true and maybe dbus=session.
Thanks for the reply!
I tried running it with the profile you suggested, but firefox doesn’t seem to get network access unless I set the network entry for “true” rather than “filtered.” What should I do to fix this?
Is there a known reason allowing access to pulseaudio would be dangerous, or is it just better to minimize access to potentially vulnerable programs when they aren’t needed? I had hoped to use arkose for websites which use flash for streaming music, so I’ll probably allow audio access if there isn’t a known risk.
Thanks again for this nifty software.
I don’t have any good explanation for why filtered networking didn’t work for you, though in firefox’s case, direct shouldn’t make a real difference. (filtered will mostly be useful once I implement the firewalling feature in arkose).
For pulseaudio, enabling it creates a separate pulseaudio socket on your pulseaudio daemon and passes it to the container, so it’s safe to use. Worst thing that can happen is your container can control your user’s pulseaudio and record what other applications are playing.
Groovy! Thanks again!
Arkose appears broken in ubuntu 13.04.
A bug was filed
It always returns requesting “terminal type”.
I need to write a program to monitor activities of Arkose container. Since this uses LXC-container, is it possible to write a program to externally (external to the container) monitor system calls executes by Arkose and applications running inside Arkose?
Your advice on this regard is highly appreciated.
Hi my friend! I want to say that this article is amazing, great written and include approximately all significant infos. I would like to look more posts like this .
I’m struggling to emulate a setup that was working perfectly before a crash.
I had the bridge device bound somehow to one of my interfaces, and the containers could get DHCP/were on the same network as the container host.
Now, I’m doing all sorts of manual bridge configuration and hoop jumping that I didn’t have to do before…
Let’s say from installing lxc, what should I have to do to have the containers on the same network as the host?
script lxc-ubuntu using arch command which is missing in newer coreutils versions – probably can be replaced by uname -m ?
Thanks for this article! This is awesome. I just started playing with LXC, and this has been really useful to understand how things work with Ubuntu 12.04. Reading articles from other distributions and older version was getting me confused.
If you have time, I’d love to see a follow up post showing how to take (say a the standard lxc ubuntu template) and limit aspects like cpu and memory – it doesn’t (yet) seem 100% clear to me.
Otherwise, thanks for your hard work! This is great!
Thanks Stephane, great article!
I was wondering are you aware of any java binding to manage LXC ?
Nope. I’m currently working with Serge on the C liblxc API and the matching python binding.
My guess is that once we publish this work upstream, quite a few other bindings will appear based on a similar API.
A new JAVA API library for LXC has been released.
Take a look at github:
Just tried it, (2/Sept/2012} and needed the following to make it work
sudo cp -rf my-container/ my-cloud-container/
Hi Stéphane, thanks for the post.
I went ahead and deployed an lxc container in a VirtualBox VM, using your description and a fresh ubuntu 12.04
Everything works as a charm except that the network starts to fails around 10 seconds after I have started the container. For instance if I do a /etc/init.d/networking restart, I can enjoy the access for 10 seconds (ping, apt-get, wget etc..) then it’s blocked.
The host works fine, this is happening only in the container. I have no clue what’s going on, and see no trace of weird stuff going on in the log
The only extra things I had to do :
– install lxc within the container otherwise lxc-execute fails
– add a +r bit to /sys/fs/cgroup otherwise cgroup-lite was failing to start (mount error)
Do you have any hint to find out what’s going on ?
Thx in any case
Thanks for lxc – great stuff!
But I’m confused about your “secure profile for firefox”. The specs in your example, like “network=filtered” don’t match the sorts of configuration options I see in the lxc.conf man page (lxc.network.type = veth). What kind of profile are you talking about?
In particular, how can I set up a sandbox, e.g. for safely executing pure python code, that doesn’t let the code connect out the the Internet and e.g. leak information?
I’ve been grappling with the following challenge, hoping someone can give me the pointers needed:
I want to bring up OVS inside an LXC container, kinda backwards from what’s usually done, without(!!) BRCOMPAT=yes and without having to compile a kernel module.
It seems doable, but I’m confused. Advice warmly accepted.
Thanks so much,
Hello, I always encountered some dpkg and locale problems when I start new containers. Both debian and ubuntu-cloud templates have those problems, and the bug reporting from launchpad said those issues had been fixed (#969299).
I’ve fixed the second locale issue by copying the host system’ locale archive file into container, but the dpkg issue is an unmovable obstacle to install packages in containers. I wonder how can you run your containers without similar issues ? Thnaks.
I have just been introduced to LXC and really quite impressed there are a few templates available but the one I really want isn’t. I have trawled the net hi and low for it but no one has managed it yet or if they have they are keeping it to themselves. The template I am most looking forward to using with LXC is for the Centos Smeserver versions 7 and 8. I have tried cloning an existing directory with lxc-clone command but not getting as far as I had hoped. I am desperate to virtualise smeserver as it would make my life so much simpler. I look forward to your comments and suggestions.
I would be happy to post a procedure on our website http://soslug.org so others might benefit from using LXC containers.
Um, question. I’m reading a lot of webpages on configuring containers but none of them are really doing what I want to do.
so I have my ubuntu 12.04 server setup and I can launch a simple container (yeah)
What I now want to do is launch my simple container with a public ip address on it. There seems to be 2 options, one is bridging, and the other is NAT’ing with some iptables juju
I’m quite happy just to assign a static IP address. But I’m not too confidant how to do that. I assume I just tinker with the network config I have in /var/lib/lxc/CN/
but lets Assume I maybe want to have a pool of accessible IP addresses, so every time I fire up a container it automatically gets an public IP
How would you do that??
A quick info for those testing the LXC-nested-into-LXC mode in Ubuntu 12.10:
You should use the provided /etc/apparmor.d/lxc/lxc-default-with-nesting profile as included at least in 12.10; the provided profile in this post no longer works (Apparmor denying a mount for the nested LXC).
@stgraber: Hi 🙂 Pretty awesome stuff these LXC containers!
Thanks for this great post!
I have a problem with juju on ubuntu 12.04. After the second boostrap, I have this error:
ubuntu@my-cloud-container:~$ juju bootstrap
error: Environments configuration error: /home/ubuntu/.juju/environments.yaml: while parsing a block mapping
in "", line 1, column 1:
expected , but found ''
in "", line 8, column 2:
You need 2 more space before data-dir to make it works.
After that change, when I try to run it again I have this:
ubuntu@my-cloud-container:~$ juju bootstrap
2013-03-13 07:24:21,368 INFO Bootstrapping environment 'sample' (origin: distro type: local)...
2013-03-13 07:24:21,369 INFO Checking for required packages...
2013-03-13 07:24:24,782 INFO Starting networking...
error: Failed to define network from /tmp/tmpFFHUGL
error: cannot create config directory '/var/lib/libvirt/dnsmasq': Permission denied
Command '['virsh', 'net-define', '/tmp/tmpFFHUGL']' returned non-zero exit status 1
2013-03-13 07:24:25,279 ERROR Command '['virsh', 'net-define', '/tmp/tmpFFHUGL']' returned non-zero exit status 1
And with sudo this:
ubuntu@my-cloud-container:~$ sudo juju bootstrap
2013-03-13 07:26:00,737 INFO Bootstrapping environment 'sample' (origin: distro type: local)...
2013-03-13 07:26:00,737 INFO Checking for required packages...
2013-03-13 07:26:05,894 INFO Starting networking...
error: Failed to start network default
error: Unable to set bridge virbr0 forward_delay: Permission denied
Command '['virsh', 'net-start', 'default']' returned non-zero exit status 1
2013-03-13 07:26:05,971 ERROR Command '['virsh', 'net-start', 'default']' returned non-zero exit status 1
It looks like an Apparmor problem, but I don’t know how to fix it!
Ok, I found that there is no bridge name virbr0, but there is one name lxcbr0. I check the juju version (0.5+bzr531-0ubuntu1.3) and maybe this version is not updated with the right bridge name…
Two related questions:
1. Is there a way to increase/change a default size of FS used by a container?
Including the directory location to use for container FS (not /var/lib/lxc/)
2. What is a way to share directory (rw) between container and host?
Presumably you can use that to share directory between containers also.
What is write semantic? Posix? file close? something else?
I’m going to try this with 12.10. Will this work the same way on that version as you have described above or will there be adjustments I need to make to ensure it goes smoothly?
I am studying linux container, and when I operated as described according to the site, it has an error.
When I execute the command: lxc-start
root@user-desktop:/home/user/work/lxc/lxc-0.9.0# lxc-start -n foo
lxc-start: failed to attach ‘vethiBVerk’ to the bridge ‘lxcbr0’ : No such device
lxc-start: failed to create netdev
lxc-start: failed to create the network
lxc-start: failed to spawn ‘foo’
root@user-desktop:/home/user/work/lxc/lxc-0.9.0# lxc-start -n foo /bin/bash
lxc-start: failed to attach ‘vethnieCsS’ to the bridge ‘lxcbr0’ : No such device
lxc-start: failed to create netdev
lxc-start: failed to create the network
lxc-start: failed to spawn ‘foo’
I don’t know if my system environment not right, or other problem?
Check to see if the bridge named lxcbr0 exists:
do: $ ifconfig
If it doesn’t then perhaps you’re seeing something I’ve had happen several times in the past year where for some reason the lxcbr0 bridge disappears. When that happens I just reinstall LXC:
$ sudo apt-get install lxc
and it gets recreated. I couldn’t explain this random disappearance of lxcbr0 on my system and its only happened maybe 3 times in the past year.
Hi brian mullan,
Thanks for your help. This problem has been bothering me for a few days.
And I checked the bridge named lxcbr0， it’s not exists.
I can’t find the “USE_LXC_BRIDGE” in my all files.
I reinstall LXC, but the problem is still exist. If have a method to create lxcbro for manual?
Cloud you give me a cue about how to find the reason of lxcbr0 bridge disappears, like a Web links, your blog, ect.
Thanks very much.
I guess I should have asked this also since your original message didn’t say what what linux are you using?
If you are using Ubuntu … then what release? 12.04, 13.04 ?
Sorry, it’s my fault.
I check my ubuntu linux, and the information:
root@user-desktop:/home/user/work/lxc/lxc-0.9.0# uname -a
Linux user-desktop 2.6.32-38-generic #83-Ubuntu SMP Wed Jan 4 11:12:07 UTC 2012 x86_64 GNU/Linux
root@user-desktop:/home/user/work/lxc/lxc-0.9.0# cat /etc/issue
Ubuntu 10.04.4 LTS \n \l
I don’t know if Ubuntu release is too old, so I want change to 12.04 to try again.
Thanks, but append the info what I know, it not usefully. I must resolve this problem.
I install lxc from two ways.
First: I download LXC-0.9.0, then, I executed depend LXC Official Website. In this way, I encounter above problem.
Second: I executed depend this web, and I encounter the problem:
root@user-desktop:/usr/src/lxc-0.9.0# sudo lxc-create -t ubuntu -n my-container
lxc-create: No config file specified, using the default config /usr/local/etc/lxc/default.conf
‘debootstrap’ command is missing
lxc-create: failed to execute template ‘ubuntu’
Oh, LXC makes me a splitting headache.
Anyway I have to thank you brian mullan.
Jason… 10.04 is a bit out of date and if possible you really ought to upgrade to at least 12.04 after which I think your efforts with LXC will be alot more successful.
I used LXC back in 10.04 and I know there was alot more work involved in getting it configured then than today.
Take a look at this site and see if it helps you:
If it hasn’t been mentioned before I’ve found and started using LXC Web Panel
Its a great & easy GUI to manage LXC containers but I think it is only available for Ubuntu.
I haven’t been able to spawn a container. Here’s my config:
Linux amy 3.2.0-53-generic #81-Ubuntu SMP Thu Aug 22 21:01:03 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
And here’s what I get
sudo lxc-start –name=contenedor
lxc-start: failed to spawn ‘contenedor’
It’s a 12.04 vanilla release. Any idea? Is there any log I could check? Or a verbose option?
I am trying to Run android on Ubuntu version on Pandaboard
I expored containers to some extent and Cross compiled lxc for armhf
But I am not able to figure out how to create https://github.com/abstrakraft/lxc-android-lxc/tree/master/templates
But I already have a File system
If I create a
/var/lib/lxc//rootfs say I copy android rootfs for pandaboard here
and lxc-start -n
This approach will it work?
LXC init script doesn’t autostart each symlinks in /etc/lxc/auto
lxc-ls –fancy -> I have all containers autostarted
liblxc0 1.0.0~alpha2+master~20131019-0306-0ubuntu1~ppa1~precise1 Linux Containers userspace tools (library)
root@eurocopter:~# cat /etc/default/lxc
Have you got any idea?
Jumping in the deep end here … but can LXC containers with their direct relationship to their host core, for want of better terminology — provide ‘instances’ RDP style for thin clients like the old HP 5135? http://www.parkytowers.me.uk/thin/hp/t5135/index.shtml
i just read your post and try it 🙂
But have a question about ipv6.
The bridge lxcbr0 created didnt ‘automagicly’ got an ipv6, (my eth0 did, by RA/RS due to my router radvd i presume) .
The lxc container didn’t have ipv6 too.
so .. what should be the stuff to do after a
sudo apt-get install lxcprocedure to have ipv6 on lxc container automagicly ?
sudo lxc-create -n my-container -t ubuntu
sudo lxc-start-ephemeral -o my-container
thanks for your post !
Hi,I met a problem when I was using lxc. I defined and started lxc with virsh command.The setting xml file is as follows:
But I failed to start the lxc with the error [lxcControllerRun:1440 : root source /var/lib/lxc/o2/rootfs does not exist: No such file or directory]
Hope for your help,thanks!
I am sorry there are some problems with xml file displaying in this page.I defined the filesystem type=mount and source dir=”/var/lib/lxc/o2/rootfs”,target dir=”/”
.But it does works when I cancel the filesystem definition in the xml.