This is post 4 out of 10 in the LXC 1.0 blog post series.
Running foreign architectures
By default LXC will only let you run containers of one of the architectures supported by the host. That makes sense since after all, your CPU doesn’t know what to do with anything else.
Except that we have this convenient package called “qemu-user-static” which contains a whole bunch of emulators for quite a few interesting architectures. The most common and useful of those is qemu-arm-static which will let you run most armv7 binaries directly on x86.
The “ubuntu” template knows how to make use of qemu-user-static, so you can simply check that you have the “qemu-user-static” package installed, then run:
sudo lxc-create -t ubuntu -n p3 -- -a armhf
After a rather long bootstrap, you’ll get a new p3 container which will be mostly running Ubuntu armhf. I’m saying mostly because the qemu emulation comes with a few limitations, the biggest of which is that any piece of software using the ptrace() syscall will fail and so will anything using netlink. As a result, LXC will install the host architecture version of upstart and a few of the networking tools so that the containers can boot properly.
stgraber@castiana:~$ file /bin/ls /bin/ls: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, """BuildID[sha1]""" =e50e0a5dadb8a7f4eaa2fd715cacb9842e157dc7, stripped stgraber@castiana:~$ sudo lxc-start -n p3 -d stgraber@castiana:~$ sudo lxc-attach -n p3 root@p3:/# file /bin/ls /bin/ls: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, """BuildID[sha1]""" =88ff013a8fd9389747fb1fea1c898547fb0f650a, stripped root@p3:/# exit stgraber@castiana:~$ sudo lxc-stop -n p3 stgraber@castiana:~$
Hooks
As we know people like to script their containers and that our configuration can’t always accommodate every single use case, we’ve introduced a set of hooks which you may use.
Those hooks are simple paths to an executable file which LXC will run at some specific time in the lifetime of the container. Those executables will also be passed a set of useful environment variables so they can easily know what container invoked them and what to do.
The currently available hooks are (details in lxc.conf(5)):
- lxc.hook.pre-start (called before any initialization is done)
- lxc.hook.pre-mount (called after creating the mount namespace but before mounting anything)
- lxc.hook.mount (called after the mounts but before pivot_root)
- lxc.hook.autodev (identical to mount but only called if using autodev)
- lxc.hook.start (called in the container right before /sbin/init)
- lxc.hook.post-stop (run after the container has been shutdown)
- lxc.hook.clone (called when cloning a container into a new one)
Additionally each network section may also define two additional hooks:
- lxc.network.script.up (called in the network namespace after the interface was created)
- lxc.network.script.down (called in the network namespace before destroying the interface)
All of those hooks may be specified as many times as you want in the configuration so you can use each hooking point multiple times.
As a simple example, let’s add the following to our “p1” container:
lxc.hook.pre-start = /var/lib/lxc/p1/pre-start.sh
And create the hook itself at /var/lib/lxc/p1/pre-start.sh:
#!/bin/sh echo "arguments: $*" > /tmp/test echo "environment:" >> /tmp/test env | grep LXC >> /tmp/test
Make it executable (chmod 755) and then start the container.
Checking /tmp/test you should see:
arguments: p1 lxc pre-start environment: LXC_ROOTFS_MOUNT=/usr/lib/x86_64-linux-gnu/lxc LXC_CONFIG_FILE=/var/lib/lxc/p1/config LXC_ROOTFS_PATH=/var/lib/lxc/p1/rootfs LXC_NAME=p1
Android containers
I’ve often been asked whether it was possible to run Android in an LXC container. Well, the short answer is yes. However it’s not very simple and it really depends on what you want to do with it.
The first thing you’ll need if you want to do this is get your machine to run an Android kernel, you’ll need to have any modules needed by Android built and loaded before you can start the container.
Once you have that, you’ll need to create a new container by hand.
Let’s put it in “/var/lib/lxc/android/”, in there, you need a configuration file similar to this one:
lxc.rootfs = /var/lib/lxc/android/rootfs lxc.utsname = armhf lxc.network.type = none lxc.devttydir = lxc lxc.tty = 4 lxc.pts = 1024 lxc.arch = armhf lxc.cap.drop = mac_admin mac_override lxc.pivotdir = lxc_putold lxc.hook.pre-start = /var/lib/lxc/android/pre-start.sh lxc.aa_profile = unconfined
/var/lib/lxc/android/pre-start.sh is where the interesting bits happen. It needs to be an executable shell script, containing something along the lines of:
#!/bin/sh mkdir -p $LXC_ROOTFS_PATH mount -n -t tmpfs tmpfs $LXC_ROOTFS_PATH cd $LXC_ROOTFS_PATH cat /var/lib/lxc/android/initrd.gz | gzip -d | cpio -i # Create /dev/pts if missing mkdir -p $LXC_ROOTFS_PATH/dev/pts
Then get the initrd for your device and place it in /var/lib/lxc/android/initrd.gz.
At that point, when starting the LXC container, the Android initrd will be unpacked on a tmpfs (similar to Android’s ramfs) and Android’s init will be started which in turn should mount any partition that Android requires and then start all of the usual services.
Because there are no apparmor, cgroup or even network configuration applied to it, the container will have a lot of rights and will typically completely crash the machine. You unfortunately have to be familiar with the way Android works and not be afraid to modify its init scripts if not even its init process to only start the bits you actually want.
I can’t provide a generic recipe there as it completely depends on what you’re interested on, what version of Android and what device you’re using. But it’s clearly possible to do and you may want to look at Ubuntu Touch to see how we’re doing it by default there.
One last note, Android’s init script isn’t in /sbin/init, so you need to tell LXC where to load it with:
lxc-start -n android -- /init
LXC on Android devices
So now that we’ve seen how to run Android in LXC, let’s talk about running Ubuntu on Android in LXC.
LXC has been ported to bionic (Android’s C library) and while not feature-equivalent with its glibc build, it’s still good enough to be used.
Unfortunately due to the kind of low level access LXC requires and the fact that our primary focus isn’t Android, installation could be easier…You won’t be finding LXC on the Google PlayStore and we won’t provide you with a .apk that you can install.
Instead every time something changes in the upstream git branch, we produce a new tarball which can be downloaded here: https://jenkins.linuxcontainers.org/view/LXC/view/LXC%20builds/job/lxc-build-android/lastSuccessfulBuild/artifact/lxc-android.tar.gz
This build is known to work with Android >= 4.2 but will quite likely work on older versions too.
For this to work, you’ll need to grab your device’s kernel configuration and run lxc-checkconfig against it to see whether it’s compatible with LXC or not. Unfortunately it’s very likely that it won’t be… In that case, you’ll need to go hunt for the kernel source for your device, add the missing feature flags, rebuild it and update your device to boot your updated kernel.
As scary as this may sound, it’s usually not that difficult as long as your device is unlocked and you’re already using an alternate ROM like Cyanogen which usually make their kernel git tree easily available.
Once your device has a working kernel, all you need to do is unpack our tarball as root in your device’s / directory, copy an arm container to /data/lxc/containers/<container name>, get into /data/lxc and run “./run-lxc lxc-start -n <container name>”.
A few seconds later you’ll be greeted by a login prompt.
Github
Twitter
LinkedIn
Mastodon
RSS feed
I checked the configuration of my android emulator using lxc-checkconfig and it is showing namespaces, cgroups, etc required. How to enable those kernel features for running lxc in android?
You need to grab the source code for your current kernel, then edit .config either by hand or using something like “make menuconfig” to find and enable the missing options.
about the hooks…
i’m trying to set up an OS container running archlinux. Here I encounter 2 distinct road bumps:
– start the container’s systemd (lower priority for me).
– get pacman running to update and install new stuff.
systemd-nspawn also has been described as “chroot on steroids” and uses cgroup to “partition” the resources. this may be an interoperability with lxc a breeze or ugly nightmare. I Guess time will tell if more distro use it as their init system (or not).
to get pacman properly running in a chroot one needs to mount/create /proc (for mtab link actually), /dev/random, /dev/urandom and /dev/pts (although my personal experience is that it is not REQUIRED). Now in all the containers I’ve tried pacman fails on a GPGME: ioctl not appropriate error even if the resources are present and accounted for.
None of the (many) variations I’ve tried so far on the container configuration file did improve that point.
I’ve been testing this with wheezy (kernel 3.2) and arch (kernel 3.12) hosts.
After that longish preamble my question: can you suggest some magic stanza to put in say lxc.hook.autodev and or lxc.hook.start ?
Hello.
I’m on the samsung arm chromebook. I’ve installed ubuntu 12.04 LTS with the kernel 3.13 already configured for lxc and android. Now I would like to know how to configure the android container to run android as guest os and ubuntu as host os….
I can’t really help you much there, though things that you need to know are that ChromeOS isn’t Android, it actually doesn’t share any code with Android that I’m aware of.
Running Android in LXC requires an Android kernel which if you’re using a ChromeOS kernel, you don’t have.
So you’d first need to find an actual Android build for your Chromebook, including its kernel, then make your Chromebook run using that kernel (or rebuild it so that it does) and after that, you should be able to run that Android build inside LXC.
I’m way late to the party here but, he said (>3 years ago) he was running Ubuntu 12.04 LTS on the device rather than ChromeOS – & that his kernel was already configured for LXC and Android (presumably a modded Android kernel to support LXC).
That said, I’m not sure what the question was. Seems like he just needed to read up on LXC commands and fire up a container (I only know Docker, so I’m no help here – not sure why I typed this tbh!)
Your hook example runs the script in the host. Is there a way to run the script inside the container?
could you please shed me the light on how to build the lxc running on Android platform? I saw the script of build-android in the lxc-ci folder in github but I can’t make it workable. thx…
Hey Stéphane, first of all, great blog series. Very helpful. Thanks for putting this together!
In the Android-in-a-container context: I was wondering if you had any pointers on borrowing the lxc generic-x86 recipe from ubuntu-touch to run on real, generic x86 hardware, like a simple Intel Atom-based SoC that has PowerVR graphics. Do you have any plans on writing on the subject?
Hi Stephane,
I just rooted a spare Galaxy S3 of mine and I was thinking about experimenting Docker on it…Is that possible today?
thanks!
I’m just trying to do the same thing!
I’m trying to translate the go lang to java using gobind then run docker on a android device!
But I really doubt if it works, for it is true that docker may require few low access.If you have more information about this,could you please share it with me?really appreciate that!thanks!
Hi Stephane,
I was trying to use the Android binaries as in paragraph “LXC on Android devices”.
Unfortunatly it does not work on 4.4.4. It keeps saying:
lxc-start: No such file or directory – failed to create symlink for kmsg
As a matter of fact lxc-checkconfig says that everything is setup correctly.
Do you may have any clue?
PS: is it possible to have access to the pathches/build configuration to be able to recompile LXC for bionic?
Thanks in advance.
Do you may have any clue on this?
This is likely due to the changes in permitions that happened in 4.4. They locked down SD cards without giving you a way to assign the permitions needed to use the files. They fixed this by adding new permitions to let programs access only the files they need on the card in 4.5… in the mean time lots of stuff is broken in 4.4. Hope this helps.
Hi Stephane; how would one go about sticking an Android container in AppArmor?
about lxc on android..
I couldn’t find a “run-lxc” file in your pre-built-tarball.
When I execute “lxc-create” or “sh lxc-create” immediately, an error message is occurred like as followings:
# /system/bin/sh ./lxc-create
./lxc-create[1]: syntax error: ‘?4’ unexpected
Running “lxc-checkconfig” was success through installing busybox, but running “lxc-create” was failed.
please help me.
Hi abraham,
The code has changed somewhat since Stephane first wrote the blog post and he has not had time to update it. On https://code.launchpad.net/~ubuntu-lxc/lxc/android-build-scripts click “Browse the code” and you will see the “run-lxc” script.
– yba
Hey,
may someone who had success in following the steps in this post can name the ROM/Kernel Version and device they used. I’m currently trying to use the Nexus 5 (hammerhead).
Thanks!
Hi, i have some problems starting a container under Android following this blog post.
Here is my setup:
Nexus 5 (Hammerhead)
Cyanogenmod 11 (Kernel 3.4, Android 4.4.4) – I added the missing options and checked them with lxc-checkconfig, then recompiled the kernel.
Created a container on my Ubuntu Machine using:
sudo lxc-create -t ubuntu -n p3 — -a armhf
Copied the container to the phone (/data/lxc/containers/p3) and adapted the path information in the p3/config file.
I downloaded the tarball containing the prebuilt lxc-utils and installed them as mentioned above (and then added the missing run-lxc and prepare.sh from the link Jonathan provided on the replies). I also downloaded the lxc-utils source and compiled them but the results i get is the same.
When i execute ./run-lxc lxc-start -n p3 -F i get the following error:
1|root@hammerhead:/data/lxc # ./run-lxc lxc-start -n p3 -F
lxc-start: cgfsng.c: all_controllers_found: 431 no systemd controller mountpoint found
lxc-start: start.c: lxc_spawn: 1036 failed initializing cgroup support
lxc-start: start.c: __lxc_start: 1286 failed to spawn ‘p3’
lxc-start: lxc_start.c: main: 344 The container failed to start.
Unfortunately, i was not able to track down the source of this error.
Best,
Torsten
Try to folow more details instructions here (in French)
http://www.enten.fr/steven/2014/12/30/lxc-et-android-5-arm64/
seem in your case that cgroup are not mounted…
Do you know what kernels are compatible with LXC(i. e. CyanogenMod)?
First please excuse me if my question doesn’t make sense. I’m very new to the LXC.
Regarding running Android in an LXC container, this post, which is admittedly quite old, states:
“The first thing you’ll need if you want to do this is get your machine to run an Android kernel, you’ll need to have any modules needed by Android built and loaded before you can start the container.”
I’ve been trying to learn more about how Google manages to run Android apps on their Chromebooks. I’m under the impression (or possibly mis-impression) that, to do so, Google now relies on containerization to run a stripped down version of Android as a container on top of Chrome OS. But the Chromebooks that can run Android apps do so using a kernel that, so I’m told, doesn’t differ much from a mainline Linux kernel. See https://groups.google.com/a/chromium.org/forum/#!topic/chromium-os-discuss/OfBln-hl7ug
Would you please share a few brief thoughts (or even speculation) on how Google is running an Android container on a machine that runs a nearly mainline Linux kernel?
Thanks in advance for any information that you can provide!
Best regards,
GizmoChicken
Hello All,
We are trying an Android container on dra7xx ARM platform but every-time when starting container, it leads to segfault.
wait4(2745, NULL, 0, NULL) = 2745
— SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2745, si_uid=0, si_status=0, si_utime=0, si_stime=0} —
socket(PF_LOCAL, SOCK_STREAM, 0) = 4
connect(4, {sa_family=AF_LOCAL, sun_path=@”lxc/ad055575fe28ddd5//var/lib/lxc”}, 36) = -1 ECONNREFUSED (Connection refused)
gettimeofday({1473837550, 356010}, NULL) = 0
— SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x8} —
+++ killed by SIGSEGV +++
Segmentation fault
Please provide your pointer, what could be the potential reason ?
Hi,
Thank you very much for your blog.
I have a some Question.
I can find “initrd.gz” file.
How can I find “initrd.gz” file?
Hi, I’ve trouble to make lxc run on android 4.4 (cyanogenmod11).
My kernel is compatible, I untared file, downloaded scripts from http://bazaar.launchpad.net/~ubuntu-lxc/lxc/android-build-scripts/files and when I try to do anything (at example lxc-ls) I get much errors:
“`
127|root@kis3:/data/lxc # sh ./run-lxc lxc-ls
chroot: can’t execute ‘lxc-ls’: No such file or directory
“`
“`
127|root@kis3:/data/lxc # sh ./prepare.sh
link failed No such file or directory
“`
This looks like running 32 bit bins on 64 bit architecture, I am also trying to compile it for 64 bit ARM so that this doesnt happen
Lxc -build-android build lxc src .Can lxc-android.tar.gz run on android x86 ?
This lxc need to use kernel>3.8？I find /proc/PID/ns/mnt can not work on kernel <3.8
Can you give me a demo or link about runing lxc on android x86? because android x86 runing on PC with x86 much more fast!
Could you possibly also host a job providing ARM64 configuration for https://jenkins.linuxcontainers.org/view/LXC/job/lxc-build-android/
I wanted to try it on 64 bit ARM processor
Hi,
Thank you very much for your blog.
I have a some Question.
I can find “initrd.gz” file.
How can I find “initrd.gz” file?