This is the eighth blog post in this series about LXD 2.0.
Introduction
In the previous post I covered how to run Docker inside LXD which is a good way to get access to the portfolio of application provided by Docker while running in the safety of the LXD environment.
One use case I mentioned was offering a LXD container to your users and then have them use their container to run Docker. Well, what if they themselves want to run other Linux distributions inside their container using LXD, or even allow another group of people to have access to a Linux system by running a container for them?
Turns out, LXD makes it very simple to allow your users to run nested containers.
Nesting LXD
The most simple case can be shown by using an Ubuntu 16.04 image. Ubuntu 16.04 cloud images come with LXD pre-installed. The daemon itself isn’t running as it’s socket-activated so it doesn’t use any resources until you actually talk to it.
So lets start an Ubuntu 16.04 container with nesting enabled:
lxc launch ubuntu-daily:16.04 c1 -c security.nesting=true
You can also set the security.nesting key on an existing container with:
lxc config set <container name> security.nesting true
Or for all containers using a particular profile with:
lxc profile set <profile name> security.nesting true
With that container started, you can now get a shell inside it, configure LXD and spawn a container:
stgraber@dakara:~$ lxc launch ubuntu-daily:16.04 c1 -c security.nesting=true Creating c1 Starting c1 stgraber@dakara:~$ lxc exec c1 bash root@c1:~# lxd init Name of the storage backend to use (dir or zfs): dir We detected that you are running inside an unprivileged container. This means that unless you manually configured your host otherwise, you will not have enough uid and gid to allocate to your containers. LXD can re-use your container's own allocation to avoid the problem. Doing so makes your nested containers slightly less safe as they could in theory attack their parent container and gain more privileges than they otherwise would. Would you like to have your containers share their parent's allocation (yes/no)? yes Would you like LXD to be available over the network (yes/no)? no Do you want to configure the LXD bridge (yes/no)? yes Warning: Stopping lxd.service, but it can still be activated by: lxd.socket LXD has been successfully configured. root@c1:~# lxc launch ubuntu:14.04 trusty Generating a client certificate. This may take a minute... If this is your first time using LXD, you should also run: sudo lxd init Creating trusty Retrieving image: 100% Starting trusty root@c1:~# lxc list +--------+---------+-----------------------+----------------------------------------------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +--------+---------+-----------------------+----------------------------------------------+------------+-----------+ | trusty | RUNNING | 10.153.141.124 (eth0) | fd7:f15d:d1d6:da14:216:3eff:fef1:4002 (eth0) | PERSISTENT | 0 | +--------+---------+-----------------------+----------------------------------------------+------------+-----------+ root@c1:~#
It really is that simple!
The online demo server
As this post is pretty short, I figured I would spend a bit of time to talk about the demo server we’re running. We also just reached the 10000 sessions mark earlier today!
That server is basically just a normal LXD running inside a pretty beefy virtual machine with a tiny daemon implementing the REST API used by our website.
When you accept the terms of service, a new LXD container is created for you with security.nesting enabled as we saw above. You are then attached to that container as you would when using “lxc exec” except that we’re doing it using websockets and javascript.
The containers you then create inside this environment are all nested LXD containers.
You can then nest even further in there if you want to.
We are using the whole range of LXD resource limitations to prevent one user’s actions from impacting the others and pretty closely monitor the server for any sign of abuse.
If you want to run your own similar server, you can grab the code for our website and the daemon with:
git clone https://github.com/lxc/linuxcontainers.org git clone https://github.com/lxc/lxd-demo-server
Extra information
The main LXD website is at: https://linuxcontainers.org/lxd
Development happens on Github at: https://github.com/lxc/lxd
Mailing-list support happens on: https://lists.linuxcontainers.org
IRC support happens in: #lxcontainers on irc.freenode.net
Try LXD online: https://linuxcontainers.org/lxd/try-it
Github
Twitter
LinkedIn
Mastodon
RSS feed
As I understand only directory (dir) is supported for the moment for nested containers.
Installing zfsutils-linux and trying to use zfs is not possible because lack of /dev/zfs.
For the nested containers, when is it expected to support zfs as storage backend?
That’s correct, only “dir” works right now for nested containers. If /var/lib/lxd happens to be on btrfs and you passed the right mount option to btrfs on the host, then btrfs will be used too (that’s what we do on the demo server to minimize disk usage).
As for zfs, right now their access control is entirely based on access to /dev/zfs. Last I talked with zfs upstream, they did mention planned work to support zfs inside containers with proper access delegation, but I have no idea as to the timeline for this.
What is “the right mount option to btrfs on the host”?
user_subvol_rm_allowed
Looking forward to access delegation, so our containers can be tasked with dataset management instead of requiring to be root on the host
I noticed the article only shows re-using the outer container’s uid range, but doesn’t show nested uid mapping. Here are some commands that worked for me to set that up:
swarren@cam:~$ cat /etc/subuid
lxd:100000:131072
root:100000:131072
swarren:300000:199608
swarren@cam:~$ cat /etc/subgid
lxd:100000:131072
root:100000:131072
swarren:300000:199608
# I assume this is needed to pick up the changes to /etc/sub[ug]id
swarren@cam:~$ sudo service lxd restart
swarren@cam:~$ lxc launch -c security.nesting=true ubuntu:16.04 uboot-jenkins
Creating uboot-jenkins
Starting uboot-jenkins
swarren@cam:~$ lxc exec uboot-jenkins /bin/bash
root@uboot-jenkins:~# cat > /etc/subuid
lxd:65536:65536
root:65536:65536
ubuntu:131072:65536
^D
root@uboot-jenkins:~# cat > /etc/subgid
lxd:65536:65536
root:65536:65536
ubuntu:131072:65536
^D
root@uboot-jenkins:~# lxd init
Name of the storage backend to use (dir or zfs): dir
We detected that you are running inside an unprivileged container.
This means that unless you manually configured your host otherwise,
you will not have enough uid and gid to allocate to your containers.
LXD can re-use your container’s own allocation to avoid the problem.
Doing so makes your nested containers slightly less safe as they could
in theory attack their parent container and gain more privileges than
they otherwise would.
Would you like to have your containers share their parent’s allocation (yes/no)? no
Would you like LXD to be available over the network (yes/no)? no
Do you want to configure the LXD bridge (yes/no)? yes
Warning: Stopping lxd.service, but it can still be activated by:
lxd.socket
LXD has been successfully configured.
root@uboot-jenkins:~# lxc remote add outer 192.168.63.3
Generating a client certificate. This may take a minute…
If this is your first time using LXD, you should also run: sudo lxd init
Certificate fingerprint: 191acad76df9d46cc8680997b1c84f36d61cce9fdfc3054cf2beb534fb3e9454
ok (y/n)? y
Admin password for outer:
Client certificate stored at server: outer
root@uboot-jenkins:~# lxc launch outer:f4c4c60a6b75 nested
Creating nested
Retrieving image: 100%
Starting nested
root@uboot-jenkins:~# lxc exec nested — bash
root@nested:~#
swarren@cam:~$ lxc list
+—————+———+——————————–+——+————+———–+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+—————+———+——————————–+——+————+———–+
| uboot-jenkins | RUNNING | 10.149.93.35 (eth0) | | PERSISTENT | 0 |
| | | 10.142.94.1 (lxdbr0) | | | |
+—————+———+——————————–+——+————+———–+
swarren@cam:~$ lxc exec uboot-jenkins — lxc list
+——–+———+———————+——+————+———–+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+——–+———+———————+——+————+———–+
| nested | RUNNING | 10.142.94.15 (eth0) | | PERSISTENT | 0 |
+——–+———+———————+——+————+———–+
Thanks, I’m sure that will be useful to some people!
Typo alert:
git clone github.com/lxc/linuxcontainers.org
git clone github.com/lxc/lxd-demo-server
You forgot the https://s there, too much ‘go get’? 🙂
I guess so 🙂 Fixed!
Is it possible to limit the number of container creation in nested lxd?