Running snaps in LXD containers

LXD logo

Introduction

The LXD and AppArmor teams have been working to support loading AppArmor policies inside LXD containers for a while. This support which finally landed in the latest Ubuntu kernels now makes it possible to install snap packages.

Snap packages are a new way of distributing software, directly from the upstream and with a number of security features wrapped around them so that these packages can’t interfere with each other or cause harm to your system.

Requirements

There are a lot of moving pieces to get all of this working. The initial enablement was done on Ubuntu 16.10 with Ubuntu 16.10 containers, but all the needed bits are now progressively being pushed as updates to Ubuntu 16.04 LTS.

The easiest way to get this to work is with:

  • Ubuntu 16.10 host
  • Stock Ubuntu kernel (4.8.0)
  • Stock LXD (2.4.1 or higher)
  • Ubuntu 16.10 container with “squashfuse” manually installed in it

Installing the nextcloud snap

First, lets get ourselves an Ubuntu 16.10 container with “squashfuse” installed inside it.

lxc launch ubuntu:16.10 nextcloud
lxc exec nextcloud -- apt update
lxc exec nextcloud -- apt dist-upgrade -y
lxc exec nextcloud -- apt install squashfuse -y

And then, lets install that “nextcloud” snap with:

lxc exec nextcloud -- snap install nextcloud

Finally, grab the container’s IP and access “http://<IP>” with your web browser:

stgraber@castiana:~$ lxc list nextcloud
+-----------+---------+----------------------+----------------------------------------------+
|    NAME   |  STATE  |         IPV4         |                     IPV6                     |
+-----------+---------+----------------------+----------------------------------------------+
| nextcloud | RUNNING | 10.148.195.47 (eth0) | fd42:ee2:5d34:25c6:216:3eff:fe86:4a49 (eth0) |
+-----------+---------+----------------------+----------------------------------------------+

Nextcloud Login screen

Installing the LXD snap in a LXD container

First, lets get ourselves an Ubuntu 16.10 container with “squashfuse” installed inside it.
This time with support for nested containers.

lxc launch ubuntu:16.10 lxd -c security.nesting=true
lxc exec lxd -- apt update
lxc exec lxd -- apt dist-upgrade -y
lxc exec lxd -- apt install squashfuse -y

Now lets clear the LXD that came pre-installed with the container so we can replace it by the snap.

lxc exec lxd -- apt remove --purge lxd lxd-client -y

Because we already have a stable LXD on the host, we’ll make things a bit more interesting by installing the latest build from git master rather than the latest stable release:

lxc exec lxd -- snap install lxd --edge

The rest is business as usual for a LXD user:

stgraber@castiana:~$ lxc exec lxd bash
root@lxd:~# lxd init
Name of the storage backend to use (dir or zfs) [default=dir]:

We detected that you are running inside an unprivileged container.
This means that unless you manually configured your host otherwise,
you will not have enough uid and gid to allocate to your containers.

LXD can re-use your container's own allocation to avoid the problem.
Doing so makes your nested containers slightly less safe as they could
in theory attack their parent container and gain more privileges than
they otherwise would.

Would you like to have your containers share their parent's allocation (yes/no) [default=yes]?
Would you like LXD to be available over the network (yes/no) [default=no]?
Would you like stale cached images to be updated automatically (yes/no) [default=yes]?
Would you like to create a new network bridge (yes/no) [default=yes]?
What should the new bridge be called [default=lxdbr0]?
What IPv4 subnet should be used (CIDR notation, “auto” or “none”) [default=auto]?
What IPv6 subnet should be used (CIDR notation, “auto” or “none”) [default=auto]?
LXD has been successfully configured.

root@lxd:~# lxd.lxc launch images:archlinux arch
If this is your first time using LXD, you should also run: sudo lxd init
To start your first container, try: lxc launch ubuntu:16.04

Creating arch
Starting arch

root@lxd:~# lxd.lxc list
+------+---------+----------------------+-----------------------------------------------+------------+-----------+
| NAME |  STATE  |         IPV4         |                      IPV6                     |    TYPE    | SNAPSHOTS |
+------+---------+----------------------+-----------------------------------------------+------------+-----------+
| arch | RUNNING | 10.106.137.64 (eth0) | fd42:2fcd:964b:eba8:216:3eff:fe8f:49ab (eth0) | PERSISTENT | 0         |
+------+---------+----------------------+-----------------------------------------------+------------+-----------+

And that’s it, you now have the latest LXD build installed inside a LXD container and running an archlinux container for you. That LXD build will update very frequently as we publish new builds to the edge channel several times a day.

Conclusion

It’s great to have snaps now install properly inside LXD containers. Production users can now setup hundreds of different containers, network them the way they want, setup their storage and resource limits through LXD and then install snap packages inside them to get the latest upstream releases of the software they want to run.

That’s not to say that everything is perfect yet. This is all built on some really recent kernel work, using unprivileged FUSE filesystem mounts and unprivileged AppArmor profile stacking and namespacing. There very likely still are some issues that need to get resolved in order to get most snaps to work identically to when they’re installed directly on the host.

If you notice discrepancies between a snap running directly on the host and a snap running inside a LXD container, you’ll want to look at the “dmesg” output, looking for any DENIED entry in there which would indicate AppArmor rejecting some request from the snap.

This typically indicates either a bug in AppArmor itself or in the way the AppArmor profiles are generated by snapd. If you find one of those issues, you can report it in #snappy on irc.freenode.net or file a bug at https://launchpad.net/snappy/+filebug so it can be investigated.

Extra information

More information on snap packages can be found at: http://snapcraft.io

The main LXD website is at: https://linuxcontainers.org/lxd
Development happens on Github at: https://github.com/lxc/lxd
Mailing-list support happens on: https://lists.linuxcontainers.org
IRC support happens in: #lxcontainers on irc.freenode.net
Try LXD online: https://linuxcontainers.org/lxd/try-it

About Stéphane Graber

Project leader of Linux Containers, Linux hacker, Ubuntu core developer, conference organizer and speaker.
This entry was posted in Canonical voices, LXD, Planet Ubuntu and tagged . Bookmark the permalink.

12 Responses to Running snaps in LXD containers

  1. Luca Dionisi says:

    Kudos!

    I wonder if, in future, we’ll be able to have an image of Ubuntu Core run as a LXD container inside a Ubuntu Server host.

    1. I do have such an image mostly working on my laptop. It’s pretty far from being a clean image that you can download from the image server though 🙂

      1. Luca Dionisi says:

        Well, a bright future lies ahead! Or at least quite so for me.

  2. Mark Dahlhauser says:

    So I am trying to get nextcloud snap to work in LXD container with ubuntu 16.10. I followed you instructions exactly. I get the web interface but when it tries to create the user is just spins and never continues. I think it is AppArmor because I see DENY for file_perm but I have no idea how to fix it. Thank for any help you might be able to provide

  3. Had to run `echo “. /etc/profile.d/apps-bin-path.sh” >> ~/.bashrc` to got it fully working.

    A big thank you !

  4. ttbek says:

    Does this work with the current Arch kernel? My intuition wants to say that this relies on user namespaces and hence won’t work, but I’m no expert with lxc.

    1. You need to build and run linux-userns if you want unprivileged containers, otherwise you’ll be limited to privileged containers (security.privileged=true).

  5. NothingCtrl says:

    Great post, thanks you.

  6. Greg Bray says:

    If you get this:

    Jul 19 19:21:46 k8s2 systemd[1]: Starting Snappy daemon…
    Jul 19 19:21:46 k8s2 snapd[1265]: AppArmor status: apparmor is enabled and all features are available
    Jul 19 19:21:46 k8s2 snapd[1265]: error: cannot start snapd: cannot mount squashfs image using “fuse.squashfuse”:
    Jul 19 19:21:46 k8s2 snapd[1265]: —–
    Jul 19 19:21:46 k8s2 snapd[1265]: mount: wrong fs type, bad option, bad superblock on /tmp/selftest-squashfs-297231164,
    Jul 19 19:21:46 k8s2 snapd[1265]: missing codepage or helper program, or other error
    Jul 19 19:21:46 k8s2 snapd[1265]: In some cases useful info is found in syslog – try
    Jul 19 19:21:46 k8s2 snapd[1265]: dmesg | tail or so.

    You need to install the fuse dependencies:

    lxc exec $name — apt-get install squashfuse fuse

  7. hurelhuyag says:

    Is it work with ubuntu 16.04, kernel 4.15, lxd 2.21?

    1. hurelhuyag says:

      Apparmor blocking squashfs mount inside lxd container ubuntu 16.04

  8. hurelhuyag says:

    Someone needs to document somewhere snapd is depends on fuse and squashfuse

Leave a Reply to NothingCtrl Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.