LXD 2.0: LXD and OpenStack [11/12]

This is the eleventh blog post in this series about LXD 2.0.

LXD logo

Introduction

First of all, sorry for the delay. It took quite a long time before I finally managed to get all of this going. My first attempts were using devstack which ran into a number of issues that had to be resolved. Yet even after all that, I still wasn’t be able to get networking going properly.

I finally gave up on devstack and tried “conjure-up” to deploy a full Ubuntu OpenStack using Juju in a pretty user friendly way. And it finally worked!

So below is how to run a full OpenStack, using LXD containers instead of VMs and running all of this inside a LXD container (nesting!).

Requirements

This post assumes you’ve got a working LXD setup, providing containers with network access and that you have a pretty beefy CPU, around 50GB of space for the container to use and at least 16GB of RAM.

Remember, we’re running a full OpenStack here, this thing isn’t exactly light!

Setting up the container

OpenStack is made of a lof of different components, doing a lot of different things. Some require some additional privileges so to make our live easier, we’ll use a privileged container.

We’ll configure that container to support nesting, pre-load all the required kernel modules and allow it access to /dev/mem (as is apparently needed).

Please note that this means that most of the security benefit of LXD containers are effectively disabled for that container. However the containers that will be spawned by OpenStack itself will be unprivileged and use all the normal LXD security features.

lxc init ubuntu:16.04 openstack -c security.privileged=true -c security.nesting=true -c "linux.kernel_modules=iptable_nat, ip6table_nat, ebtables, openvswitch, nbd"
printf "lxc.cap.drop=\nlxc.aa_profile=unconfined\n" | lxc config set openstack raw.lxc -
lxc config device add openstack mem unix-char path=/dev/mem
lxc start openstack

 

Then we need to add a couple of PPAs and install conjure-up, the deployment tool we’ll use to get OpenStack going.

lxc exec openstack -- apt update
lxc exec openstack -- apt dist-upgrade -y
lxc exec openstack -- apt install squashfuse -y
lxc exec openstack -- ln -s /bin/true /usr/local/bin/udevadm
lxc exec openstack -- snap install conjure-up --classic

And the last setup step is to configure LXD networking inside the container.
Answer with the default for all questions, except for:

  • Use the “dir” storage backend (“zfs” doesn’t work in a nested container)
  • Do NOT configure IPv6 networking (conjure-up/juju don’t play well with it)
lxc exec openstack -- lxd init

And that’s it for the container configuration itself, now we can deploy OpenStack!

Deploying OpenStack with conjure-up

As mentioned earlier, we’ll be using conjure-up to deploy OpenStack.
This is a nice, user friendly, tool that interfaces with Juju to deploy complex services.

Start it with:

lxc exec openstack -- sudo -u ubuntu -i conjure-up
  • Select “OpenStack with NovaLXD”
  • Then select “localhost” as the deployment target (uses LXD)
  • And hit “Deploy all remaining applications”

This will now deploy OpenStack. The whole process can take well over an hour depending on what kind of machine you’re running this on. You’ll see all services getting a container allocated, then getting deployed and finally interconnected.

Conjure-Up deploying OpenStack

Once the deployment is done, a few post-install steps will appear. This will import some initial images, setup SSH authentication, configure networking and finally giving you the IP address of the dashboard.

Access the dashboard and spawn a container

The dashboard runs inside a container, so you can’t just hit it from your web browser.
The easiest way around this is to setup a NAT rule with:

lxc exec openstack -- iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to <IP>

Where “<ip>” is the dashboard IP address conjure-up gave you at the end of the installation.

You can now grab the IP address of the “openstack” container (from “lxc info openstack”) and point your web browser to: http://<container ip>/horizon

This can take a few minutes to load the first time around. Once the login screen is loaded, enter the default login and password (admin/openstack) and you’ll be greeted by the OpenStack dashboard!

oslxd-dashboard

You can now head to the “Project” tab on the left and the “Instances” page. To start a new instance using nova-lxd, click on “Launch instance”, select what image you want, network, … and your instance will get spawned.

Once it’s running, you can assign it a floating IP which will let you reach your instance from within your “openstack” container.

Conclusion

OpenStack is a pretty complex piece of software, it’s also not something you really want to run at home or on a single server. But it’s certainly interesting to be able to do it anyway, keeping everything contained to a single container on your machine.

Conjure-Up is a great tool to deploy such complex software, using Juju behind the scene to drive the deployment, using LXD containers for every individual service and finally for the instances themselves.

It’s also one of the very few cases where multiple level of container nesting actually makes sense!

Extra information

The conjure-up website can be found at: http://conjure-up.io
The Juju website can be found at: http://www.ubuntu.com/cloud/juju

The main LXD website is at: https://linuxcontainers.org/lxd
Development happens on Github at: https://github.com/lxc/lxd
Mailing-list support happens on: https://lists.linuxcontainers.org
IRC support happens in: #lxcontainers on irc.freenode.net
Try LXD online: https://linuxcontainers.org/lxd/try-it

This entry was posted in Canonical voices, LXD, Planet Ubuntu and tagged . Bookmark the permalink.

17 Responses to LXD 2.0: LXD and OpenStack [11/12]

  1. Henry Lawson says:

    Thank you for this. I had attempted open stack on LXD a few months ago but got stuck! I didn’t know about conjure up then.

    I am looking forward to giving this a go!

  2. tom zhou says:

    seems this is a dev/test setup. Not for production .

    • Absolutely. Someone who would want to run a production setup will want to deploy OpenStack on a bunch of physical servers after doing some research on the exact options they want for storage, network, …

      The setup described here is a way to get OpenStack going so you can play around, see how things work and try nova-lxd.

  3. Satya Kada says:

    Hi Stéphane Graber,
    Thank you for the post. Had lot of information. Will try deploying the openstack with nova-lxd. I am new to LXD, I need help on the following
    1. Networking details to setup LXD.
    2. I want to test live-migration between remote hosts(here containers). So I want to add multiple nova-compute containers with nova-lxd. How can I add multiple compute nodes(here multiple compute node containers) using conjure-up.

  4. Wayne says:

    Is it possible to do this in a network without IPv6? I get lots of messages like this.

    ?: invalid HTTP proxy (http://[fe80::1%eth0]:13128): bad URI

  5. Harry says:

    Excellent guide.
    I am using devstack within LXD with only a few problems.
    I have to ask though.
    Why do you pass /dev/mem ?

    • I don’t actually know 🙂

      The reason why I’m passing it to the main container is because Juju through conjure-up attempts to pass it to the nested openstack containers. So not having it in the parent container was causing deployment to fail.

      I can’t think of a good reason for them to actually use /dev/mem, so hopefully someone can sort out Juju/charms so that they stop passing it through.

  6. Gaetan QUENTIN says:

    lxc exec openstack — ln -s /bin/true /usr/local/bin/modprobe
    error: flag provided but not defined: -s

    works better with the ” ‘ ” :
    lxc exec openstack — ‘ln -s /bin/true /usr/local/bin/modprobe’

    Tried your install on ubuntu 16.10

    i have a lot of:
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup.service: Operation not permitted
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/proc-swaps.mount: Operation not permitted
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/conjure-up.service: Operation not permitted
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/systemd-update-utmp.service: Operation not permitted
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/cron.service: Operation not permitted
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/dev-hugepages.mount: Operation not permitted
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/sys-devices-virtual-net.mount: Operation not permitted
    Dec 8 19:35:13 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:35:13 openstack systemd[1]: Starting LSB: AppArmor initialization…
    Dec 8 19:35:13 openstack apparmor[75]: /etc/init.d/apparmor: 256: /etc/init.d/apparmor: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
    Dec 8 19:35:13 openstack apparmor[75]: * Not starting AppArmor in container
    Dec 8 19:35:13 openstack apparmor[75]: …done.
    Dec 8 19:35:13 openstack systemd[1]: Started LSB: AppArmor initialization.
    Dec 8 19:35:13 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:38:35 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:39:56 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:43:05 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:43:07 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:47:19 openstack lxc-apparmor-load[3900]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
    Dec 8 19:47:19 openstack lxc-apparmor-load[3900]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
    Dec 8 19:47:19 openstack lxd[3905]: t=2016-12-08T19:47:19+0000 lvl=warn msg=”Per-container AppArmor profiles are disabled because the mac_admin capability is missing.”
    Dec 8 19:47:19 openstack lxd[3905]: t=2016-12-08T19:47:19+0000 lvl=warn msg=”Per-container AppArmor profiles are disabled because LXD is already protected by AppArmor.”
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:50:38 openstack lxc-apparmor-load[4242]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
    Dec 8 19:50:38 openstack lxc-apparmor-load[4242]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
    Dec 8 19:50:38 openstack lxd[4247]: t=2016-12-08T19:50:38+0000 lvl=warn msg=”Per-container AppArmor profiles are disabled because the mac_admin capability is missing.”
    Dec 8 19:50:38 openstack lxd[4247]: t=2016-12-08T19:50:38+0000 lvl=warn msg=”Per-container AppArmor profiles are disabled because LXD is already protected by AppArmor.”

    And:

    Dec 8 19:56:00 openstack conjure-up/openstack-novalxd: Running pre-deploy for OpenStack
    Dec 8 19:56:05 openstack conjure-up/openstack-novalxd: Processing lxd profile: conjure-up
    Dec 8 19:56:13 openstack conjure-up/openstack-novalxd: Running pre-deploy for OpenStack
    Dec 8 19:56:18 openstack conjure-up/openstack-novalxd: Processing lxd profile: conjure-up
    Dec 8 20:01:00 openstack conjure-up/openstack-novalxd: Running deploy-done for OpenStack installation.

    And in lxd.log:
    It looks like it is looping:
    t=2016-12-08T20:08:53+0000 lvl=info msg=”Deleted container” creation date=2016-12-08T20:08:47+0000 ephemeral=false name=juju-5c77ff-3
    t=2016-12-08T20:09:05+0000 lvl=info msg=”Deleting container” name=juju-5c77ff-10 creation date=2016-12-08T20:08:59+0000 ephemeral=false
    t=2016-12-08T20:09:06+0000 lvl=info msg=”Deleted container” name=juju-5c77ff-10 creation date=2016-12-08T20:08:59+0000 ephemeral=false
    t=2016-12-08T20:09:27+0000 lvl=info msg=”Deleting container” creation date=2016-12-08T20:09:22+0000 ephemeral=false name=juju-5c77ff-10
    t=2016-12-08T20:09:28+0000 lvl=info msg=”Deleted container” creation date=2016-12-08T20:09:22+0000 ephemeral=false name=juju-5c77ff-10
    t=2016-12-08T20:09:46+0000 lvl=info msg=”Deleting container” creation date=2016-12-08T20:09:42+0000 ephemeral=false name=juju-5c77ff-10
    t=2016-12-08T20:09:46+0000 lvl=info msg=”Deleted container” creation date=2016-12-08T20:09:42+0000 ephemeral=false name=juju-5c77ff-10

    lxc list:
    root@openstack:~# lxc list
    +—————-+———+——————–+——+————+———–+
    | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
    +—————-+———+——————–+——+————+———–+
    | juju-5c77ff-11 | STOPPED | | | PERSISTENT | 0 |
    +—————-+———+——————–+——+————+———–+
    | juju-83da42-0 | RUNNING | 10.182.1.45 (eth0) | | PERSISTENT | 0 |
    +—————-+———+——————–+——+————+———–+

    In the conjure-up screen , all is at “waiting for machine state”

  7. Gaetan QUENTIN says:

    FYI:

    tested again today .
    This guide does not launch conjure-up anymore with updated 16.04 and conjure-up (0.1.2):

    – lxc exec openstack — sudo -u ubuntu -i conjure-up : need spell
    usage: conjure-up [-h] [-d] [-s] [–version] spell
    conjure-up: error: the following arguments are required: spell

    so trying: lxc exec openstack — sudo -u ubuntu -i conjure-up openstack
    (it shows tty error message)
    i choose nova-lxd and then in the “clouds” list: localhost

    -> oops there was a problem with your install
    Reason: flags provided but not defined: –upload-tools

    • Sounds like you forgot to add the two PPAs. Without them you’ll be getting that old conjure-up (0.1.2) instead of up to date (2.x) and same goes for the Juju version.

      • Gaetan QUENTIN says:

        i tested it today again: this time all the lxd containers are deployed immediatly (less than 2 seconds each …) in fact not depoyed at all as you can imagine: it shows immediatly the ‘additional application configuration’ screen.

        I think i have something rotten on my host but i can’t find what it is.

        in openstack container i have these errors: (/var/log/syslog)
        Feb 15 21:37:09 openstack systemd-remount-fs[47]: /bin/mount for / exited with exit status 1.
        Feb 15 21:37:09 openstack systemd-remount-fs[47]: mount: can’t find LABEL=cloudimg-rootfs
        Feb 15 21:37:09 openstack systemd[1]: Failed to reset devices.list on /system.slice/systemd-random-seed.service: Operation not permitted
        Feb 15 21:37:09 openstack systemd[1]: Starting Load/Save Random Seed…
        Feb 15 21:37:09 openstack systemd[1]: Failed to reset devices.list on /system.slice/cloud-init-local.service: Operation not permitted
        Feb 15 21:37:09 openstack systemd[1]: Starting Initial cloud-init job (pre-networking)…
        Feb 15 21:37:09 openstack systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
        Feb 15 21:37:09 openstack systemd[1]: Failed to reset devices.list on /system.slice/systemd-journal-flush.service: Operation not permitted
        Feb 15 21:37:09 openstack systemd[1]: Starting Flush Journal to Persistent Storage…
        Feb 15 21:37:09 openstack systemd[1]: Started Load/Save Random Seed.
        Feb 15 21:37:09 openstack systemd[1]: Started Flush Journal to Persistent Storage.
        Feb 15 21:37:09 openstack systemd[1]: lvm2-lvmetad.socket: Trigger limit hit, refusing further activation.
        Feb 15 21:37:09 openstack systemd[1]: lvm2-lvmetad.socket: Unit entered failed state.
        Feb 15 21:37:09 openstack lvm[48]: Daemon lvmetad returned error 104
        Feb 15 21:37:09 openstack lvm[48]: WARNING: Failed to connect to lvmetad. Falling back to internal scanning.
        Feb 15 21:37:09 openstack systemd[1]: Started Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
        Feb 15 21:37:09 openstack systemd[1]: Reached target Local File Systems (Pre).
        Feb 15 21:37:09 openstack systemd[1]: Reached target Local File Systems.
        Feb 15 21:37:09 openstack systemd[1]: Failed to reset devices.list on /system.slice/plymouth-read-write.service: Operation not permitted
        Feb 15 21:37:09 openstack systemd[1]: Starting Tell Plymouth To Write Out Runtime Data…
        Feb 15 21:37:09 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
        Feb 15 21:37:09 openstack systemd[1]: Starting LSB: AppArmor initialization…
        Feb 15 21:37:09 openstack systemd[1]: Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup.service: Operation not permitted
        Feb 15 21:37:09 openstack systemd[1]: Starting Create Volatile Files and Directories…

        Feb 15 21:37:09 openstack apparmor[71]: /etc/init.d/apparmor: 256: /etc/init.d/apparmor: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
        Feb 15 21:37:09 openstack apparmor[71]: * Not starting AppArmor in container

        [ …·]
        Feb 15 21:38:36 openstack systemd[1]: Failed to reset devices.list on /system.slice/dev-.lxd\x2dmounts.mount: Operation not permitted
        Feb 15 21:38:36 openstack systemd[1]: Failed to reset devices.list on /system.slice/networking.service: Operation not permitted
        Feb 15 21:38:36 openstack systemd[1]: Failed to reset devices.list on /system.slice/proc-diskstats.mount: Operation not permitted

        […]
        Feb 15 22:05:15 openstack lxc-apparmor-load[487]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
        Feb 15 22:05:15 openstack lxc-apparmor-load[487]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
        Feb 15 22:05:15 openstack lxd[492]: lvl=warn msg=”Per-container AppArmor profiles are disabled because the mac_admin capability is missing.” t=2017-02-15T22:05:15+0000
        Feb 15 22:05:15 openstack lxd[492]: lvl=warn msg=”Per-container AppArmor profiles are disabled because LXD is already protected by AppArmor.” t=2017-02-15T22:05:15+0000
        Feb 15 22:05:15 openstack lxd[492]: lvl=warn msg=”CGroup memory swap accounting is disabled, swap limits will be ignored.” t=2017-02-15T22:05:15+0000

        in lxd.log:
        creation date=2017-02-15T22:15:37+0000 ephemeral=false lvl=info msg=”Deleting container” name=juju-4820c9-14 t=2017-02-15T22:15:37+0000
        creation date=2017-02-15T22:15:37+0000 ephemeral=false lvl=info msg=”Deleted container” name=juju-4820c9-14 t=2017-02-15T22:15:37+0000
        ephemeral=false lvl=info msg=”Creating container” name=juju-4820c9-14 t=2017-02-15T22:15:48+0000
        ephemeral=false lvl=info msg=”Created container” name=juju-4820c9-14 t=2017-02-15T22:15:48+0000

        on the first juju container created , there is only this log:

        root@openstack:/var/log/lxd# cat ./juju-f55c5e-0/lxc.log
        lxc 20160215220533.450 WARN lxc_start – start.c:signal_handler:322 – Invalid pid for SIGCHLD. Received pid 593, expected pid 594.

        Where to make bug report?

    • Gaetan QUENTIN says:

      I made a mistake: i have not added the profile with network when launching the container so the update did not work but it didn’t groaned. So it was a bad conjure-up version.

      Sorry for the bad post.

  8. Brian Baird says:

    Thank you for this. Great guide. In the past there were known bug running Docker privileged containers within LXD. If someone wanted to add the docker driver for this to run Docker containers from images within this OpenStack config , would the old privileged Docker container running with LXD container cause a failure ?

    https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1599121
    https://github.com/lxc/lxd/issues/2172

    • Yeah, I’d expect some Docker images that are not meant to run inside unprivileged containers to run into problems in such an environment.

      Depending on how much you care about security, you could just have LXD run privileged containers with apparmor disabled so that a privileged Docker daemon can run in the container, but that’s effectively turning off all the security features that make LXD containers safe to use for random tasks…

  9. Mark Biggers says:

    This was a *very* impressive demo (above post) by Canonical folks at the OpenStack meet-up, in Raleigh NC late last summer (2016).
    However, unlike OSA’s “devstack”, I have seen no info on how to modify this configuration, for Developers — I am not very familiar with Juju. For one, would like to drop Ceph, and modify some of the other service-nodes and such. OSA permits this to be doable, through configs of Ansible playbooks.
    Please let us know, thanks!

    • conjure-up should let you modify your deployment a bit through that text user interface as well as configure some properties of the different services.

      For anything more advanced, you’d need to use Juju directly, either to deploy the whole thing, or after the fact to change your deployment to what you actually want. I’m no expert in either OpenStack or Juju unfortunately, but your best bet would be to reach out to the OpenStack team that maintains the Juju charms for all those components.

Leave a Reply

Your email address will not be published. Required fields are marked *