This is the eleventh blog post in this series about LXD 2.0.
Introduction
First of all, sorry for the delay. It took quite a long time before I finally managed to get all of this going. My first attempts were using devstack which ran into a number of issues that had to be resolved. Yet even after all that, I still wasn’t be able to get networking going properly.
I finally gave up on devstack and tried “conjure-up” to deploy a full Ubuntu OpenStack using Juju in a pretty user friendly way. And it finally worked!
So below is how to run a full OpenStack, using LXD containers instead of VMs and running all of this inside a LXD container (nesting!).
Requirements
This post assumes you’ve got a working LXD setup, providing containers with network access and that you have a pretty beefy CPU, around 50GB of space for the container to use and at least 16GB of RAM.
Remember, we’re running a full OpenStack here, this thing isn’t exactly light!
Setting up the container
OpenStack is made of a lof of different components, doing a lot of different things. Some require some additional privileges so to make our live easier, we’ll use a privileged container.
We’ll configure that container to support nesting, pre-load all the required kernel modules and allow it access to /dev/mem (as is apparently needed).
Please note that this means that most of the security benefit of LXD containers are effectively disabled for that container. However the containers that will be spawned by OpenStack itself will be unprivileged and use all the normal LXD security features.
lxc init ubuntu:16.04 openstack -c security.privileged=true -c security.nesting=true -c "linux.kernel_modules=iptable_nat, ip6table_nat, ebtables, openvswitch, nbd" printf "lxc.cap.drop=\nlxc.aa_profile=unconfined\n" | lxc config set openstack raw.lxc - lxc config device add openstack mem unix-char path=/dev/mem lxc start openstack
Then we need to add a couple of PPAs and install conjure-up, the deployment tool we’ll use to get OpenStack going.
lxc exec openstack -- apt update lxc exec openstack -- apt dist-upgrade -y lxc exec openstack -- apt install squashfuse -y lxc exec openstack -- ln -s /bin/true /usr/local/bin/udevadm lxc exec openstack -- snap install conjure-up --classic
And the last setup step is to configure LXD networking inside the container.
Answer with the default for all questions, except for:
- Use the “dir” storage backend (“zfs” doesn’t work in a nested container)
- Do NOT configure IPv6 networking (conjure-up/juju don’t play well with it)
lxc exec openstack -- lxd init
And that’s it for the container configuration itself, now we can deploy OpenStack!
Deploying OpenStack with conjure-up
As mentioned earlier, we’ll be using conjure-up to deploy OpenStack.
This is a nice, user friendly, tool that interfaces with Juju to deploy complex services.
Start it with:
lxc exec openstack -- sudo -u ubuntu -i conjure-up
- Select “OpenStack with NovaLXD”
- Then select “localhost” as the deployment target (uses LXD)
- And hit “Deploy all remaining applications”
This will now deploy OpenStack. The whole process can take well over an hour depending on what kind of machine you’re running this on. You’ll see all services getting a container allocated, then getting deployed and finally interconnected.
Once the deployment is done, a few post-install steps will appear. This will import some initial images, setup SSH authentication, configure networking and finally giving you the IP address of the dashboard.
Access the dashboard and spawn a container
The dashboard runs inside a container, so you can’t just hit it from your web browser.
The easiest way around this is to setup a NAT rule with:
lxc exec openstack -- iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to <IP>
Where “<ip>” is the dashboard IP address conjure-up gave you at the end of the installation.
You can now grab the IP address of the “openstack” container (from “lxc info openstack”) and point your web browser to: http://<container ip>/horizon
This can take a few minutes to load the first time around. Once the login screen is loaded, enter the default login and password (admin/openstack) and you’ll be greeted by the OpenStack dashboard!
You can now head to the “Project” tab on the left and the “Instances” page. To start a new instance using nova-lxd, click on “Launch instance”, select what image you want, network, … and your instance will get spawned.
Once it’s running, you can assign it a floating IP which will let you reach your instance from within your “openstack” container.
Conclusion
OpenStack is a pretty complex piece of software, it’s also not something you really want to run at home or on a single server. But it’s certainly interesting to be able to do it anyway, keeping everything contained to a single container on your machine.
Conjure-Up is a great tool to deploy such complex software, using Juju behind the scene to drive the deployment, using LXD containers for every individual service and finally for the instances themselves.
It’s also one of the very few cases where multiple level of container nesting actually makes sense!
Extra information
The conjure-up website can be found at: http://conjure-up.io
The Juju website can be found at: http://www.ubuntu.com/cloud/juju
The main LXD website is at: https://linuxcontainers.org/lxd
Development happens on Github at: https://github.com/lxc/lxd
Mailing-list support happens on: https://lists.linuxcontainers.org
IRC support happens in: #lxcontainers on irc.freenode.net
Try LXD online: https://linuxcontainers.org/lxd/try-it
Github
Twitter
LinkedIn
Mastodon
RSS feed
Thank you for this. I had attempted open stack on LXD a few months ago but got stuck! I didn’t know about conjure up then.
I am looking forward to giving this a go!
seems this is a dev/test setup. Not for production .
Absolutely. Someone who would want to run a production setup will want to deploy OpenStack on a bunch of physical servers after doing some research on the exact options they want for storage, network, …
The setup described here is a way to get OpenStack going so you can play around, see how things work and try nova-lxd.
Disagree, openstack on physical servers is pretty lame..
Hi Stéphane Graber,
Thank you for the post. Had lot of information. Will try deploying the openstack with nova-lxd. I am new to LXD, I need help on the following
1. Networking details to setup LXD.
2. I want to test live-migration between remote hosts(here containers). So I want to add multiple nova-compute containers with nova-lxd. How can I add multiple compute nodes(here multiple compute node containers) using conjure-up.
Is it possible to do this in a network without IPv6? I get lots of messages like this.
?: invalid HTTP proxy (http://[fe80::1%eth0]:13128): bad URI
Excellent guide.
I am using devstack within LXD with only a few problems.
I have to ask though.
Why do you pass /dev/mem ?
I don’t actually know 🙂
The reason why I’m passing it to the main container is because Juju through conjure-up attempts to pass it to the nested openstack containers. So not having it in the parent container was causing deployment to fail.
I can’t think of a good reason for them to actually use /dev/mem, so hopefully someone can sort out Juju/charms so that they stop passing it through.
lxc exec openstack — ln -s /bin/true /usr/local/bin/modprobe
error: flag provided but not defined: -s
works better with the ” ‘ ” :
lxc exec openstack — ‘ln -s /bin/true /usr/local/bin/modprobe’
Tried your install on ubuntu 16.10
i have a lot of:
Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup.service: Operation not permitted
Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/proc-swaps.mount: Operation not permitted
Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/conjure-up.service: Operation not permitted
Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/systemd-update-utmp.service: Operation not permitted
Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/cron.service: Operation not permitted
Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/dev-hugepages.mount: Operation not permitted
Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/sys-devices-virtual-net.mount: Operation not permitted
Dec 8 19:35:13 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
Dec 8 19:35:13 openstack systemd[1]: Starting LSB: AppArmor initialization…
Dec 8 19:35:13 openstack apparmor[75]: /etc/init.d/apparmor: 256: /etc/init.d/apparmor: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
Dec 8 19:35:13 openstack apparmor[75]: * Not starting AppArmor in container
Dec 8 19:35:13 openstack apparmor[75]: …done.
Dec 8 19:35:13 openstack systemd[1]: Started LSB: AppArmor initialization.
Dec 8 19:35:13 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
Dec 8 19:38:35 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
Dec 8 19:39:56 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
Dec 8 19:43:05 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
Dec 8 19:43:07 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
Dec 8 19:47:19 openstack lxc-apparmor-load[3900]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
Dec 8 19:47:19 openstack lxc-apparmor-load[3900]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
Dec 8 19:47:19 openstack lxd[3905]: t=2016-12-08T19:47:19+0000 lvl=warn msg=”Per-container AppArmor profiles are disabled because the mac_admin capability is missing.”
Dec 8 19:47:19 openstack lxd[3905]: t=2016-12-08T19:47:19+0000 lvl=warn msg=”Per-container AppArmor profiles are disabled because LXD is already protected by AppArmor.”
Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
Dec 8 19:50:38 openstack lxc-apparmor-load[4242]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
Dec 8 19:50:38 openstack lxc-apparmor-load[4242]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
Dec 8 19:50:38 openstack lxd[4247]: t=2016-12-08T19:50:38+0000 lvl=warn msg=”Per-container AppArmor profiles are disabled because the mac_admin capability is missing.”
Dec 8 19:50:38 openstack lxd[4247]: t=2016-12-08T19:50:38+0000 lvl=warn msg=”Per-container AppArmor profiles are disabled because LXD is already protected by AppArmor.”
And:
Dec 8 19:56:00 openstack conjure-up/openstack-novalxd: Running pre-deploy for OpenStack
Dec 8 19:56:05 openstack conjure-up/openstack-novalxd: Processing lxd profile: conjure-up
Dec 8 19:56:13 openstack conjure-up/openstack-novalxd: Running pre-deploy for OpenStack
Dec 8 19:56:18 openstack conjure-up/openstack-novalxd: Processing lxd profile: conjure-up
Dec 8 20:01:00 openstack conjure-up/openstack-novalxd: Running deploy-done for OpenStack installation.
And in lxd.log:
It looks like it is looping:
t=2016-12-08T20:08:53+0000 lvl=info msg=”Deleted container” creation date=2016-12-08T20:08:47+0000 ephemeral=false name=juju-5c77ff-3
t=2016-12-08T20:09:05+0000 lvl=info msg=”Deleting container” name=juju-5c77ff-10 creation date=2016-12-08T20:08:59+0000 ephemeral=false
t=2016-12-08T20:09:06+0000 lvl=info msg=”Deleted container” name=juju-5c77ff-10 creation date=2016-12-08T20:08:59+0000 ephemeral=false
t=2016-12-08T20:09:27+0000 lvl=info msg=”Deleting container” creation date=2016-12-08T20:09:22+0000 ephemeral=false name=juju-5c77ff-10
t=2016-12-08T20:09:28+0000 lvl=info msg=”Deleted container” creation date=2016-12-08T20:09:22+0000 ephemeral=false name=juju-5c77ff-10
t=2016-12-08T20:09:46+0000 lvl=info msg=”Deleting container” creation date=2016-12-08T20:09:42+0000 ephemeral=false name=juju-5c77ff-10
t=2016-12-08T20:09:46+0000 lvl=info msg=”Deleted container” creation date=2016-12-08T20:09:42+0000 ephemeral=false name=juju-5c77ff-10
lxc list:
root@openstack:~# lxc list
+—————-+———+——————–+——+————+———–+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+—————-+———+——————–+——+————+———–+
| juju-5c77ff-11 | STOPPED | | | PERSISTENT | 0 |
+—————-+———+——————–+——+————+———–+
| juju-83da42-0 | RUNNING | 10.182.1.45 (eth0) | | PERSISTENT | 0 |
+—————-+———+——————–+——+————+———–+
In the conjure-up screen , all is at “waiting for machine state”
FYI:
tested again today .
This guide does not launch conjure-up anymore with updated 16.04 and conjure-up (0.1.2):
– lxc exec openstack — sudo -u ubuntu -i conjure-up : need spell
usage: conjure-up [-h] [-d] [-s] [–version] spell
conjure-up: error: the following arguments are required: spell
so trying: lxc exec openstack — sudo -u ubuntu -i conjure-up openstack
(it shows tty error message)
i choose nova-lxd and then in the “clouds” list: localhost
-> oops there was a problem with your install
Reason: flags provided but not defined: –upload-tools
Sounds like you forgot to add the two PPAs. Without them you’ll be getting that old conjure-up (0.1.2) instead of up to date (2.x) and same goes for the Juju version.
i tested it today again: this time all the lxd containers are deployed immediatly (less than 2 seconds each …) in fact not depoyed at all as you can imagine: it shows immediatly the ‘additional application configuration’ screen.
I think i have something rotten on my host but i can’t find what it is.
in openstack container i have these errors: (/var/log/syslog)
Feb 15 21:37:09 openstack systemd-remount-fs[47]: /bin/mount for / exited with exit status 1.
Feb 15 21:37:09 openstack systemd-remount-fs[47]: mount: can’t find LABEL=cloudimg-rootfs
Feb 15 21:37:09 openstack systemd[1]: Failed to reset devices.list on /system.slice/systemd-random-seed.service: Operation not permitted
Feb 15 21:37:09 openstack systemd[1]: Starting Load/Save Random Seed…
Feb 15 21:37:09 openstack systemd[1]: Failed to reset devices.list on /system.slice/cloud-init-local.service: Operation not permitted
Feb 15 21:37:09 openstack systemd[1]: Starting Initial cloud-init job (pre-networking)…
Feb 15 21:37:09 openstack systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
Feb 15 21:37:09 openstack systemd[1]: Failed to reset devices.list on /system.slice/systemd-journal-flush.service: Operation not permitted
Feb 15 21:37:09 openstack systemd[1]: Starting Flush Journal to Persistent Storage…
Feb 15 21:37:09 openstack systemd[1]: Started Load/Save Random Seed.
Feb 15 21:37:09 openstack systemd[1]: Started Flush Journal to Persistent Storage.
Feb 15 21:37:09 openstack systemd[1]: lvm2-lvmetad.socket: Trigger limit hit, refusing further activation.
Feb 15 21:37:09 openstack systemd[1]: lvm2-lvmetad.socket: Unit entered failed state.
Feb 15 21:37:09 openstack lvm[48]: Daemon lvmetad returned error 104
Feb 15 21:37:09 openstack lvm[48]: WARNING: Failed to connect to lvmetad. Falling back to internal scanning.
Feb 15 21:37:09 openstack systemd[1]: Started Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
Feb 15 21:37:09 openstack systemd[1]: Reached target Local File Systems (Pre).
Feb 15 21:37:09 openstack systemd[1]: Reached target Local File Systems.
Feb 15 21:37:09 openstack systemd[1]: Failed to reset devices.list on /system.slice/plymouth-read-write.service: Operation not permitted
Feb 15 21:37:09 openstack systemd[1]: Starting Tell Plymouth To Write Out Runtime Data…
Feb 15 21:37:09 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
Feb 15 21:37:09 openstack systemd[1]: Starting LSB: AppArmor initialization…
Feb 15 21:37:09 openstack systemd[1]: Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup.service: Operation not permitted
Feb 15 21:37:09 openstack systemd[1]: Starting Create Volatile Files and Directories…
…
Feb 15 21:37:09 openstack apparmor[71]: /etc/init.d/apparmor: 256: /etc/init.d/apparmor: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
Feb 15 21:37:09 openstack apparmor[71]: * Not starting AppArmor in container
[ …·]
Feb 15 21:38:36 openstack systemd[1]: Failed to reset devices.list on /system.slice/dev-.lxd\x2dmounts.mount: Operation not permitted
Feb 15 21:38:36 openstack systemd[1]: Failed to reset devices.list on /system.slice/networking.service: Operation not permitted
Feb 15 21:38:36 openstack systemd[1]: Failed to reset devices.list on /system.slice/proc-diskstats.mount: Operation not permitted
[…]
Feb 15 22:05:15 openstack lxc-apparmor-load[487]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
Feb 15 22:05:15 openstack lxc-apparmor-load[487]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
Feb 15 22:05:15 openstack lxd[492]: lvl=warn msg=”Per-container AppArmor profiles are disabled because the mac_admin capability is missing.” t=2017-02-15T22:05:15+0000
Feb 15 22:05:15 openstack lxd[492]: lvl=warn msg=”Per-container AppArmor profiles are disabled because LXD is already protected by AppArmor.” t=2017-02-15T22:05:15+0000
Feb 15 22:05:15 openstack lxd[492]: lvl=warn msg=”CGroup memory swap accounting is disabled, swap limits will be ignored.” t=2017-02-15T22:05:15+0000
in lxd.log:
creation date=2017-02-15T22:15:37+0000 ephemeral=false lvl=info msg=”Deleting container” name=juju-4820c9-14 t=2017-02-15T22:15:37+0000
creation date=2017-02-15T22:15:37+0000 ephemeral=false lvl=info msg=”Deleted container” name=juju-4820c9-14 t=2017-02-15T22:15:37+0000
ephemeral=false lvl=info msg=”Creating container” name=juju-4820c9-14 t=2017-02-15T22:15:48+0000
ephemeral=false lvl=info msg=”Created container” name=juju-4820c9-14 t=2017-02-15T22:15:48+0000
on the first juju container created , there is only this log:
root@openstack:/var/log/lxd# cat ./juju-f55c5e-0/lxc.log
lxc 20160215220533.450 WARN lxc_start – start.c:signal_handler:322 – Invalid pid for SIGCHLD. Received pid 593, expected pid 594.
Where to make bug report?
I made a mistake: i have not added the profile with network when launching the container so the update did not work but it didn’t groaned. So it was a bad conjure-up version.
Sorry for the bad post.
Thank you for this. Great guide. In the past there were known bug running Docker privileged containers within LXD. If someone wanted to add the docker driver for this to run Docker containers from images within this OpenStack config , would the old privileged Docker container running with LXD container cause a failure ?
https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1599121
https://github.com/lxc/lxd/issues/2172
Yeah, I’d expect some Docker images that are not meant to run inside unprivileged containers to run into problems in such an environment.
Depending on how much you care about security, you could just have LXD run privileged containers with apparmor disabled so that a privileged Docker daemon can run in the container, but that’s effectively turning off all the security features that make LXD containers safe to use for random tasks…
This was a *very* impressive demo (above post) by Canonical folks at the OpenStack meet-up, in Raleigh NC late last summer (2016).
However, unlike OSA’s “devstack”, I have seen no info on how to modify this configuration, for Developers — I am not very familiar with Juju. For one, would like to drop Ceph, and modify some of the other service-nodes and such. OSA permits this to be doable, through configs of Ansible playbooks.
Please let us know, thanks!
conjure-up should let you modify your deployment a bit through that text user interface as well as configure some properties of the different services.
For anything more advanced, you’d need to use Juju directly, either to deploy the whole thing, or after the fact to change your deployment to what you actually want. I’m no expert in either OpenStack or Juju unfortunately, but your best bet would be to reach out to the OpenStack team that maintains the Juju charms for all those components.
Hello,
I tried to follow your tutorial. After lxc exec openstack — sudo -u ubuntu -i conjure-up and choosing localhost I get the error message
Problem running lxd init: error: Unable to talk to LXD: Get http://unix.socket/1.0: dial unix /var/snap/conjure-up/common/lxd/unix.socket: connect: no such file or directory
What might be the problem?
Andy
I’ve the same …Maybe I am misreading it, but there are some PPA mentioned to install, yet I don’t see which PPA’s? Maybe they are missing in the commands list?
Oh and I see it’s already reported : https://github.com/conjure-up/conjure-up/issues/1077
@Andy: facing the same issue
ubuntu@openstack:~$ snap list
Name Version Rev Developer Notes
conjure-up 2.3.1 745 canonical classic
core 16-2.28.1 3017 canonical core
ubuntu@openstack:~$ lxd –version
2.0.10
ubuntu@openstack:~$ which lxd
/usr/bin/lxd
when I install openstack with conjure-up, I choose OpenStack with NovaLXD. And next the localhost choice shows “LXD not found, please install and wait for this message to disappear”. What’s wrong with my installation? different conjure-up version?
I reinstall lxd with snap and apt remove the previous lxd. The conjure-up seems working correctly for a moment and then it suddenly stucks at waiting for a machine. The bottom line shows that “Setting relation glance:shared-db mysql:shared-db”. I find somebody post the same question on conjure-up github(https://github.com/conjure-up/conjure-up/issues/890) without more detail information and solutions.
This guide doesnt work anymore. @Stephane, I think you´re missing to mention the PPA you´re using.
I think you referred to
sudo add-apt-repository ppa:ubuntu-lxc/lxd-stable
sudo apt-get update
sudo apt-get dist-upgrade
Hello Stephane,
I was able to install openstack using conjure-up ( nova-lxd) and getting it up and running. But, I do not see the “volumes”option under compute. I tried installation twice but I can’t see the option. Not sure if this is a bug ’cause I don’t this impacting others. I’m not sure where I am going wrong . I do not see any option to attach volumes to an instance.
Here’s a screen shot
https://launchpadlibrarian.net/348476167/Screenshot%20from%202017-12-08%2003-43-56.png
I was wondering if you could see the “volumes” options in the horizon menu? Been searching for help for the last one week with no success. Any help is sincerely appreciated.
I’m currently trying out a 17.10 version of Ubuntu with all the instructions showed except for the PPA, which lxd is at version 2.21.
After I install the conjure-up from snap, selected Nova-LXD and localhost, all the services show “waiting for machine” and then conjure-up gave an error that the connection has shut down. I will check the log after I run conjure-up a second time. In the future, shouldn’t I run “conjure-down” after the first failure?
Hello! why sometimes, when restarting the PC that has openstack with openstack lxd containers, everything fails, neither the networks nor the instances are shown?