This is post 5 out of 10 in the LXC 1.0 blog post series.
Storage backingstores
LXC supports a variety of storage backends (also referred to as backingstore).
It defaults to “none” which simply stores the rootfs under
/var/lib/lxc/<container>/rootfs but you can specify something else to lxc-create or lxc-clone with the -B option.
Currently supported values are:
directory based storage (“none” and “dir)
This is the default backingstore, the container rootfs is stored under
/var/lib/lxc/<container>/rootfs
The --dir option (when using “dir”) can be used to override the path.
btrfs
With this backingstore LXC will setup a new subvolume for the container which makes snapshotting much easier.
lvm
This one will use a new logical volume for the container.
The LV can be set with --lvname (the default is the container name).
The VG can be set with --vgname (the default is “lxc”).
The filesystem can be set with --fstype (the default is “ext4”).
The size can be set with --fssize (the default is “1G”).
You can also use LVM thinpools with --thinpool
overlayfs
This one is mostly used when cloning containers to create a container based on another one and storing any changes in an overlay.
When used with lxc-create it’ll create a container where any change done after its initial creation will be stored in a “delta0” directory next to the container’s rootfs.
zfs
Very similar to btrfs, as I’ve not used either of those myself I can’t say much about them besides that it should also create some kind of subvolume for the container and make snapshots and clones faster and more space efficient.
Standard paths
One quick word with the way LXC usually works and where it’s storing its files:
- /var/lib/lxc (default location for containers)
- /var/lib/lxcsnap (default location for snapshots)
- /var/cache/lxc (default location for the template cache)
- $HOME/.local/share/lxc (default location for unprivileged containers)
- $HOME/.local/share/lxcsnap (default location for unprivileged snapshots)
- $HOME/.cache/lxc (default location for unprivileged template cache)
The default path, also called lxcpath can be overridden on the command line with the -P option or once and for all by setting “lxcpath = /new/path” in /etc/lxc/lxc.conf (or $HOME/.config/lxc/lxc.conf for unprivileged containers).
The snapshot directory is always “snap” appended to lxcpath so it’ll magically follow lxcpath. The template cache is unfortunately hardcoded and can’t easily be moved short of relying on bind-mounts or symlinks.
The default configuration used for all containers at creation time is taken from
/etc/lxc/default.conf (no unprivileged equivalent yet).
The templates themselves are stored in /usr/share/lxc/templates.
Cloning containers
All those backingstores only really shine once you start cloning containers.
For example, let’s take our good old “p1” Ubuntu container and let’s say you want to make a usable copy of it called “p4”, you can simply do:
sudo lxc-clone -o p1 -n p4
And there you go, you’ve got a working “p4” container that’ll be a simple copy of “p1” but with a new mac address and its hostname properly set.
Now let’s say you want to do a quick test against “p1” but don’t want to alter that container itself, yet you don’t want to wait the time needed for a full copy, you can simply do:
sudo lxc-clone -o p1 -n p1-test -B overlayfs -s
And there you go, you’ve got a new “p1-test” container which is entirely based on the “p1” rootfs and where any change will be stored in the “delta0” directory of “p1-test”.
The same “-s” option also works with lvm and btrfs (possibly zfs too) containers and tells lxc-clone to use a snapshot rather than copy the whole rootfs across.
Snapshotting
So cloning is nice and convenient, great for things like development environments where you want throw away containers. But in production, snapshots tend to be a whole lot more useful for things like backup or just before you do possibly risky changes.
In LXC we have a “lxc-snapshot” tool which will let you create, list, restore and destroy snapshots of your containers.
Before I show you how it works, please note that “lxc-snapshot” currently doesn’t appear to work with directory based containers. With those it produces an empty snapshot, this should be fixed by the time LXC 1.0 is actually released.
So, let’s say we want to backup our “p1-lvm” container before installing “apache2” into it, simply run:
echo "before installing apache2" > snap-comment sudo lxc-snapshot -n p1-lvm -c snap-comment
At which point, you can confirm the snapshot was created with:
sudo lxc-snapshot -n p1-lvm -L -C
Now you can go ahead and install “apache2” in the container.
If you want to revert the container at a later point, simply use:
sudo lxc-snapshot -n p1-lvm -r snap0
Or if you want to restore a snapshot as its own container, you can use:
sudo lxc-snapshot -n p1-lvm -r snap0 p1-lvm-snap0
And you’ll get a new “p1-lvm-snap0” container which will contain a working copy of “p1-lvm” as it was at “snap0”.
Github
Twitter
LinkedIn
Mastodon
RSS feed
Good stuff.
Some questions:
– I tried using btrfs as a backing store ~6mo ago, and had a lot of problems with left-over subvolumes that had to be cleaned up by hand? Has this been addressed in any way in 1.0?
– How reliable is overlayfs backing store? I recall a co-worker having some problems with things like unix sockets not working properly?
Found the overlayfs bug, seems unfixed: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1214500
Simon
Using lxc and btrfs does start to create some obvious benefits with cloning or backups of containers and subsequent storage requirements. I’ll be watching for any answers to your question.
brian
Hi Simon,
I’ve been using btrfs as my backing store for the past few months and I have not had any problems like you described.
One thing I will point out however is a lot of documentation/tutorials out there say LXC automatically uses btrfs as the backing store if the /var/lib/lxc directory is formatted with btrfs. This is no longer true.
Now you always have to specify btrfs when you lxc-create a container.
It does however continue to be true that if a container was created with a btrfs backing store then lxc-clone -s will still automatically use btrfs when snapshot cloning.
https://lists.linuxcontainers.org/pipermail/lxc-devel/2013-August/004928.html
Thanks for this series! I actually changed the default container root dir on my box recently as /var/ gets too crammed. It seems in newer versions that option is now called “lxc.lxcpath” instead of “lxcpath”, and neither falls back to the other.
Hi Martin,
So on the upstream side we technically never had a stable release with lxc.conf so we didn’t bother with migration code.
For Ubuntu, my feeling was that as lxc.conf wasn’t documented until last week and that we weren’t shipping one by default, we could get away without migration code, however in light of you and someone else pinging me about this, I’ll reconsider.
Hi,
is there any possibility to backup running container without stopping it? If not how do you do a secure backups and what solution do you use on production evironment? openVZ for eg. has vzdump which allows backups of running containers.
If you’re using a btrfs, zfs or lvm backed container, you can use lxc-snapshot for that. Otherwise, you can simply backup the container’s rootfs at /var/lib/lxc//rootfs with a regular backup tool.
This will however not dump the container’s running state (memory, cpu, …), we need checkpoint/restart for that (CRIU) and hope to have it in LXC 1.1. Work is still needed both on the CRIU and LXC side to make this happen though…
I’m using lvm backed container. And as far as I found lxc-snapshot is not an option here because it requires us to stop container and then run a snapshot. I was hoping for doing security snapshots w/out stopping container.
So as I understand this is not possible now.
Thanks.
JFYI openvz can create backup without stop of containers. But anyway with very short downtime. With vzdump/vzrestore tools.
Thank you for the blog series, really good and helpful to get into lxc!
It seems that in the stable release snapshotting of directory-based containers was not fixed, am I right? Given the output I get.
lxc_container: Snapshot of directory-backed container requested.lxc_container: Making a copy-clone. If you do want snapshots, then
lxc_container: please create an aufs or overlayfs clone first, snapshot that
lxc_container: and keep the original container pristine.
lxc_container: Error creating a snapshot
Hi there,
I faced the same error. Did you run snap-comment>. You must input snapshot description into a file.
Regards,
Tom
reverting snapshot delete ALL?
– Fresh Ubuntu 14.04
– New LVM-Container (Ubuntu 14.04 inside)
lxc-create -n mail -t download -B lvm –vgname mygroup –fssize 10G
After that…
# lxc-snapshot -n mail
Logical volume “snap0” created
# lxc-snapshot -n mail -L -C
snap0 (/containersnaps/mail) 2014:07:24 14:13:35
# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
mail solid-3-0 owi-a-s– 10,00g
snap0 solid-3-0 swi-a-s– 10,00g mail 0,00
# lxc-snapshot -n mail -r snap0
File descriptor 3 (/run/lock/lxc/container/mail) leaked on lvremove invocation. Parent PID 29998: lxc-snapshot
Logical volume “snap0” successfully removed
Logical volume “mail” successfully removed
lxc_container: failed to detect blockdev type for /dev/solid-3-0/snap0
lxc_container: Error copying storage
lxc_container: Error restoring snapshot snap0
# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
ALL GONE!
reverting snapshot delete ALL?
– Fresh Ubuntu 14.04
– New LVM-Container (Ubuntu 14.04 inside)
lxc-create -n mail -t download -B lvm –vgname mygroup –fssize 10G
After that…
# lxc-snapshot -n mail
Logical volume “snap0” created
# lxc-snapshot -n mail -L -C
snap0 (/containersnaps/mail) 2014:07:24 14:13:35
# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
mail solid-3-0 owi-a-s– 10,00g
snap0 solid-3-0 swi-a-s– 10,00g mail 0,00
# lxc-snapshot -n mail -r snap0
File descriptor 3 (/run/lock/lxc/container/mail) leaked on lvremove invocation. Parent PID 29998: lxc-snapshot
Logical volume “snap0” successfully removed
Logical volume “mail” successfully removed
lxc_container: failed to detect blockdev type for /dev/solid-3-0/snap0
lxc_container: Error copying storage
lxc_container: Error restoring snapshot snap0
# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
ALL GONE!
I maybe incorrect, but I believe it the path for changing the default directory in /etc/lxc/lxc.conf or ~.config/lxc/lxc.conf should be “lxc.lxcpath= /new/path” ?
If I use just “lxcpath” it does not seem to work. I am not sure if this is a change or not but I am using ubuntu 14.04.3.
Great write ups by the way.
Hello,
Is there a way to restore a snapshot created via ‘lxc-copy -s –name c1 -N c1_snap’ ? I want to restore c1_snap on to c1.
How to change /home/user/.cache/lxc/downloads path permanently. (to centralize it for various users)
> “lxc-snapshot” currently doesn’t appear to work with directory based containers.
Is it true for current version? I tried for lxc-3.0.3 and failed.
How can I change a root container path for LXC-containers ? Is it possible ?
`lxc.rootfs`?