Ubuntu Touch images

For those not yet familiar with this, Ubuntu Touch systems are setup using a read-only root filesystem on top of which writable paths are mounted using bind-mounts from persistent or ephemeral storage.

The default update mechanism is therefore image based. We build new images on our build infrastructure, generate diffs between images and publish the result on the public server.

Each image is made of a bunch of xz compreseed tarballs, the actual number of tarballs may vary, so can their name. At the end of the line, the upgrader simply mounts the partitions and unpacks the tarball in the order it’s given them. It has a list of files to remove and the rest of the files are simply unpacked on top of the existing system.

Delta images only contain the files that are different from the previous image, full images contain them all. Partition images are stored in binary format in a partitions/ directory which the upgrader checks and flashes automatically.

The current list of tarballs we tend to use for the official images are:

• ubuntu: Ubuntu root filesystem (common to all devices)
• device: Device specific data (partition images and Android image)
• custom: Customization tarball (applied on top of the root filesystem in /custom)
• version: Channel/device/build metadata

For more details on how this all works, I’d recommend reading our wiki pages which act as the go-to specification for the server, clients and upgrader.

Running a server

There are a lot of reasons why you may want to run your own system-image server but the main ones seem to be:

• Supporting your own Ubuntu Touch port with over-the-air updates
• Publishing your own customized version of an official image
• QA infrastructure for Ubuntu Touch images
• Using it as an internal buffer/mirror for your devices

Up until now, doing this was pretty tricky as there wasn’t an easy way to import files from the public system-image server into a local one nor was there a simple way to replace the official GPG keys by your own (which would result in your updates to be considered invalid).

This was finally resolved on Friday when I landed the code for a few new file generators in the main system-image server branch.

It’s now reasonably easy to setup your own server, have it mirror some bits from the main public server, swap GPG keys and include your local tarballs.

Before I start with step by step instructions, please note that due to bug 1278589, you need a valid SSL certificate (https) on your server. This may be a problem to some porters who don’t have a separate IP for their server or can’t afford an SSL certificate. We plan on having this resolved in the system-image client soon.

Installing your server

Those instructions have been tried on a clean Ubuntu 13.10 cloud instance, it assumes that you are running them as an “ubuntu” user with “/home/ubuntu” as its home directory.

Install some required packages:

sudo apt-get install -y bzr abootimg android-tools-fsutils 
    python-gnupg fakeroot pxz pep8 pyflakes python-mock apache2

You’ll need a fair amount of available entropy to generate all the keys used by the test suite and production server. If you are doing this for testing only and don’t care much about getting strong keys, you may want to install “haveged” too.

Then setup the web server:

sudo adduser $USER www-data
sudo chgrp www-data /var/www/
sudo chmod g+rwX /var/www/
sudo rm -f /var/www/index.html
newgroups www-data

That being done, now let’s grab the server code, generate some keys and run the testsuite:

bzr branch lp:~ubuntu-system-image/ubuntu-system-image/server system-image
cd system-image
tests/generate-keys
tests/run
cp -R tests/keys/*/ secret/gpg/keys/
bin/generate-keyrings

Now all you need is some configuration. We’ll define a single “test” channel which will contain a single device “mako” (nexus4). It’ll mirror both the ubuntu and device tarball from the main public server (using the trusty-proposed channel over there), repack the device tarball to swap the GPG keys, then download a customization tarball from an http server, stack a keyring tarball (overriding the keys in the ubuntu tarball) and finally generating a version tarball. This channel will contain up to 15 images and will start at image ID “1”.

Doing all this can be done with that bit of configuration (you’ll need to change your server’s FQDN accordingly) in etc/config:

[global]
base_path = /home/ubuntu/system-image/
channels = test
gpg_key_path = secret/gpg/keys/
gpg_keyring_path = secret/gpg/keyrings/
publish_path = /var/www/
state_path = state/
public_fqdn = system-image.test.com
public_http_port = 80
public_https_port = 443

[channel_test]
type = auto
versionbase = 1
fullcount = 15
files = ubuntu, device, custom-savilerow, keyring, version
file_ubuntu = remote-system-image;https://system-image.ubuntu.com;trusty-proposed;ubuntu
file_device = remote-system-image;https://system-image.ubuntu.com;trusty-proposed;device;keyring=archive-master
file_custom-savilerow = http;https://jenkins.qa.ubuntu.com/job/savilerow-trusty/lastSuccessfulBuild/artifact/build/custom.tar.xz;name=custom-savilerow,monitor=https://jenkins.qa.ubuntu.com/job/savilerow-trusty/lastSuccessfulBuild/artifact/build/build_number
file_keyring = keyring;archive-master
file_version = version

Lastly we need to actual create the channel and device in the server, this is done by calling “bin/si-shell” and then doing:

pub.create_channel("test")
pub.create_device("test", "mako")
for keyring in ("archive-master", "image-master", "image-signing", "blacklist"):
    pub.publish_keyring(keyring)

And that’s it! Your server is now ready to use.
To generate your first image, simply run “bin/import-images”.
This will take a while as it’ll need to download files from those external servers, repack some bits but once it’s done, you’ll have a new image published.

You’ll probably want to run that command from cron every few minutes so that whenever any of the referenced files change a new image is generated and published (deltas will also be automatically generated).

To look at the result of the above, I have setup a server here: https://phablet.stgraber.org

To use that server, you’d flash using: phablet-flash ubuntu-system –alternate-server phablet.stgraber.org –channel test