Using FAI to set up XEN domains

Introduction

This section provides short guidelines on how to set up XEN base systems (aka domain-0) and virtual machines (dom-U) using FAI. Henning Glawe provided many ideas and some scripts - thanks!

Apart from configuring FAI, the following things need to be considered:

• Which XEN packages should be used?
• XEN kernel images need to be built
• Find a partitioning scheme and figure out whether you want to use LVM, native partitions or files

Preparing FAI:

For the domain-0 only the kernel-image must be set correctly. As I'm using LVM for the unpriveledged domains and FAI can't do that yet, I wrote a little script to set up all of them later (it failed when I tried to do it during the FAI installation, should debug that).

The more interesting part is the preparation of unpriveledged domains. As these require no partitioning during installation a hook must be created:

partition.XENU:

[ -f $LOGDIR/our.skip.partition ] && exit 0

for our_cl in $classes; do
  [ $our_cl = "XENU" ] && continue
  [ -f $FAI/hooks/partition.$our_cl ] && exit 0
done

mke2fs -j /dev/hda1 > /dev/null
mkswap /dev/hda2 > /dev/null
mke2fs -j /dev/hda3 > /dev/null
mke2fs -j /dev/hda4 > /dev/null
mke2fs -j /dev/hda5 > /dev/null

cat > /tmp/fai/fstab <<EOF
/dev/hda1 / ext3 defaults 0 0
/dev/hda2 none swap sw 0 0
/dev/hda3 /tmp ext3 defaults 0 0
/dev/hda4 /usr ext3 defaults 0 0
/dev/hda5 /var ext3 defaults 0 0
EOF

echo "#!" > $diskvar
touch $LOGDIR/skip.partition

Additionally, you need

Booting XEN domains from the network

Starting from the example config provided by xen, xmexample2, I made my scripts support the following:

xm create <domainname>

starts the domain as usual,

xm create <domainname> vminstall=1

starts a FAI installation.

The configuration for my domain named boss is thus (obey spaces, this is python code...)

#============================================================================
# This script sets the parameters used when a domain is created using 'xm create'.
#
# "xm create vminstall=1" will (re)install the domain or
# "xm create" will boot the domain 
#
# vminstall is purely a script variable, and has no effect on the the domain
# id assigned to the new domain.
#============================================================================

# Define script variables here.
# xm_vars is defined automatically, use xm_vars.var() to define a variable.

# This function checks whether 'vminstall' has been given a value.
# It is called automatically by 'xm create'.
def vminstall_check(var, val):
  try:
    val = int(val)
  except: 
    val = 0 
  if val <= 0:
    val = 0 
  return val

# Define the 'vminstall' variable so that 'xm create' knows about it.
xm_vars.var('vminstall',
            use="Reinstall domain if value > 0.",
            check=vminstall_check)

# Check the defined variables have valid values..
xm_vars.check()

kernel = "/boot/xen-linux-2.6.11.12-xen"

memory = 384

name = "boss"

cpu = 0

vif = [ 'mac=aa:00:00:00:00:0d,bridge=xen-br2' ]

disk = [ 'phy:/dev/XENU/boss_,hda1,w',
         'phy:/dev/XENU/bossswap,hda2,w',
         'phy:/dev/XENU/boss_tmp,hda3,w',
         'phy:/dev/XENU/boss_usr,hda4,w',
         'phy:/dev/XENU/boss_var,hda5,w' ]

if vminstall > 0:
  vif = [ 'mac=aa:00:00:aa:aa:aa,bridge=xen-br1' ]
  # Set if you want dhcp to allocate the IP address.
  dhcp = "dhcp"
  # Set netmask.
  #netmask=
  # Set default gateway.
  #gateway=
  # Set the hostname.
  hostname = "boss"

  # Set root device.
  root = "/dev/nfs"
  # provided by dhcp
  # The nfs server.
  #nfs_server = '' 
  # Root directory on the nfs server.
  #nfs_root = '/usr/lib/fai/nfsroot'

  extra = "FAI_ACTION=install FAI_FLAGS=verbose,sshd,createvt,syslogd"
else:
  root = "/dev/hda1 ro"

Well, that's it.

Further hints:

example configspace for xen domU's

• I use this configspace to install Xen DomU's: File:Fai-configspace.tar.gz (it's a mix of simple examples, fai-distributions from people/lazyboy in Subversion, some additions for xen and some personal stuff...

combining FAI with xen-tools

• Here I wrote some things to use xen-tools and FAI together: Xen-tools and FAI softupdates

adjustments needed to the fai nfsroot

When doing network installs, the nfsroot need to be runnable in a Xen domU. That means, modules needed by the kernel you start the domain with must be there (simply install the debian xen kernel into it - this can be added in make-fai-nfsroot.conf), and you should have libc6xen in there, plus some other minor tweaks.

I use a script called tunenfsroot to make some adjustments for installing xen DomU's via network:

#!/bin/sh

prefix=/usr/lib/fai/nfsroot

touch /usr/lib/fai/nfsroot/.nocolorlogo


# most of them stolen from xen-tools scripts...

# von 30-fix-inittab
sed -i -e 's/tty[0-9]$/console/g' -e 's/^\([2-6].*:respawn*\)/#\1/' -e 's/^T/#\t/' ${prefix}/etc/inittab

#  von 10-disable-tls
mv ${prefix}/lib/tls ${prefix}/lib/tls.disabled
mkdir ${prefix}/lib/tls

# von 15-disable-hwclock
chmod -x ${prefix}/etc/init.d/hwclock.sh
chmod -x ${prefix}/etc/init.d/hwclockfirst.sh


# von 55-create-dev
cd ${prefix}/dev
./MAKEDEV generic
./MAKEDEV sd

# only works, if the right xen kernel you also boot the install client with 
# is on the install server.
cp -a /lib/modules/* /usr/lib/fai/nfsroot/lib/modules/

TODO: some of the above script is old and only useful for a sarge nfsroot. In Etch, you'll want to install the packages libc6-xen in the nfsroot and a Xen kernel package which gives you the modules from the Kernel which you start the domU with. And edit the inittab... The packages to install can be added in the DEBOOTSTRAP_OPTS in make-fai-nfsroot.conf. In FAI 3.2beta4 and above there is the file /etc/fai/NFSROOT (might change in future versions) where packages to be installed can be configured.

initrd for netboot

There was a time when it was required to build an extra netboot initrd to do net installs. Nowadays, since Etch, "Normally", the netboot should work with the standard debian initrd, too.

In case you have troubles with this, you might still try this:

To get a netboot-specific initrd, set "MODULES=netboot" in /etc/initramfs-tools/initramfs.conf and create a new initrd with "mkinitramfs" or "update-initramfs". Use this only for installing the domU, otherwise use the normal initrd.

Changes in 3.2.X

Kernel

You need an initrd with unionfs (and maybe live-helper?).

Install the unionfs kernel modules, and rebuild the intrd. Use the new initrd to boot your Xen DomU.

Boot options

For installing Xen Domains via net/nfsroot, the correct kernel options have changed quite a bit. The rootpath is not given from dhcp anymore, and the config should be different - I use this successfully:

extra = "ip=dhcp root=/dev/nfs nfsroot=/srv/fai/nfsroot boot=live FAI_ACTION=install FAI_FLAGS=verbose,sshd,syslogd,createvt debug"

Addiontally, you need to install the packages live-initrd and build unionfs modules for your xen kernel with

m-a a-i unionfs-source