Setup-storage

Introduction

As setup_harddisks is not portable due to its dependency on sfdisk and the lack of support for RAID and LVM, it has been re-implemented from scratch. However, it is not yet integrated with FAI and deserves rigorous testing. Meanwhile you might also want to look at

• Software RAID hook with mdadm
• SWRaid and LVM hook (with mdadm)
• patched version of setup_harddisks: Code refactoring, LVM/RAID support and POD manual page: Copyright (c) 2003, Sam Vilain (not updated with recent changes in FAI)
• LVM & RAID

To test my current implementation, keep reading this one. There are some TODOs left and probably a lot more will arise as soon as people actually start testing it. Before you start, just let me clarify that this is a very dangerous piece of software that has not been sufficiently tested for any cases where you want to retain any of your data. It will definitely destroy any data on any of your disks in your system. You have been warned.

If you're still interested, the following steps are required next:

• install libparse-recdescent-perl, parted, lvm2 within your NFSROOT

• grab setup_harddisks_2 from the SVN:

[1] (get all the shdd2* files)

• For a first shot copy the files somewhere into your NFSROOT

• create a disk_config for <CLASS> from the old one apart from (at least) the following

changes:

• • the filesystem is now given as the 4th column, the mount options are now in

column 5

• • there is no ;-hack anymore -- anything given after the mount options is

passed on to mkfs.<filesystem>

• • Partitions are marked as bootable using the bootable:<nr> option in the

disk_config line

• • The other things should at least work like they did before, but there are

some new things as well, like RAID support (which is surely incomplete), LVM support, mounting by label or UUID. Please take a look at the attached examples to get an idea of the new format.

All details of the implemented syntax and examples are given below.

• create a hook partition.<CLASS> as follows

#!/bin/sh

cd <somewhere>
debug=1 ./shdd2
# if you really brave, you can get things written to disk as well
debug=1 ./shdd2 -X
# skip the original parititioning
skip partition

• Give it a go and hope... There will be lots of debug output; if it finishes

with the printout of an fstab file that is to your liking, it should have succeeded.

Proposal for a new configuration file syntax

In the following, we present a complete EBNF description of a modified configuration file syntax, as well as some examples.

file ::= <lines> EOF 

lines ::= EOL 
          /* empty lines or whitespace only */
          | <comment> EOL 
          | <config> EOL 

comment ::= #.* 

config ::= disk_config lvm 
           | disk_config raid
           | disk_config end 
           | disk_config disk[[:digit:]]+( <option>)*
           | disk_config [^[:space:]]+( <option>)*
           /* fully qualified device-path or short form, like hda, whereby full
            * path is assumed to be /dev/hda */
           | <volume>

option ::= /* empty */
           | preserve:[[:digit:]]+(,[[:digit:]]+)*
           /* preserve partitions */
           | disklabel:(msdos|sun)
           /* write a disklabel - default is msdos */
           | bootable:[[:digit:]]
           /* mark a partition bootable, default is / */
           | virtual
           /* do not assume the disk to be a physical device, use with xen */

volume ::= <type> <mountpoint> <size> <filesystem> <mount_options> <fs_options>
           | vg <name> <size>
           /* lvm vg */

type ::= primary
         /* for physical disks only */
         | logical
         /* for physical disks only */
         | raid[015]
         /* raid level */
         | [^/[:space:]]+-[^/[:space:]]+
         /* lvm logical volume: vg name and lv name*/

mountpoint ::= -
               /* do not mount */
               | swap
               /* swap space */
               | /[^[:space:]]*
               /* fully qualified path */

name ::= [^/[:space:]]+
         /* lvm volume group name */

size ::= [[:digit:]]+%?(-[[:digit:]]+%?)?(:resize)?
         /* size in megabytes or %, possibly given as a range; physical
          * partitions or lvm logical volumes only */
         | -[[:digit:]]+%?(:resize)?
         /* size in megabytes or % given as upper limit; physical partitions 
          * or lvm logical volumes only */
         | preserve[[:digit:]]+
         /* do not modify this partition */
         | [^,:[:space:]]+(:(spare|missing))*(,[^,:[:space:]]+(:(spare|missing))*)*
         /* devices and options for a raid or lvm vg */

mount_options ::= [^[:space:]]+

filesystem ::= -
               | swap
               | [^[:space:]]
               /* mkfs.xxx must exist */

fs_options ::= .*
               /* options appended to mkfs.xxx call */

The major differences to the prior format:

• the disk_config ... line allows for the keywords lvm and raid
• options may be appended to the disk_config line as well, but this may be subject to change
• the ";" is not used anymore, the options that were given there have now been split up
  • the filesystem is now an explicit parameter; note, that the order of filesystem/mount-options is the same /etc/fstab, as opposed to the previous format of disk_config
  • any options to mkfs.xxx may be given
  • the "boot"-option is one of the options now given on the disk_config line
• support for LVM and RAID is completely new :-)
• resizing partitions is supported

Some examples

# Configure the device /dev/hda
disk_config hda   preserve:6,7   disklabel:msdos  bootable:3
# preserve the 6th and the 7th partition. Alternatively, 
# one could say preserve7 below. The disklabel is msdos, which is the default
# for x86. Furthermore the 3rd partition is made bootable, which would have
# happened as well by default as it is mounted as /
primary /boot     20-100        ext3            rw
# create a primary partition /dev/hda1 with a size between 20 and 100 MB and mount it
# read-write as /boot; it is formatted using ext3 filesystem
primary swap      1000     swap       sw
# /dev/hda2 will be a swap space of 1000 MB
primary /         12000      ext3           rw        -b 2048
# /dev/hda3 should be formatted using ext3 filesystem; when calling mkfs.ext3
# the option "-b 2048" is appended.
logical /tmp      1000      ext3            rw,nosuid
# create the logical partition /dev/hda5
logical /usr      preserve6      ext3          rw
logical /var      10%-      ext3               rw
# make /dev/hda7 at least 10% of the disk size
logical /nobackup 0-        xfs                rw
# use mkfs.xfs to format the partition

# Configure the virtual device /dev/sda (for, e.g., Xen setups), all
# partitions are primary and sizes are ignored (so one could as well specify
# anything other than 0)
disk_config sda virtual
primary /boot     0            ext3                 rw
primary /         0            ext3                 rw
primary /tmp      0            ext3                 rw
primary /usr      0            ext3                 rw
primary /var      0            ext3                 rw

# resizing partitions is possible by appending :resize to any given size
disk_config /dev/scsi/host0/bus0/target1/lun0 
primary /      3000-6000:resize         ext3         rw
# resize to any value between 3000 and 6000 MB, as limited by the disk size,
# the desired sizes of the other partitions, and the data on the partition
primary /tmp   1000                     ext3         rw

# Create a softRAID
disk_config raid
raid1        /    sda1,sdd1  ext2        rw,errors=remount-ro
# create a RAID-1 on /dev/sda1 and /dev/sdd1, format using mkfs.ext2 and mount
# it as /
raid0        -    disk2.2,sdc1,sde1:spare:missing  ext2       default
# create a RAID-0 on the second partition of the second disk, /dev/sdc1, and
# /dev/sde1 as a spare partition, which (may?) me missing

Implementation

Some ideas for the implementation:

• A single function to read partition table for a given physical device and store it in some internal format

 /sbin/parted -s /dev/hda unit B print

might be helpful

• backend to write things to disk, using e.g. parted (parted -s)
• test mode, regression test suite
• command script:
  • init_script
  • make_cmds
    • make_part_cmds
    • make_raid_cmds
    • make_lvm_cmds
  • make_filesystems
• dependencies between LVM/RAID commands -> should be detected by parser/semantic analysis multiple RAID/LVM stanzas, but deps must be satisfied, global RAID-dev counter
• suggestions for modifications of the above format
  • put options on a separate line
  • size specification via size_sda1
  • size specs by RAM size
  • options for lvcreate (e.g. physical volume)
• internal format, generated by parser:
  • list of records: line_no[int], mode[string], type[string], size[string], fs[string], fs_opts[string]
  • map to store fstab-info: dev(unique), mount_point(unique), fs, mount_opts

A parser for the EBNF above is available in the svn, /people/michael/features/setup_harddisks_2