Helper scripts

FAI-Tools

Jan Jansen sent me an interesting tool that should help with management of the FAI configdir get the file

his mail:

Hi,
As long as there were no responses to this posting, i don't dealed with
this idea (GPL) but if some user would find it usefull that would be
also nice for me because i don't have that much time anymore now to fix
all known (and probably unknown) bugs alone. A problem could also be the
currently, only with shell-options available, documentation.
So i post some hints here:
- if a file doc/<class> exists in your fairep, it will be shown as
class-description in info mode
- new script styles ([0-9][0-9]-*) can cause little errors
- some extravagant string values can cause qouting errors (shell
problem)
- create mode not fully funtional

Installation:
Unzip the archive to a directory and adjust the lib and tmp paths in
file 'ftool' to the right place. Now the script should be funtional.
Keep attention to test it the first times with a copy of your repository
to not destroy your classes/files in case of missusing or unexpected
errors (which hopefully aren't that much).

FAIlint

failint.sh a lint checker for FAI. (detects common mistakes)

FAI development and testing helpers

I created some scripts that help setting up a FAI developemnt environment, and running builds and tests in there. At the moment it's mostly about creating fai-cd's and testing them in a qemu vm. Eventually I will add functions for network installation testing with a real hardware host and a qemu client, as well as testing network install with a qemu hist and a qemu client. I am quite interested if someboy finds them useful, they are in subversion:

http://svn.debian.org/wsvn/fai/people/lazyboy/fai-dev-helpers/

--lazyboy 11:12, 21 Oct 2005 (CEST)

fast fai-cd creation script

when developing a new fai-cd it can happen that you need to create new fai-cd's very often. It gets annoying that you always need to delete fai-mirror and the old image... unless you use a script like this (change your tmp path for image and mirror):

#!/bin/sh

FAI_MIRROR_LOCATION=/data/produktion/tmp/fai-mirror

DATE=`date +%Y-%m-%d_%h-%M-%S`

if [ -z $1 ];then
  CDIMAGE=/data/produktion/tmp/fai-cd_${DATE}.iso
  echo "using default cdimage location $CDIMAGE"
else
  CDIMAGE=$1
fi

if [ -z $FAI_MIRROR_LOCATION ]; then
  echo "error - FAI_MIRROR_LOCATION is empty - exiting"
  exit 1
else
  echo "cleaning up fai-mirror at $FAI_MIRROR_LOCATION"
  rm -r $FAI_MIRROR_LOCATION
fi


mkdir -p $FAI_MIRROR_LOCATION

fai-mirror -a $FAI_MIRROR_LOCATION

CDIMAGE_BACKUP=${CDIMAGE}_bak_date +%Y-%m-%d_%h-%M-%S

if [ -f $CDIMAGE ];then
  echo "moving old cd image to $CDIMAGE_BACKUP"
fi

fai-cd -m $FAI_MIRROR_LOCATION $CDIMAGE

--lazyboy 10:59, 5 Jan 2006 (CET)

turn kernel bootprompt parameters into classes

Every kernel parameter after the -- separator is changed to uppercase and used as a classname.

class/35-bootprompt :

#!/usr/bin/perl
# this will define classes from all kernel parameters that
# appear after a parameter '--', as defined in $separator
use strict;
use warnings;

my $separator = '--';

my $seen;
foreach ( split /\s+/, `cat /proc/cmdline` )
{
  if ( $_ eq $separator )
  {
    $seen = 1;
    next;
  }
  next unless $seen;
  print uc $_, "\n"
    if m/^\w+$/;
}

--Ingo Wichmann / Linuxhotel 17 Jan 2006 (CET)

Define a class for a list of hosts

This makes it easy to maintain lists of hosts that should share a class.

Just write a list into a file $FAI_CONFIG/class/FOO.list and it will define your (arbitrary) class FOO for each host that appears in that file.

Thus, a new class GRONK is created and defined for hosts zapp and whirr by simply creating a file GRONK.list that looks like this

foo
bar

class/55-host-lists :

#!/bin/sh
# checks for an appearance of this host's name in each list and, if it
# finds this host in any of the lists, will print the name of the
# list (without the postfix '.list')
#
for thislist in *.list ; do
	thisclass=$(basename $thislist .list)
	egrep -q "\<$HOSTNAME\>" $thislist && echo $thisclass
done

--sanso 14:27, 30 Aug 2006 (CEST)

dmidetect : define classes based on dmidecode and lspci output

This script is run in the $FAICONFIG/class space to define classes based on information reported by the dmidecode and lspci programs. In particular, we use it to extract the manufacturer and product ID of the machine, which allows us to handle specific hardware types simply. For each hardware model, it may generate one or more classes, from most to least specific. For example, for a generation 5 HP ProLiant DL380 server, it generates the following class list:

PROLIANT DL380 DL380-G5

As it stands, it also looks for QLogic qla2xxx fiberchannel devices, and sets a class appropriately, so if the above server also had one of these cards, the class list printed would be:

PROLIANT DL380 DL380-G5 QLA2XXX

The program is designed to be easily extensible to new hardware types, and to look for arbitrary PCI devices. The version as presented is fairly limited, and understands the following hardware models only:

• Most HP ProLiant servers (but not blades - yet)
• Sun X4?00 servers
• Most IBM Blade servers
• RLX Blade servers
• Some HP Deskpro and EVO desktops

A third thing this script does for us is for Desktop class machines, where we used Dynamic DNS. We don't want to have to harvest MAC addresses, so what we do is use the machine's asset tag number as its hostname. This is just another piece of the dmidecode output, so it's easy to pick up here and then override the hostname FAI was going to use. This means that the procedure for installing a desktop here at Sanger is simply to enter its asset tag number in its BIOS, and then PXE boot it. Nothing else is required.

Here's the script:

#!/usr/bin/perl

# This program detects the type of machine on which it is running using the
# output of lspci and dmidetect, and prints out classes appropriately.

# This tries to be as generic as possible, and sets a number of
# classes for a given host, steadily more specific.  See the section
# for the IBM Blades as an example.

# To add a new host type:
#
# 1) Run the program as it stands with the -d option, which will print out
#    the appropriate sections from the dmidecode output, and also a full
#    lspci listing.
#
# 2) Use the above information to add a case to the general_machine_class
#    function.
#
# 3) pci_card_classes is for defining classes which can apply to multiple
#    machine types.  Currently the only entry is QLA2XXX which is defined
#    if a QLogic QLA2XXX fibrechannel adapter is found

# AUTHOR:  Tim Cutts <tjrc@sanger.ac.uk>

use strict;
use warnings;
use Getopt::Std qw(getopts);

# The default DNS domain for machines
my $default_domain = 'internal.sanger.ac.uk';

my %opt;
getopts('d', \%opt);

die "This program needs to run as root\n"
    unless ($> == 0);

# Gather any classes defined by previous scripts
my @previous_classes = exists($ENV{'classes'}) ?
    split(/\s+/, $ENV{'classes'}) : ();

# Gather information from the machine
my $dmi = &get_dmi_info;
my $pci = &get_pci_info;
my ($vendor, $product, $classes);

# Construct the list of classes
push @$classes,
    @{&general_machine_class($dmi)},
    @{&pci_card_classes($pci)};

# That's the general stuff.  From here on, we work with the
# list of classes already defined, and define other things
# based on what we find.  This is highly site-specific, and
# is here purely to provide examples.

# Nasty special case for Sanger's odd mix of IBM blades
if (class_defined('HS20')) {
    if (class_defined('8678')) {
        if (class_defined('QLA2XXX')) {
            push @$classes, 'IBM8678-fibre';
        } else {
            push @$classes, 'IBM8678-ide';
        }
    } 
}

# If this is a deskpro, we need to override the hostname and domain name
# according to the asset tag from the dmi info and place the
# machine in our Dynamic DNS domain.

if (class_defined('DESKPRO') && -d '/tmp/fai') {
   set_host_name_vars($dmi->{'Chassis Information'}->{'Asset Tag'}->[0],
                      'dynamic.sanger.ac.uk');
}

# Same for SEQPROC ML370 machines
if (class_defined('ML370') && -d '/tmp/fai') {
   set_host_name_vars($dmi->{'Chassis Information'}->{'Asset Tag'}->[0],
                      'dynamic.sanger.ac.uk');
   push @$classes, 'SEQPROC';
}


# Print the class list out
local $, = ' ';
print @$classes, "\n";

exit 0;

######################################################################
#
# general_machine_class
# 
# This function returns a list of hardware classes to which this
# machine type belongs.  These should be of increasing specificity
# from left to right; e.g. "DL380 DL380-G3"

sub general_machine_class {

    my $dmi = shift;

    $vendor  = $dmi->{'System Information'}->{'Manufacturer'}->[0];
    $product = $dmi->{'System Information'}->{'Product Name'}->[0];
    
    printf("Vendor: %s\nProduct: %s\n\n", $vendor, $product) if ($opt{'d'});

    if ($vendor =~ /H(P|ewlett-Packard)/) {
        if ($product =~ /HP Compaq (\S+)/) {
            return [ 'DESKPRO', uc($1) ];
        } elsif ($product =~ /ProLiant ([BDM]L\S+)\s+(G\d)/) {
            return [ 'PROLIANT', $1, "$1-$2" ];
        }
    } elsif ($vendor eq 'Compaq') {
        if ($product =~ /EVO *([\S+]*)/i) {
          return [ 'DESKPRO', 'EVO', "EVO-$1" ];
        }
    } elsif ($vendor eq 'IBM') {
        # Regex for IBM BladeCenter machines
        # Three classes for the different aspects of the model number
        if ($product =~ /(.S[24][01]) -\[(\d{4})([^\]]*)\]-/) {
            return [ $1, $2, $2.$3 ];
        }
    } elsif ($vendor =~ /Sun/) {
        if ($product =~ /Sun Fire (X\d+)/) {
            return [ 'SUNFIRE', $1 ];
        }
    } elsif ($vendor =~ /(RLX)/) {
        return [ $1 ];
    }
    return [];
}

######################################################################
#
# pci_card_classes
#
# This function returns a list of classes defined according to PCI
# devices in the system.  Currently this is only being used to detect
# fibrechannel cards, but could also be used for example to define
# classes depending on graphics adapters present (although FAI does a
# good job of this on its own)

sub pci_card_classes {
    my $pci = shift;
    my %tmp;
    foreach (@$pci) {
        if (/QLA2\d\d\d/) {
            # Found QLogic Fibre Channel controller
            $tmp{'QLA2XXX'} = undef;
        }
    }
    return [ keys %tmp ];
}

######################################################################
#
# The remaining functions are simple information gatherers and should
# not need modification.
#

######################################################################
#
# class_defined:  returns true if this class has already been defined

sub class_defined {
    my $class = shift;
    my @a = (@previous_classes, @$classes);
    return (grep { $_ eq $class } @a)[0];
}


sub get_dmi_info {

    my $dmi = {};

    local *DMIDECODE;

    open DMIDECODE, "dmidecode|" or die "Could not execute dmidecode : $!\n";
    local $/ = "\nHandle ";
    my ($section, $subsection, $data);

    # Throw header away
    <DMIDECODE>;

    while (<DMIDECODE>) {
        s/Handle $//s;
        my @a = split(/\n/);
        shift(@a);
        foreach (@a) {
            if (/^([^\t]+)/) {
                $section = $1;
                next;
            }
            if (/^\t([^\t:]+): ?(.*)/) {
                ($subsection, $data) = ($1, $2);
                if ($data =~ /\S/) {
                    $dmi->{$section}->{$subsection} = [ $data ];
                } else {
                    $dmi->{$section}->{$subsection} = [];
                }
                next;
            }
            if (/^\t\t(.+)/) {
                push @{$dmi->{$section}->{$subsection}}, $1;
                next;
            }
            warn "Unparsed dmidecode line: $_";
        }
    }

    close DMIDECODE;

    return $dmi;
}

sub get_pci_info {
    local *LSPCI;
    local $/ = "\n";
    my $info;
    
    open(LSPCI, "lspci -m|") or die "Could not run lspci : $!";
    @$info = (<LSPCI>);
    close LSPCI;
    print @$info if ($opt{'d'});
    return $info;
}

sub set_host_name_vars {
   my $host = shift;
   my $domain = shift || $default_domain;
   local *O;
   open O, ">>/tmp/fai/additional.var" or
        die "Could not append to /tmp/fai/additional.var: $!\n";
   print O qq{HOSTNAME=$dmi->{'Chassis Information'}->{'Asset Tag'}->[0]\n};
   print O qq{DOMAIN=dynamic.sanger.ac.uk\n};
   close O;
}

--TimCutts

fai-conf-update

Update the FAI config space according to the changes on the running FAI server. produces a shell script so that this tool can be run on any host installed with FAI. the resulting shell script can be used to update the configuration space. For now, only the FAI server config can be updated, as the clients do not have write permission to the FAI config space on the server.

fai-conf-update

#! /bin/sh
# fai-conf-update
# script to update the FAI configspace based on FAI Classes.
#
# ToDo:
# - update of the package config
# - flags to change output: output in patch form.
# - tesing of installscripts with the new config on the running installserver
# 
# FixMe:
# -ugly grep after the find. 
#
# Thanks:
# -fly out to MT for first fixes, MrFai for FAI, h01ger for motivation!

FAI_CONFIGSPACE=/srv/fai/config
CLASSES=`cat /var/log/fai/localhost/last/FAI_CLASSES`
SHELLSCRIPT="/root/update_fai.sh"

if [ -f $SHELLSCRIPT ]; then
    echo "$SHELLSCRIPT exists, aborting."
    exit 1
fi

# Loop through the classes

for i in $CLASSES; do

#    Find the config files, strip off the class name in the end
#    loop through the files

    for j in `find $FAI_CONFIGSPACE/files -type f -name ".svn" -prune -o -name $i -print | grep -v \.svn`; do
	name=`echo $j | sed "s#$FAI_CONFIGSPACE/files##"| sed s/\\\/$i//`

	if [ -f $name ]; then
	    if  ! diff -q $name $j  >& /dev/null ; then
		echo "class $i needs update from $name to $j"
		echo "cp $name $j" >> $SHELLSCRIPT
	    fi
	    if [ ! -f $name ]; then
		echo file "$name is not existing on this machine, removing it from the config space"
		echo "rm $j" >> $SHELLSCRIPT
	    fi
	fi
    done
done

echo "review the results in $SHELLSCRIPT and run it to actually update the FAI config space."

I consider this to be a neat thing together with fai-cd: Install your server with FAI. Run the server. As needs change with time, the config files need to be updated. run fai-cd, the result will be a clone of the old server which is adapted to new hardware with FAI. This script is a starter to do this automatically, see the FixMe and ToDo in the script. Could result in a kind of automatik FAI evolution. ;)

OliverOsburg 20:41, 17 Nov 2006 (CET)

Configure FAI classes depending on IP addresses or a network the host is in

getFAIClassesFromIP.pl does loop up if there is an entry for the given 'ip_address' or a subnet including 'ip_address' in the 'network.conf' file and return the FAI classes specified for the fitting entries as a space separated list to stdout.

For the format of 'network.conf' see the examples file provided.

The script:

#!/usr/bin/perl -w
#
# getFAIClassesFromIP.pl
#
# Get the FAI classes configured for a given IP address out of a given
# network.conf file.
#
# See README and examples network.conf files for further information.
#
#
# Copyright 2006-2007 by Maximilian Wilhelm <max@rfc2324.org>
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License version 2 as 
#   published by the Free Software Foundation.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License
#   along with this program; if not, write to the Free Software Foundation,
#   Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
#
# On Debian GNU/Linux systems, the complete text of the GNU General
# Public License can be found in `/usr/share/common-licenses/GPL'.
#
# Maximilian Wilhelm <max@rfc2324.org>
#  -- Tue, 04 Apr 2006 15:18:20 +0200
#

use strict;
use Net::CIDR;

# Set to 1 if you want to be told about possible badness...
my $debug = 0;

my $network_conf;
my $ip;

my %classes;
my @indexes;
my $i = 0;

my $output_class_string;

if ( scalar( @ARGV ) == 2 ) {
	$network_conf = $ARGV[0];
	$ip = $ARGV[1];

	my $valid_ip = 0;
	if ( Net::CIDR::cidrvalidate( $ip ) ) {
		$valid_ip = 1;
	} else {
		die "Error: Invalid IP $ip provided!\n";
	}

} else {
	print STDERR "Usage: $0 [network.conf] [ip]\n";
	exit 1;
}

if ( -f $network_conf ) {
	open ( NETWORK_CONF, "<$network_conf" )
		or die "Error: could not open network configuration \"$network_conf\"\n";
	while( <NETWORK_CONF> ) {
		if ( m/^#|^$/ ) {
			next;
		}

		# Match for a subnet (network/mask)
		if ( m/^(\d+\.\d+\.\d+\.\d+\/\d+)\s+(.*)$/ ) {
			my @network = ( $1 );
			my @network_classes = split ( /\ /, $2 );
			if ( Net::CIDR::cidrlookup( $ip, @network ) ) {
				print STDERR "Found network \"$network[0]\" fitting to $ip.\n" if ($debug);

				foreach my $network_class ( @network_classes ) {
					$classes{$network_class} = $i++;
					push ( @indexes, "$network_class" );

					print STDERR "saved class $network_class at " . ($i-1) . "\n" if ($debug);
				}
			}
		}

		# Match for s single IP address
		elsif ( m/^(\d+\.\d+\.\d+\.\d+)\s+(.*)$/ ) {

			my $read_ip = $1;
			my @ip_classes = split ( /\ /, $2 );
			chomp $read_ip;

			if ( "$read_ip" eq "$ip" ) {
				print STDERR "Found IP \"$read_ip\" fitting to $ip.\n" if ($debug);

				foreach my $ip_class ( @ip_classes ) {
					$classes{$ip_class} = $i++;
					push ( @indexes, "$ip_class" );

					print STDERR "saved class $ip_class at ". ($i-1) . "\n" if ($debug);
				}
			}
		}

		elsif ( $debug ) {
			print STDERR "Error: Unparseable line \"$_\" in \"${network_conf}\"...\n";
		}		
	}
	close ( NETWORK_CONF );

} else {
	die "Error: Network configuration \"$network_conf\" does not exist.";
}

if ($i != 0) {
	foreach my $n ( values %classes ) {
		$output_class_string .= "$indexes[$n] ";
	}

	print $output_class_string . "\n";
}

1;

The example network.conf

# 
# network.conf
#
# This file is used to defined relations between IP subnets and FAI classes
# (maybe only one, too).
#
# This file is read by the getFAIClassesFromIP.pl script.
# (See http://files.rfc2324.org/misc_tools/FAI)
#
# It is possible to put single IPs or IP subnet declarations in CIDR notation
# into this file and write some class name on the right of it.
#
# ATTENTION: Order *does* matter.
#  Put more important things at the end of the file.
# If for any IP a class is defined twice, the second (more important) occurance
# will be used when building up the class list pushed to FAI.
#
# Format:
# IP			class1 [ class2 [ ...  [ classN ] ] ] 
# network/mask		class1 [ class2 [ ...  [ classN ] ] ] 
#
# Let me show you: (I know, this is a very silly example, but..)

192.168.0.0/24		MY_HOME_NETWORK MY_NETWORK

172.16.0.0/16		COMPANY MY_BIG_COMPANY_NETWORK

192.168.0.1		MY_HOME_ROUTER MY_HOME_NETWORK

The current version can be found in may checked out GIT tree at http://files.rfc2324.org/misc_tools/FAI/getFAIClassesFromIP/

or be using my GIT directly via

git clone http://git.rfc2324.org/repositories/misc_tools.git/

MaximilianWilhelm Thu, 21 Jun 2007 17:19:12 +0200