Faiwatch

• Should work with FAI version(s): Almost everyone
• Tested with following FAI version(s): 2.8.4, 3.1.8, 3.3.2 - 3.4.7
• Operating System(s): Debian GNU/Linux

About

With Faiwatch you can evaluate your (daily) softupdate-logs, to see if everything went well. For example package updates, config updates or if the host has other problems.

A possible output could be:

################
# Faiwatch log #
################

Logdate: Wed Jul 21 18:00:00 CEST 2010

Number of hosts that have run a softupdates: 200
Number of hosts that have not run a softupdates: 1
uhura - last run on Fri Jun 25 17:20:40 CEST 2010

Unknown lines for checkov
----------------------------------------- 
FATAL: Error inserting sd_mod (/lib/modules/2.6.33-686/kernel/drivers/scsi/sd_mod.ko): Unknown symbol in module, or unknown parameter (see dmesg)

Unknown lines for sulu
---------------------------------------------
fcopy: copied /var/lib/fai/config/files/etc/apt/sources.list/sulu to //etc/apt/sources.list

Faiwatch consists of three files:

• faiwatch (the main script - e.g. run it as a daily cronjob and mail the results to your mail address)
• patterns (GNU grep extended regex for things you don't want to know about - maybe you need to to customize the file to fill in your needs, i.e. region or language)
• ignore (hosts you don't want to know about)

Script

faiwatch:

#!/bin/sh
#
# About:                Faiwatch is a script for summarizing softupdate-logs of many hosts.
#
# Copyright:            Systemadministration of the Mathematical Institute of the University of Bonn (2010)
#                       <support@math.uni-bonn.de>
#
# Authors:              Initial release by Mario Domgoergen. 
#                       Modifications by Manuel Hachtkemper.
#
# Options:              -d GNU_COMPATIBLE_DATE allows you to run faiwatch with a previous date.
#
# License:              This program is free software: you can redistribute it and/or modify
#                       it under the terms of the GNU General Public License as published by
#                       the Free Software Foundation, either version 3 of the License, or
#                       (at your option) any later version.
#
#                       This program is distributed in the hope that it will be useful,
#                       but WITHOUT ANY WARRANTY; without even the implied warranty of
#                       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#                       GNU General Public License for more details.
#
#                       You should have received a copy of the GNU General Public License
#                       along with this program.  If not, see <http://www.gnu.org/licenses/>.
#                       (On Debian GNU/Linux systems you will find the license under
#                       /usr/share/common-licenses )


### variables ###

FAI_LOG_DIR="" # set this value, if your logs aren't in /var/log/fai
FAIWATCH_SHARE_DIR="/usr/local/share/faiwatch"
DELETE_OLD_SOFTUPDATE_LOGS_AFTER_N_DAYS="" # set this value if you want to delete old logs


### functions ###

header(){
        header="$1"
        char="${2:--}"
        underline=""
        while [ ${#header} -ne ${#underline} ];do
                underline="$underline$char"
        done
        printf "$header\n$underline\n"
}

box(){
        char="${2:-#}"
        header="$char $1 $char"
        while [ ${#header} -ne ${#underline} ];do
                underline="$underline$char"
        done 
        printf "$underline\n$header\n$underline\n"
}

error_exit(){
        echo error: $*;
        exit 1;
}
        

### test if everything necessary exists ###

# test if FAI_LOG_DIR is set, or set the value to /var/log/fai
FAI_LOG_DIR=${FAI_LOG_DIR:=/var/log/fai}

# test if FAIWATCH_SHARE_DIR is set
if [ -z ${FAIWATCH_SHARE_DIR} ]; then
        error_exit FAIWATCH_SHARE_DIR is not set
fi

# test if file patterns exists and is readable (ignore isn't required)
if ! [ -r ${FAIWATCH_SHARE_DIR}/patterns ]; then
        error_exit ${FAIWATCH_SHARE_DIR}/patterns doesn\'t exist or is not readable
fi


### script ###

# set another date (compatible with GNU date) for faiwatch with the trigger "-d"
while getopts "d:" opt;do
        case $opt in
                d ) date="$OPTARG" ;;
        esac
done

filedate=$(date ${date:+-d "$date"} "+%Y%m%d")

# define hosts with or without update
for path in $(find ${FAI_LOG_DIR} -mindepth 1 -maxdepth 1 -type d ! -name '.*');do
        host=${path#${FAI_LOG_DIR}/}
        host=${host%%/last-softupdate/fai.log}
        if grep "^\(${host}\) *\(#.*\)\?$" ${FAIWATCH_SHARE_DIR}/ignore > /dev/null; then
                continue
        fi
        set -- ${FAI_LOG_DIR}/${host}/softupdate-${filedate}_*
        if [ -d $1 ];then
                hosts_with_softupdate="$hosts_with_softupdate $host"
        else
                hosts_without_softupdate="$hosts_without_softupdate $host"
        fi
done

# print information
printf "\n$(box 'Faiwatch log')\n\n"

printf "Logdate: $(date)\n\n"

set -- $hosts_with_softupdate
printf "Number of hosts that have run a softupdate: $#\n"

set -- $hosts_without_softupdate
printf "Number of hosts that have not run a softupdate: $#\n"

for host in $hosts_without_softupdate;do
        link="$(readlink ${FAI_LOG_DIR%/}/${host}/last-softupdate)"
        if [ -n "$link" ];then
                _date=${link##softupdate-}
                _date=${_date%%_*}
                _time=${link##*_}
                _hour=${_time%????}
                _minute=${_time%??}
                _minute=${_minute#??}
                _second=${_time#????}
                _last_run=$(date -d "$_date $_hour:$_minute:$_second")
                printf "$host - last run on $_last_run\n"
        else
                printf "$host - never run a softupdate before\n"
        fi
done
        

for host in $hosts_with_softupdate;do
        set -- ${FAI_LOG_DIR%/}/${host}/softupdate-${filedate}_*
        fai_log=${1}/fai.log
        unknown_lines=$(grep -E -v -f ${FAIWATCH_SHARE_DIR}/patterns $fai_log)
        if [ -n "$unknown_lines" ];then
                printf "\n"
                header "Unknown lines for $host" "-"
                printf "%s\n" "$unknown_lines"
        fi
        if [ -f "${FAI_LOG_DIR}/${host}/last-softupdate/error.log" ]; then
                cat ${FAI_LOG_DIR}/${host}/last-softupdate/error.log
        fi
done

# if DELETE_OLD_SOFTUPDATE_LOGS_AFTER_N_DAYS has a numeric value delete old logs
if [ "$DELETE_OLD_SOFTUPDATE_LOGS_AFTER_N_DAYS" -eq "$DELETE_OLD_SOFTUPDATE_LOGS_AFTER_N_DAYS" ] 2> /dev/null ; then
        _delete=$(find $FAI_LOG_DIR -type d -name 'softupdate-*' -mtime +$DELETE_OLD_SOFTUPDATE_LOGS_AFTER_N_DAYS)
        if [ -n "$_delete" ]; then
                echo "$_delete" | xargs rm -r
                printf "\n\n$(echo $_delete | wc -w) old logs have been deleted, if removing worked correctly\n"
        fi
fi

patterns example (FAI 3.3.5 + Debian Squeeze):

^[ ]+-+[ ]*$
^[ ]+Fully Automatic Installation  -  FAI$
^[ ]*$
^[ ]+FAI [0-9.]+, [0-9]+ (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) 20[0-9][0-9]   \(c\) 1999-20[0-9][0-9]$
^[ ]+Thomas Lange  <lange@informatik.uni-koeln.de>$
^Using configuration files from .*$
^Calling task_[A-Za-z]+$
^FAI_FLAGS: .*$
^Can't connect to monserver on .* port 4711\. Monitoring disabled\.$
^FAI_CONFIG_SRC is set to .*$
^Updating SVN in /var/lib/fai/config$
^[0-9][0-9]-.*OK.$
^\+\+ .*=.*$
^List of all classes: .*$
^[A-Za-z]+/[0-9][0-9]-.*OK.$
^Skipping Module .*\. It's already loaded\.$
^Calling hook: (instsoft|updatebase).[A-Za-z]+$
^preserving .*$
^[0-9: -]+ URL:(http|ftp)://.*\.(key|asc) \[[0-9]+/[0-9]+\] -> "-" \[1\]$
^OK$
^Executing[ ]+(shell|cfagent): [A-Z_]+/[0-9]+-[a-z_.-]+$
^FAI_ACTION: softupdate$
^Installing software may take a while$
^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) ([1-3]?| )[0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] (CEST|CET) 20[0-9][0-9]$
^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) [1-3]?[0-9]\. (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [0-9][0-9]:[0-9][0-9]:[0-9][0-9] (CEST|CET) 20[0-9][0-9]$
^Performing FAI system update. All data may be overwritten!$
^Source hook: savelog.LAST.source$
^The softupdate took [0-9]+ seconds.$
^[A-Z_]+/[0-9]+-[a-z._-]+[ ]+OK.$
^[a-z]+.[A-Z-]+[ ]+OK.$
^updatebase.DEFAULT   OK.$
^savelog.LAST.source  OK.$
^Save log files via ssh to .*:[a-z]+.*/softupdate-[0-9]+_[0-9]+$

ignore example:

kirk # broken mainboard
scotty
mccoy # test environment