Faiwatch
Jump to navigation
Jump to search
About
With Faiwatch you can evaluate your (daily) softupdate-logs, to see if everything went well. For example package updates, config updates or if the host has other problems.
A possible output could be:
################ # Faiwatch log # ################ Logdate: Wed Jul 21 18:00:00 CEST 2010 Number of hosts that have run a softupdates: 200 Number of hosts that have not run a softupdates: 1 uhura - last run on Fri Jun 25 17:20:40 CEST 2010 Unknown lines for checkov ----------------------------------------- FATAL: Error inserting sd_mod (/lib/modules/2.6.33-686/kernel/drivers/scsi/sd_mod.ko): Unknown symbol in module, or unknown parameter (see dmesg) Unknown lines for sulu --------------------------------------------- fcopy: copied /var/lib/fai/config/files/etc/apt/sources.list/sulu to //etc/apt/sources.list
Faiwatch consists of three files:
- faiwatch (the main script - e.g. run it as a daily cronjob and mail the results to your mail address)
- patterns (GNU grep extended regex for things you don't want to know about - maybe you need to to customize the file to fill in your needs, i.e. region or language)
- ignore (hosts you don't want to know about)
Script
faiwatch:
#!/bin/sh
#
# About: Faiwatch is a script for summarizing softupdate-logs of many hosts.
#
# Copyright: Systemadministration of the Mathematical Institute of the University of Bonn (2018)
# <support@math.uni-bonn.de>
#
# Authors: Initial release by Mario Domgoergen.
# Modifications by
# - Manuel Hachtkemper <hacman@math.uni-bonn.de>
# - Christian Meyer <c2h5oh@web.de>
#
# Options: -d GNU_COMPATIBLE_DATE allows you to run faiwatch with a previous date.
# -l select last available log in case more than one log exists for a host on a particular day (default: first)
#
# License: This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# (On Debian GNU/Linux systems you will find the license under
# /usr/share/common-licenses )
### variables ###
FAI_LOG_DIR="" # set this value, if your logs aren't in /var/log/fai
FAIWATCH_SHARE_DIR="/usr/local/share/faiwatch"
DELETE_SOFTUPDATE_DAYS="" # set this value if you want to delete old logs
### functions ###
header(){
header="$1"
char="${2:--}"
underline=""
while [ ${#header} -ne ${#underline} ]; do
underline="$underline$char"
done
printf "$header\n$underline\n"
}
box(){
char="${2:-#}"
header="$char $1 $char"
while [ ${#header} -ne ${#underline} ]; do
underline="$underline$char"
done
printf "$underline\n$header\n$underline\n"
}
error_exit(){
printf "error: $*\n" >&2;
exit 1;
}
### test if everything necessary exists ###
# test if FAI_LOG_DIR is set, or set a default value
FAI_LOG_DIR=${FAI_LOG_DIR:=/var/log/fai/remote-logs}
# test if FAIWATCH_SHARE_DIR is set
if [ -z ${FAIWATCH_SHARE_DIR} ]; then
error_exit FAIWATCH_SHARE_DIR is not set
fi
# test if file patterns exists and is readable (ignore isn't required)
if ! [ -r ${FAIWATCH_SHARE_DIR}/patterns ]; then
error_exit ${FAIWATCH_SHARE_DIR}/patterns doesn\'t exist or is not readable
fi
### script ###
# select first available log in case more than one log exists for a host on a particular day
print_first=true
# set default date (today)
date="$(date -R)"
# change default date for faiwatch with the trigger "-d" (formats compatible with GNU date are allowed)
while getopts "d:l" opt; do
case $opt in
d) date="$OPTARG" ;;
l) print_first=false ;;
esac
done
filedate=$(date -d "$date" "+%Y%m%d")
# define hosts with or without update
for path in $(find ${FAI_LOG_DIR} -mindepth 1 -maxdepth 1 -type d ! -name '.*'); do
host=${path#${FAI_LOG_DIR}/}
host=${host%%/last-softupdate/fai.log}
if grep "^\(${host}\) *\(#.*\)\?$" ${FAIWATCH_SHARE_DIR}/ignore > /dev/null; then
continue
fi
set -- ${FAI_LOG_DIR}/${host}/softupdate-${filedate}_*
if [ -d $1 ];then
hosts_with_softupdate="$hosts_with_softupdate $host"
else
hosts_without_softupdate="$hosts_without_softupdate $host"
fi
done
# print information
printf "\n$(box 'Faiwatch log')\n\n"
printf "Logdate: %s\n\n" "$(date -d "$date" "+%a, %d. %b %Y")"
set -- $hosts_with_softupdate
printf "Number of hosts that have run a softupdate: $#\n"
set -- $hosts_without_softupdate
printf "Number of hosts that have not run a softupdate: $#\n"
for host in $hosts_without_softupdate; do
link="$(readlink ${FAI_LOG_DIR%/}/${host}/last-softupdate)"
if [ -n "$link" ];then
_date=${link##softupdate-}
_date=${_date%%_*}
_time=${link##*_}
_hour=${_time%????}
_minute=${_time%??}
_minute=${_minute#??}
_second=${_time#????}
_last_run=$(date -d "$_date $_hour:$_minute:$_second")
printf "$host - last run on $_last_run\n"
else
printf "$host - never run a softupdate before\n"
fi
done
for host in $hosts_with_softupdate; do
set -- ${FAI_LOG_DIR%/}/${host}/softupdate-${filedate}_*
if $print_first; then
fai_host_log="$1"
else
# print last
eval fai_host_log=\"\${$#}\"
fi
fai_log="${fai_host_log}/fai.log"
error_log="${fai_host_log}/error.log"
unknown_lines=$(grep -E -v -f ${FAIWATCH_SHARE_DIR}/patterns $fai_log)
if [ -n "$unknown_lines" ];then
printf "\n"
header "Unknown lines for $host" "-"
printf "%s\n" "$unknown_lines"
fi
if [ -f "$error_log" ]; then
cat $error_log
fi
done
# if DELETE_SOFTUPDATE_DAYS has a numeric value delete old logs
if [ "$DELETE_SOFTUPDATE_DAYS" -eq "$DELETE_SOFTUPDATE_DAYS" ] 2> /dev/null ; then
_delete=$(find $FAI_LOG_DIR -type d -name 'softupdate-*' -mtime +$DELETE_SOFTUPDATE_DAYS)
if [ -n "$_delete" ]; then
echo "$_delete" | xargs rm -r
printf "\n\n$(echo $_delete | wc -w) old logs have been deleted, if removing worked correctly\n"
fi
fi
patterns example (FAI 3.3.5 + Debian Squeeze):
^[ ]+-+[ ]*$ ^[ ]+Fully Automatic Installation - FAI$ ^[ ]*$ ^[ ]+FAI [0-9.]+, [0-9]+ (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) 20[0-9][0-9] \(c\) 1999-20[0-9][0-9]$ ^[ ]+Thomas Lange <lange@informatik.uni-koeln.de>$ ^Using configuration files from .*$ ^Calling task_[A-Za-z]+$ ^FAI_FLAGS: .*$ ^Can't connect to monserver on .* port 4711\. Monitoring disabled\.$ ^FAI_CONFIG_SRC is set to .*$ ^Updating SVN in /var/lib/fai/config$ ^[0-9][0-9]-.*OK.$ ^\+\+ .*=.*$ ^List of all classes: .*$ ^[A-Za-z]+/[0-9][0-9]-.*OK.$ ^Skipping Module .*\. It's already loaded\.$ ^Calling hook: (instsoft|updatebase).[A-Za-z]+$ ^preserving .*$ ^[0-9: -]+ URL:(http|ftp)://.*\.(key|asc) \[[0-9]+/[0-9]+\] -> "-" \[1\]$ ^OK$ ^Executing[ ]+(shell|cfagent): [A-Z_]+/[0-9]+-[a-z_.-]+$ ^FAI_ACTION: softupdate$ ^Installing software may take a while$ ^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) ([1-3]?| )[0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] (CEST|CET) 20[0-9][0-9]$ ^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) [1-3]?[0-9]\. (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [0-9][0-9]:[0-9][0-9]:[0-9][0-9] (CEST|CET) 20[0-9][0-9]$ ^Performing FAI system update. All data may be overwritten!$ ^Source hook: savelog.LAST.source$ ^The softupdate took [0-9]+ seconds.$ ^[A-Z_]+/[0-9]+-[a-z._-]+[ ]+OK.$ ^[a-z]+.[A-Z-]+[ ]+OK.$ ^updatebase.DEFAULT OK.$ ^savelog.LAST.source OK.$ ^Save log files via ssh to .*:[a-z]+.*/softupdate-[0-9]+_[0-9]+$
ignore example:
kirk # broken mainboard scotty mccoy # test environment