Faiwatch

From FAIWiki
Jump to navigation Jump to search

About

With Faiwatch you can evaluate your (daily) softupdate-logs, to see if everything went well. For example package updates, config updates or if the host has other problems.

A possible output could be:

################
# Faiwatch log #
################

Logdate: Wed Jul 21 18:00:00 CEST 2010

Number of hosts that have run a softupdates: 200
Number of hosts that have not run a softupdates: 1
uhura - last run on Fri Jun 25 17:20:40 CEST 2010

Unknown lines for checkov
----------------------------------------- 
FATAL: Error inserting sd_mod (/lib/modules/2.6.33-686/kernel/drivers/scsi/sd_mod.ko): Unknown symbol in module, or unknown parameter (see dmesg)

Unknown lines for sulu
---------------------------------------------
fcopy: copied /var/lib/fai/config/files/etc/apt/sources.list/sulu to //etc/apt/sources.list

Faiwatch consists of three files:

  • faiwatch (the main script - e.g. run it as a daily cronjob and mail the results to your mail address)
  • patterns (GNU grep extended regex for things you don't want to know about - maybe you need to to customize the file to fill in your needs, i.e. region or language)
  • ignore (hosts you don't want to know about)

Script

faiwatch:

#!/bin/sh
#
# About:		Faiwatch is a script for summarizing softupdate-logs of many hosts.
#
# Copyright:		Systemadministration of the Mathematical Institute of the University of Bonn (2018)
#			<support@math.uni-bonn.de>
#
# Authors: 		Initial release by Mario Domgoergen. 
#			Modifications by
#			- Manuel Hachtkemper <hacman@math.uni-bonn.de>
#			- Christian Meyer <c2h5oh@web.de>
#
# Options:		-d GNU_COMPATIBLE_DATE allows you to run faiwatch with a previous date.
#			-l select last available log in case more than one log exists for a host on a particular day (default: first)
#
# License:		This program is free software: you can redistribute it and/or modify
#			it under the terms of the GNU General Public License as published by
#			the Free Software Foundation, either version 3 of the License, or
#			(at your option) any later version.
#
#			This program is distributed in the hope that it will be useful,
#			but WITHOUT ANY WARRANTY; without even the implied warranty of
#			MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#			GNU General Public License for more details.
#
#			You should have received a copy of the GNU General Public License
#			along with this program.  If not, see <http://www.gnu.org/licenses/>.
#			(On Debian GNU/Linux systems you will find the license under
#			/usr/share/common-licenses )


### variables ###

FAI_LOG_DIR="" # set this value, if your logs aren't in /var/log/fai
FAIWATCH_SHARE_DIR="/usr/local/share/faiwatch"
DELETE_SOFTUPDATE_DAYS="" # set this value if you want to delete old logs


### functions ###

header(){
	header="$1"
	char="${2:--}"
	underline=""
	while [ ${#header} -ne ${#underline} ]; do
		underline="$underline$char"
	done
	printf "$header\n$underline\n"
}

box(){
	char="${2:-#}"
	header="$char $1 $char"
	while [ ${#header} -ne ${#underline} ]; do
                underline="$underline$char"
        done 
	printf "$underline\n$header\n$underline\n"
}

error_exit(){
	printf "error: $*\n" >&2;
	exit 1;
}
	

### test if everything necessary exists ###

# test if FAI_LOG_DIR is set, or set a default value
FAI_LOG_DIR=${FAI_LOG_DIR:=/var/log/fai/remote-logs}

# test if FAIWATCH_SHARE_DIR is set
if [ -z ${FAIWATCH_SHARE_DIR} ]; then
	error_exit FAIWATCH_SHARE_DIR is not set
fi

# test if file patterns exists and is readable (ignore isn't required)
if ! [ -r ${FAIWATCH_SHARE_DIR}/patterns ]; then
	error_exit ${FAIWATCH_SHARE_DIR}/patterns doesn\'t exist or is not readable
fi


### script ###

# select first available log in case more than one log exists for a host on a particular day
print_first=true
# set default date (today)
date="$(date -R)"
# change default date for faiwatch with the trigger "-d" (formats compatible with GNU date are allowed)
while getopts "d:l" opt; do
	case $opt in
		d) date="$OPTARG" ;;
		l) print_first=false ;;
	esac
done

filedate=$(date -d "$date" "+%Y%m%d")

# define hosts with or without update
for path in $(find ${FAI_LOG_DIR} -mindepth 1 -maxdepth 1 -type d ! -name '.*'); do
	host=${path#${FAI_LOG_DIR}/}
	host=${host%%/last-softupdate/fai.log}
	if grep "^\(${host}\) *\(#.*\)\?$" ${FAIWATCH_SHARE_DIR}/ignore > /dev/null; then
		continue
	fi
	set -- ${FAI_LOG_DIR}/${host}/softupdate-${filedate}_*
	if [ -d $1 ];then
		hosts_with_softupdate="$hosts_with_softupdate $host"
	else
		hosts_without_softupdate="$hosts_without_softupdate $host"
	fi
done

# print information
printf "\n$(box 'Faiwatch log')\n\n"

printf "Logdate: %s\n\n" "$(date -d "$date" "+%a, %d. %b %Y")"

set -- $hosts_with_softupdate
printf "Number of hosts that have run a softupdate: $#\n"

set -- $hosts_without_softupdate
printf "Number of hosts that have not run a softupdate: $#\n"

for host in $hosts_without_softupdate; do
	link="$(readlink ${FAI_LOG_DIR%/}/${host}/last-softupdate)"
	if [ -n "$link" ];then
		_date=${link##softupdate-}
		_date=${_date%%_*}
		_time=${link##*_}
		_hour=${_time%????}
		_minute=${_time%??}
		_minute=${_minute#??}
		_second=${_time#????}
		_last_run=$(date -d "$_date $_hour:$_minute:$_second")
		printf "$host - last run on $_last_run\n"
	else
		printf "$host - never run a softupdate before\n"
	fi
done
	

for host in $hosts_with_softupdate; do
	set -- ${FAI_LOG_DIR%/}/${host}/softupdate-${filedate}_*
	if $print_first; then
		fai_host_log="$1"
	else
		# print last
		eval fai_host_log=\"\${$#}\"
	fi
	fai_log="${fai_host_log}/fai.log"
	error_log="${fai_host_log}/error.log"
	unknown_lines=$(grep -E -v -f ${FAIWATCH_SHARE_DIR}/patterns $fai_log)
	if [ -n "$unknown_lines" ];then
		printf "\n"
		header "Unknown lines for $host" "-"
		printf "%s\n" "$unknown_lines"
	fi
	if [ -f "$error_log" ]; then
		cat $error_log
	fi
done

# if DELETE_SOFTUPDATE_DAYS has a numeric value delete old logs
if [ "$DELETE_SOFTUPDATE_DAYS" -eq "$DELETE_SOFTUPDATE_DAYS" ] 2> /dev/null ; then
	_delete=$(find $FAI_LOG_DIR -type d -name 'softupdate-*' -mtime +$DELETE_SOFTUPDATE_DAYS)
	if [ -n "$_delete" ]; then
		echo "$_delete" | xargs rm -r
		printf "\n\n$(echo $_delete | wc -w) old logs have been deleted, if removing worked correctly\n"
	fi
fi

patterns example (FAI 3.3.5 + Debian Squeeze):

^[ ]+-+[ ]*$
^[ ]+Fully Automatic Installation  -  FAI$
^[ ]*$
^[ ]+FAI [0-9.]+, [0-9]+ (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) 20[0-9][0-9]   \(c\) 1999-20[0-9][0-9]$
^[ ]+Thomas Lange  <lange@informatik.uni-koeln.de>$
^Using configuration files from .*$
^Calling task_[A-Za-z]+$
^FAI_FLAGS: .*$
^Can't connect to monserver on .* port 4711\. Monitoring disabled\.$
^FAI_CONFIG_SRC is set to .*$
^Updating SVN in /var/lib/fai/config$
^[0-9][0-9]-.*OK.$
^\+\+ .*=.*$
^List of all classes: .*$
^[A-Za-z]+/[0-9][0-9]-.*OK.$
^Skipping Module .*\. It's already loaded\.$
^Calling hook: (instsoft|updatebase).[A-Za-z]+$
^preserving .*$
^[0-9: -]+ URL:(http|ftp)://.*\.(key|asc) \[[0-9]+/[0-9]+\] -> "-" \[1\]$
^OK$
^Executing[ ]+(shell|cfagent): [A-Z_]+/[0-9]+-[a-z_.-]+$
^FAI_ACTION: softupdate$
^Installing software may take a while$
^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) ([1-3]?| )[0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] (CEST|CET) 20[0-9][0-9]$
^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) [1-3]?[0-9]\. (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [0-9][0-9]:[0-9][0-9]:[0-9][0-9] (CEST|CET) 20[0-9][0-9]$
^Performing FAI system update. All data may be overwritten!$
^Source hook: savelog.LAST.source$
^The softupdate took [0-9]+ seconds.$
^[A-Z_]+/[0-9]+-[a-z._-]+[ ]+OK.$
^[a-z]+.[A-Z-]+[ ]+OK.$
^updatebase.DEFAULT   OK.$
^savelog.LAST.source  OK.$
^Save log files via ssh to .*:[a-z]+.*/softupdate-[0-9]+_[0-9]+$

ignore example:

kirk # broken mainboard
scotty
mccoy # test environment