Faiwatch: Difference between revisions

From FAIWiki
Jump to navigation Jump to search
m (specify format of patterns)
(Script update: Fixed some bugs regarding time, changed default FAI_LOG_DIR and other small improvements)
 
(4 intermediate revisions by the same user not shown)
Line 1: Line 1:
* '''Should work with FAI version(s):''' Almost everyone
* '''Tested with following FAI version(s):''' 2.8.4, 3.1.8, 3.3.2 - 3.3.5
* '''Operating System(s):''' Debian GNU/Linux
=About=
=About=
With Faiwatch you can evaluate your (daily) softupdate-logs, to see if everything went well. For example package updates, config updates or if the host has other problems.
With Faiwatch you can evaluate your (daily) softupdate-logs, to see if everything went well. For example package updates, config updates or if the host has other problems.
Line 38: Line 34:
#!/bin/sh
#!/bin/sh
#
#
# About:               Faiwatch is a script for summarizing softupdate-logs of many hosts.
# About: Faiwatch is a script for summarizing softupdate-logs of many hosts.
#
#
# Copyright:           Systemadministration of the Mathematical Institute of the University of Bonn (2010)
# Copyright: Systemadministration of the Mathematical Institute of the University of Bonn (2018)
#                       <support@math.uni-bonn.de>
# <support@math.uni-bonn.de>
#
#
# Authors:             Initial release by Mario Domgoergen.
# Authors: Initial release by Mario Domgoergen.  
#                       Modifications by Manuel Hachtkemper.
# Modifications by
# - Manuel Hachtkemper <hacman@math.uni-bonn.de>
# - Christian Meyer <c2h5oh@web.de>
#
#
# Options:             -d GNU_COMPATIBLE_DATE allows you to run faiwatch with a previous date.
# Options: -d GNU_COMPATIBLE_DATE allows you to run faiwatch with a previous date.
# -l select last available log in case more than one log exists for a host on a particular day (default: first)
#
#
# License:             This program is free software: you can redistribute it and/or modify
# License: This program is free software: you can redistribute it and/or modify
#                       it under the terms of the GNU General Public License as published by
# it under the terms of the GNU General Public License as published by
#                       the Free Software Foundation, either version 3 of the License, or
# the Free Software Foundation, either version 3 of the License, or
#                       (at your option) any later version.
# (at your option) any later version.
#
#
#                       This program is distributed in the hope that it will be useful,
# This program is distributed in the hope that it will be useful,
#                       but WITHOUT ANY WARRANTY; without even the implied warranty of
# but WITHOUT ANY WARRANTY; without even the implied warranty of
#                       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#                       GNU General Public License for more details.
# GNU General Public License for more details.
#
#
#                       You should have received a copy of the GNU General Public License
# You should have received a copy of the GNU General Public License
#                       along with this program.  If not, see <http://www.gnu.org/licenses/>.
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
#                       (On Debian GNU/Linux systems you will find the license under
# (On Debian GNU/Linux systems you will find the license under
#                       /usr/share/common-licenses )
# /usr/share/common-licenses )




Line 68: Line 67:
FAI_LOG_DIR="" # set this value, if your logs aren't in /var/log/fai
FAI_LOG_DIR="" # set this value, if your logs aren't in /var/log/fai
FAIWATCH_SHARE_DIR="/usr/local/share/faiwatch"
FAIWATCH_SHARE_DIR="/usr/local/share/faiwatch"
DELETE_OLD_SOFTUPDATE_LOGS_AFTER_N_DAYS="" # set this value if you want to delete old logs
DELETE_SOFTUPDATE_DAYS="" # set this value if you want to delete old logs




Line 74: Line 73:


header(){
header(){
        header="$1"
header="$1"
        char="${2:--}"
char="${2:--}"
        underline=""
underline=""
        while [ ${#header} -ne ${#underline} ];do
while [ ${#header} -ne ${#underline} ]; do
                underline="$underline$char"
underline="$underline$char"
        done
done
        printf "$header\n$underline\n"
printf "$header\n$underline\n"
}
}


box(){
box(){
        char="${2:-#}"
char="${2:-#}"
        header="$char $1 $char"
header="$char $1 $char"
        while [ ${#header} -ne ${#underline} ];do
while [ ${#header} -ne ${#underline} ]; do
                 underline="$underline$char"
                 underline="$underline$char"
         done
         done  
        printf "$underline\n$header\n$underline\n"
printf "$underline\n$header\n$underline\n"
}
}


error_exit(){
error_exit(){
        echo error: $*;
printf "error: $*\n" >&2;
        exit 1;
exit 1;
}
}


### test if everything necessary exists ###


### test if everything necessary exists###
# test if FAI_LOG_DIR is set, or set a default value
 
FAI_LOG_DIR=${FAI_LOG_DIR:=/var/log/fai/remote-logs}
# test if FAI_LOG_DIR is set, or set the value to /var/log/fai
${FAI_LOG_DIR:=/var/log/fai}


# test if FAIWATCH_SHARE_DIR is set
# test if FAIWATCH_SHARE_DIR is set
if [ -z ${FAIWATCH_SHARE_DIR} ]; then
if [ -z ${FAIWATCH_SHARE_DIR} ]; then
        error_exit FAIWATCH_SHARE_DIR is not set
error_exit FAIWATCH_SHARE_DIR is not set
fi
fi


# test if file patterns exists and is readable (ignore isn't required)
# test if file patterns exists and is readable (ignore isn't required)
if ! [ -r ${FAIWATCH_SHARE_DIR}/patterns ]; then
if ! [ -r ${FAIWATCH_SHARE_DIR}/patterns ]; then
        error_exit ${FAIWATCH_SHARE_DIR}/patterns doesn\'t exist or is not readable
error_exit ${FAIWATCH_SHARE_DIR}/patterns doesn\'t exist or is not readable
fi
fi


Line 116: Line 115:
### script ###
### script ###


# set another date (compatible with GNU date) for faiwatch with the trigger "-d"
# select first available log in case more than one log exists for a host on a particular day
while getopts "d:" opt;do
print_first=true
        case $opt in
# set default date (today)
                d ) date="$OPTARG" ;;
date="$(date -R)"
        esac
# change default date for faiwatch with the trigger "-d" (formats compatible with GNU date are allowed)
while getopts "d:l" opt; do
case $opt in
d) date="$OPTARG" ;;
l) print_first=false ;;
esac
done
done


filedate=$(date ${date:+-d "$date"} "+%Y%m%d")
filedate=$(date -d "$date" "+%Y%m%d")


# define hosts with or without update
# define hosts with or without update
for path in $(find ${FAI_LOG_DIR}/*/last-softupdate/fai.log);do
for path in $(find ${FAI_LOG_DIR} -mindepth 1 -maxdepth 1 -type d ! -name '.*'); do
        host=${path#${FAI_LOG_DIR}/}
host=${path#${FAI_LOG_DIR}/}
        host=${host%%/last-softupdate/fai.log}
host=${host%%/last-softupdate/fai.log}
        if grep "^\(${host}\) *\(#.*\)\?$" ${FAIWATCH_SHARE_DIR}/ignore > /dev/null; then
if grep "^\(${host}\) *\(#.*\)\?$" ${FAIWATCH_SHARE_DIR}/ignore > /dev/null; then
                continue
continue
        fi
fi
        set -- ${FAI_LOG_DIR}/${host}/softupdate-${filedate}_*
set -- ${FAI_LOG_DIR}/${host}/softupdate-${filedate}_*
        if [ -d $1 ];then
if [ -d $1 ];then
                hosts_with_softupdate="$hosts_with_softupdate $host"
hosts_with_softupdate="$hosts_with_softupdate $host"
        else
else
                hosts_without_softupdate="$hosts_without_softupdate $host"
hosts_without_softupdate="$hosts_without_softupdate $host"
        fi
fi
done
done


Line 143: Line 147:
printf "\n$(box 'Faiwatch log')\n\n"
printf "\n$(box 'Faiwatch log')\n\n"


printf "Logdate: $(date)\n\n"
printf "Logdate: %s\n\n" "$(date -d "$date" "+%a, %d. %b %Y")"


set -- $hosts_with_softupdate
set -- $hosts_with_softupdate
printf "Number of hosts that have run a softupdates: $#\n"
printf "Number of hosts that have run a softupdate: $#\n"


set -- $hosts_without_softupdate
set -- $hosts_without_softupdate
printf "Number of hosts that have not run a softupdates: $#\n"
printf "Number of hosts that have not run a softupdate: $#\n"


for host in $hosts_without_softupdate;do  
for host in $hosts_without_softupdate; do
        link="$(readlink ${FAI_LOG_DIR%/}/${host}/last-softupdate)"
link="$(readlink ${FAI_LOG_DIR%/}/${host}/last-softupdate)"
        if [ -n "$link" ];then
if [ -n "$link" ];then
                _date=${link##softupdate-}
_date=${link##softupdate-}
                _date=${_date%%_*}
_date=${_date%%_*}
                _time=${link##*_}
_time=${link##*_}
                _hour=${_time%????}
_hour=${_time%????}
                _minute=${_time%??}
_minute=${_time%??}
                _minute=${_minute#??}
_minute=${_minute#??}
                _second=${_time#????}
_second=${_time#????}
                _last_run=$(date -d "$_date $_hour:$_minute:$_second")
_last_run=$(date -d "$_date $_hour:$_minute:$_second")
                printf "$host - last run on $_last_run\n"
printf "$host - last run on $_last_run\n"
        else
else
                printf "$host - never run a softupdate before\n"
printf "$host - never run a softupdate before\n"
        fi  
fi
done
done


 
for host in $hosts_with_softupdate; do
for host in $hosts_with_softupdate;do
set -- ${FAI_LOG_DIR%/}/${host}/softupdate-${filedate}_*
        set -- ${FAI_LOG_DIR%/}/${host}/softupdate-${filedate}_*
if $print_first; then
        fai_log=${1}/fai.log
fai_host_log="$1"
        unknown_lines=$(grep -E -v -f ${FAIWATCH_SHARE_DIR}/patterns $fai_log)
else
        if [ -n "$unknown_lines" ];then
# print last
                printf "\n"  
eval fai_host_log=\"\${$#}\"
                header "Unknown lines for $host" "-"
fi
                printf "%s\n" "$unknown_lines"
fai_log="${fai_host_log}/fai.log"
        fi
error_log="${fai_host_log}/error.log"
unknown_lines=$(grep -E -v -f ${FAIWATCH_SHARE_DIR}/patterns $fai_log)
if [ -n "$unknown_lines" ];then
printf "\n"
header "Unknown lines for $host" "-"
printf "%s\n" "$unknown_lines"
fi
if [ -f "$error_log" ]; then
cat $error_log
fi
done
done


# if DELETE_OLD_SOFTUPDATE_LOGS_AFTER_N_DAYS has a numeric value delete old logs
# if DELETE_SOFTUPDATE_DAYS has a numeric value delete old logs
if [ "$DELETE_OLD_SOFTUPDATE_LOGS_AFTER_N_DAYS" -eq "$DELETE_OLD_SOFTUPDATE_LOGS_AFTER_N_DAYS" ] 2> /dev/null ; then
if [ "$DELETE_SOFTUPDATE_DAYS" -eq "$DELETE_SOFTUPDATE_DAYS" ] 2> /dev/null ; then
        _delete=$(find $FAI_LOG_DIR -type d -name 'softupdate-*' -mtime +$DELETE_OLD_SOFTUPDATE_LOGS_AFTER_N_DAYS)
_delete=$(find $FAI_LOG_DIR -type d -name 'softupdate-*' -mtime +$DELETE_SOFTUPDATE_DAYS)
        if [ -n "$_delete" ]; then
if [ -n "$_delete" ]; then
                printf "\n\n"
echo "$_delete" | xargs rm -r
                echo "$_delete" | xargs rm -R --interactive=never
printf "\n\n$(echo $_delete | wc -w) old logs have been deleted, if removing worked correctly\n"
                printf "$(echo $_delete | wc -w) old logs have been deleted, if removing worked correctly\n"
fi
        fi
fi
fi
</pre>
</pre>


patterns:
patterns example (FAI 3.3.5 + Debian Squeeze):
<pre>
<pre>
^[ ]+-+[ ]*$
^[ ]+-+[ ]*$

Latest revision as of 12:45, 9 December 2018

About

With Faiwatch you can evaluate your (daily) softupdate-logs, to see if everything went well. For example package updates, config updates or if the host has other problems.

A possible output could be: 

################
# Faiwatch log #
################

Logdate: Wed Jul 21 18:00:00 CEST 2010

Number of hosts that have run a softupdates: 200
Number of hosts that have not run a softupdates: 1
uhura - last run on Fri Jun 25 17:20:40 CEST 2010

Unknown lines for checkov
----------------------------------------- 
FATAL: Error inserting sd_mod (/lib/modules/2.6.33-686/kernel/drivers/scsi/sd_mod.ko): Unknown symbol in module, or unknown parameter (see dmesg)

Unknown lines for sulu
---------------------------------------------
fcopy: copied /var/lib/fai/config/files/etc/apt/sources.list/sulu to //etc/apt/sources.list

Faiwatch consists of three files:

  • faiwatch (the main script - e.g. run it as a daily cronjob and mail the results to your mail address)
  • patterns (GNU grep extended regex for things you don't want to know about - maybe you need to to customize the file to fill in your needs, i.e. region or language)
  • ignore (hosts you don't want to know about)

Script

faiwatch: 

#!/bin/sh
#
# About:		Faiwatch is a script for summarizing softupdate-logs of many hosts.
#
# Copyright:		Systemadministration of the Mathematical Institute of the University of Bonn (2018)
#			<support@math.uni-bonn.de>
#
# Authors: 		Initial release by Mario Domgoergen. 
#			Modifications by
#			- Manuel Hachtkemper <hacman@math.uni-bonn.de>
#			- Christian Meyer <c2h5oh@web.de>
#
# Options:		-d GNU_COMPATIBLE_DATE allows you to run faiwatch with a previous date.
#			-l select last available log in case more than one log exists for a host on a particular day (default: first)
#
# License:		This program is free software: you can redistribute it and/or modify
#			it under the terms of the GNU General Public License as published by
#			the Free Software Foundation, either version 3 of the License, or
#			(at your option) any later version.
#
#			This program is distributed in the hope that it will be useful,
#			but WITHOUT ANY WARRANTY; without even the implied warranty of
#			MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#			GNU General Public License for more details.
#
#			You should have received a copy of the GNU General Public License
#			along with this program.  If not, see <http://www.gnu.org/licenses/>.
#			(On Debian GNU/Linux systems you will find the license under
#			/usr/share/common-licenses )


### variables ###

FAI_LOG_DIR="" # set this value, if your logs aren't in /var/log/fai
FAIWATCH_SHARE_DIR="/usr/local/share/faiwatch"
DELETE_SOFTUPDATE_DAYS="" # set this value if you want to delete old logs


### functions ###

header(){
	header="$1"
	char="${2:--}"
	underline=""
	while [ ${#header} -ne ${#underline} ]; do
		underline="$underline$char"
	done
	printf "$header\n$underline\n"
}

box(){
	char="${2:-#}"
	header="$char $1 $char"
	while [ ${#header} -ne ${#underline} ]; do
                underline="$underline$char"
        done 
	printf "$underline\n$header\n$underline\n"
}

error_exit(){
	printf "error: $*\n" >&2;
	exit 1;
}
	

### test if everything necessary exists ###

# test if FAI_LOG_DIR is set, or set a default value
FAI_LOG_DIR=${FAI_LOG_DIR:=/var/log/fai/remote-logs}

# test if FAIWATCH_SHARE_DIR is set
if [ -z ${FAIWATCH_SHARE_DIR} ]; then
	error_exit FAIWATCH_SHARE_DIR is not set
fi

# test if file patterns exists and is readable (ignore isn't required)
if ! [ -r ${FAIWATCH_SHARE_DIR}/patterns ]; then
	error_exit ${FAIWATCH_SHARE_DIR}/patterns doesn\'t exist or is not readable
fi


### script ###

# select first available log in case more than one log exists for a host on a particular day
print_first=true
# set default date (today)
date="$(date -R)"
# change default date for faiwatch with the trigger "-d" (formats compatible with GNU date are allowed)
while getopts "d:l" opt; do
	case $opt in
		d) date="$OPTARG" ;;
		l) print_first=false ;;
	esac
done

filedate=$(date -d "$date" "+%Y%m%d")

# define hosts with or without update
for path in $(find ${FAI_LOG_DIR} -mindepth 1 -maxdepth 1 -type d ! -name '.*'); do
	host=${path#${FAI_LOG_DIR}/}
	host=${host%%/last-softupdate/fai.log}
	if grep "^\(${host}\) *\(#.*\)\?$" ${FAIWATCH_SHARE_DIR}/ignore > /dev/null; then
		continue
	fi
	set -- ${FAI_LOG_DIR}/${host}/softupdate-${filedate}_*
	if [ -d $1 ];then
		hosts_with_softupdate="$hosts_with_softupdate $host"
	else
		hosts_without_softupdate="$hosts_without_softupdate $host"
	fi
done

# print information
printf "\n$(box 'Faiwatch log')\n\n"

printf "Logdate: %s\n\n" "$(date -d "$date" "+%a, %d. %b %Y")"

set -- $hosts_with_softupdate
printf "Number of hosts that have run a softupdate: $#\n"

set -- $hosts_without_softupdate
printf "Number of hosts that have not run a softupdate: $#\n"

for host in $hosts_without_softupdate; do
	link="$(readlink ${FAI_LOG_DIR%/}/${host}/last-softupdate)"
	if [ -n "$link" ];then
		_date=${link##softupdate-}
		_date=${_date%%_*}
		_time=${link##*_}
		_hour=${_time%????}
		_minute=${_time%??}
		_minute=${_minute#??}
		_second=${_time#????}
		_last_run=$(date -d "$_date $_hour:$_minute:$_second")
		printf "$host - last run on $_last_run\n"
	else
		printf "$host - never run a softupdate before\n"
	fi
done
	

for host in $hosts_with_softupdate; do
	set -- ${FAI_LOG_DIR%/}/${host}/softupdate-${filedate}_*
	if $print_first; then
		fai_host_log="$1"
	else
		# print last
		eval fai_host_log=\"\${$#}\"
	fi
	fai_log="${fai_host_log}/fai.log"
	error_log="${fai_host_log}/error.log"
	unknown_lines=$(grep -E -v -f ${FAIWATCH_SHARE_DIR}/patterns $fai_log)
	if [ -n "$unknown_lines" ];then
		printf "\n"
		header "Unknown lines for $host" "-"
		printf "%s\n" "$unknown_lines"
	fi
	if [ -f "$error_log" ]; then
		cat $error_log
	fi
done

# if DELETE_SOFTUPDATE_DAYS has a numeric value delete old logs
if [ "$DELETE_SOFTUPDATE_DAYS" -eq "$DELETE_SOFTUPDATE_DAYS" ] 2> /dev/null ; then
	_delete=$(find $FAI_LOG_DIR -type d -name 'softupdate-*' -mtime +$DELETE_SOFTUPDATE_DAYS)
	if [ -n "$_delete" ]; then
		echo "$_delete" | xargs rm -r
		printf "\n\n$(echo $_delete | wc -w) old logs have been deleted, if removing worked correctly\n"
	fi
fi

patterns example (FAI 3.3.5 + Debian Squeeze): 

^[ ]+-+[ ]*$
^[ ]+Fully Automatic Installation  -  FAI$
^[ ]*$
^[ ]+FAI [0-9.]+, [0-9]+ (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) 20[0-9][0-9]   \(c\) 1999-20[0-9][0-9]$
^[ ]+Thomas Lange  <lange@informatik.uni-koeln.de>$
^Using configuration files from .*$
^Calling task_[A-Za-z]+$
^FAI_FLAGS: .*$
^Can't connect to monserver on .* port 4711\. Monitoring disabled\.$
^FAI_CONFIG_SRC is set to .*$
^Updating SVN in /var/lib/fai/config$
^[0-9][0-9]-.*OK.$
^\+\+ .*=.*$
^List of all classes: .*$
^[A-Za-z]+/[0-9][0-9]-.*OK.$
^Skipping Module .*\. It's already loaded\.$
^Calling hook: (instsoft|updatebase).[A-Za-z]+$
^preserving .*$
^[0-9: -]+ URL:(http|ftp)://.*\.(key|asc) \[[0-9]+/[0-9]+\] -> "-" \[1\]$
^OK$
^Executing[ ]+(shell|cfagent): [A-Z_]+/[0-9]+-[a-z_.-]+$
^FAI_ACTION: softupdate$
^Installing software may take a while$
^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) ([1-3]?| )[0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9] (CEST|CET) 20[0-9][0-9]$
^(Mon|Tue|Wed|Thu|Fri|Sat|Sun) [1-3]?[0-9]\. (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [0-9][0-9]:[0-9][0-9]:[0-9][0-9] (CEST|CET) 20[0-9][0-9]$
^Performing FAI system update. All data may be overwritten!$
^Source hook: savelog.LAST.source$
^The softupdate took [0-9]+ seconds.$
^[A-Z_]+/[0-9]+-[a-z._-]+[ ]+OK.$
^[a-z]+.[A-Z-]+[ ]+OK.$
^updatebase.DEFAULT   OK.$
^savelog.LAST.source  OK.$
^Save log files via ssh to .*:[a-z]+.*/softupdate-[0-9]+_[0-9]+$

ignore example: 

kirk # broken mainboard
scotty
mccoy # test environment
Retrieved from "https://wiki.fai-project.org/index.php?title=Faiwatch&oldid=3550"