bash: simple script to send alerts if service is down

Here's a simple shell script I cooked up to alert me if my REST service went down for any reason. This was never meant for a production system (where we have nagios and other robust alerting mechanisms) but for an integration/sandbox environment. An external team (subcontractor) was integrating with our REST service deployed on this sandbox and we couldn't afford to have downtime, or the team's productivity would suffer.

It worked like a charm, took just 30 mins to write, test and deploy. Since the external REST service was deployed on a public IP [otherwise how could the external team reach it - we couldn't allow them to VPN in], this script could potentially run from any machine anywhere.

Saving this script here since it was so simple and useful.

#!/bin/bash
#
# sandbox_sanity_check.sh: a simple utility to alert relevant folks if our integration (i.e. sandbox-ext) Trybe Service goes down
# sandbox-ext is in the /etc/hosts file
targetbox=sandbox-ext
# who should we send alert mails to?
recipients="ambar@xyz.com, rakesh@xyz.com, sandeep@xyz.com"
while [ 1 ] 
do
    logfile=sandbox.`date '+%A'`.log

    curl --silent "http://${targetbox}/trybe/v1/config/TEST_067e6162-3b6f.2L_20k_60k?uid=%7B%22aid%22:%22889835751ebf3e49%22%7D&api_key=shared_key&api_nonce=8nk9pbnhacfvgc&api_ts=1333042920376&channel_id=1&api_sig=aba00fdd0058e00111b286c6356f2a70" | grep "trialConfig"

    if [ $? -eq 0 ] 
    then
        echo "[INFO] [`date '+%d_%m_%Y_%H-%M-%S'`] getConfig succeeded " | tee -a ${logfile} ; echo;  
    else
        echo "[ERROR] [`date '+%d_%m_%Y_%H-%M-%S'`] getConfig FAILED... here is the curl output:" | tee -a ${logfile} ; echo; 
        curl -v "http://${targetbox}/trybe/v1/config/TEST_067e6162-3b6f.2L_20k_60k?uid=%7B%22aid%22:%22889835751ebf3e49%22%7D&api_key=shared_key&api_nonce=8nk9pbnhacfvgc&api_ts=1333042920376&channel_id=1&api_sig=aba00fdd0058e00111b286c6356f2a70" >> ${logfile} 2>&1
        echo | tee -a ${logfile};
# send alert email to $recipients using good ol' mutt
        mutt -s "[sandbox checker]: getConfig FAILED" ${recipients}  < /var/local/sandbox.mail.message 
    fi  

    sleep 30
done

does I3P leak memory?

I was suspecting my tomcat6 test app (I3P) was guilty of a memleak.

So I installed munin and munin-node on my box so that I could get pretty memory curves. The default munin setup doesn't plot graphs for specific processes out of the box. I eventually figured out that 'multips_memory' was the plugin for the job.

I was trying to get munin's multips_memory plugin to show me the RSS (Resident Set Size, not the other one :P - see 'man ps') of tomcat6. I wasn't getting any values in munin's multips_memory for "tomcat6" because multips_memory only checks the command name (which isn't 'tomcat' in my case). Tomcat's command line is a huge mess:

/usr/lib/jvm/java-6-sun/bin/java -Djava.util.logging.config.file=/var/lib/tomcat6/conf/logging.properties -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m -Xmx512m -Djava.endorsed.dirs=/usr/share/tomcat6/endorsed -classpath /usr/share/tomcat6/bin/bootstrap.jar -Dcatalina.base=/var/lib/tomcat6 -Dcatalina.home=/usr/share/tomcat6 -Djava.io.tmpdir=/tmp/tomcat6-tmp org.apache.catalina.startup.Bootstrap start 
 
The command name is thus /usr/lib/jvm/java-6-sun/bin/java which doesn't contain the string "tomcat" that I'm interested in. So that's why multips_memory couldn't find tomcat6 on my system... Great, here comes another tweakathon :P

So I went about adjusting the multips_memory code to suit my purposes:

[SNIP]
.
.
.
    ps -eo $monitor,args | gawk '
BEGIN              { total = "U"; } # U = Unknown. 
/grep/             { next; }
/'"$name"'/        { total = total + ($1*1024); }
END                { print "'"$fieldname"'.value", total; }'
done
.
.
.
[/SNIP]

What the above snippet does is match against the entire command+args of the processes against the string of interest using gawk. In other words, I'm now using "ps -eo args" instead of "ps -eo comm" which is necessary to find "tomcat6" in /usr/bin/java's arguments. Also, I changed the gawk regex match to search the entire line (not just $2 since the matching args for my search string could be $3 or $4 etc). I also made the search more inclusive by matching a substring instead of the exact string name  (removed the ^ and $ from the regular expression).

It works like a charm. Munin is now reporting beautiful (and worrying) memory graphs for my selected processes:

And then it struck me that this is pretty handy code. I sometimes need to see how much RAM a particular process is taking.

Every so often I see bad stuff about the tomcat app I'm testing:

4 Nov, 2011 12:58:23 AM com.aaa.bbb.application.modules.AppLauncherModule$1 uncaughtException SEVERE: Uncaught exception from Thread[Timer-0,5,main] java.lang.OutOfMemoryError: Java heap space

in the tomcat logs [/var/log/tomcat6/catalina.out]. The OOM forces a "sudo service tomcat6 restart" - really it must be a memleak.

So here's the result of this effort - a shell script inspired by the multips_memory munin plugin that that tells you how much memory all the instances of Java (e.g.) are consuming. See the usage() below for more details.

#!/bin/bash
#
# meminfo: A simple utility to display the memory usage of given process(es)
#
###############################################################################

# Default values of arguments:
VERBOSE=0
MEMORY_TYPE=rss
ARGS_TYPE=args
THIS_PROGRAM=$(basename $0)



usage()
{
cat << EOF

USAGE: $THIS_PROGRAM [arguments]

SUMMARY: A simple utility to display the memory consumption of given process(es) on this machine.

ARGUMENTS: (all optional)
    -h      Show this message
    -a      The type of arguments specified in 'ps -o'. Can be either 'args' or 'comm' (default: args)
                args: match against the full command name + argument 
                comm: match against the command name only
    -m      Specify memory type (default: rss) - see "man ps"
    -p      The process_string: can be simply a name or a regular expression.
                This argument is optional: if not supplied, all processes are considered.
    -v      Verbose mode: show debugging information

Each line of 'ps -e' is matched against the string using gawk: so you may have to escape special characters like '/' and ':" etc for gawk regex matching.

EXAMPLES:
$THIS_PROGRAM tomcat6
[show tomcat6 RSS memory usage]

$THIS_PROGRAM java
[show total memory usage by all java processes]

$THIS_PROGRAM -m vsz \\/usr\\/bin\\/java.*eclipse.*
[show memory VSZ taken by eclipse]

$THIS_PROGRAM ".usr.bin.java.+eclipse.+"
[simpler version of the above example]

$THIS_PROGRAM "\/usr\/bin\/java -Djava.library.path=\/usr\/lib\/jni -Dosgi.requiredJavaVersion=1.5 -XX:MaxPermSize=256m -Xms40m -Xmx512m -jar \/home\/ambar\/workspace\/tools\/eclipse\/\/plugins\/org.eclipse.equinox.launcher_1.2.0.v20110502.jar"
[very-specific command and args]

EOF
}


while getopts "hvm:a:p:" OPTION
do
    case $OPTION in
        h)
            usage
            exit 0
            ;;
        v)
            VERBOSE=1
            ;;
        a)
            ARGS_TYPE=$OPTARG
            ;;
        m)
            MEMORY_TYPE=$OPTARG
            ;;
        p)
            PROCESS_STRING=$OPTARG
            ;;
        ?)
            usage
            exit 1
            ;;
    esac
done



# another way to set default values, not needed here though
#: ${MEMORY_TYPE:=rss}
#: ${ARGS_TYPE:=args}
#: ${VERBOSE:=0}


if [[ -z $PROCESS_STRING ]]  # if no process name, then override ARGS_TYPE to args so that we calculate the FULL memory usage of all processes
then
    ARGS_TYPE=args
fi

ps -eo $MEMORY_TYPE,$ARGS_TYPE | gawk '
BEGIN                   { total = "U"; } # U = Unknown. 
/'$THIS_PROGRAM'/       { next; }
/grep/                  { next; }
/'"$PROCESS_STRING"'/   { total = total + ($1*1024); if('$VERBOSE'==1) {print "\n\t", $0; print "\tCUMULATIVE USAGE: ", total} }
END                     { mbs = total/(1024*1024); printf("\nTotal '$MEMORY_TYPE' memory used by all '$PROCESS_STRING' processes: %d bytes == %11.3f MB\n", total, mbs); }'

I learnt quite a bit: it's the first time I used getopts, basename, and integrated an awk script (that consumes bash variables) in a shell script. The whole endeavor seemed like a pointless digression at first, but now I think it was totally worth my time :)

And now it's time to show off the results of this little adventure:

[02:43:27] /var/log/tomcat6 $ meminfo -p eclipse -a args -m vsz

Total vsz memory used by all eclipse processes: 5109342208 bytes ==    4872.648 MB
[02:43:36] /var/log/tomcat6 $ meminfo -p tomcat6

Total rss memory used by all tomcat6 processes: 474861568 bytes ==     452.863 MB
[02:43:41] /var/log/tomcat6 $ meminfo 

Total rss memory used by all  processes: 4642054144 bytes ==    4427.008 MB

linux: my usual ~/.bashrc file

This is the typical ~/.bashrc file I use, especially on cygwin/mintty on Windows boxes. It's work-in-progress and keeps evolving.

Note the handy little shell function, tailHelpAlert, that was designed to run on cygwin, on the Windows box where HelpAlert is running. It determines which is the correct (i.e. most recent) HelpAlert log, and then tails it. You don't have to know which build of HelpAlert you're running, or which temp directory the logs are going to right now.

Another handy little utility is the diskhoggers alias which is a cute little bit of nixcraft (works only on RPM-based distros, of course) to determine which packages (i.e. RPMs) are hogging the most disk space. Very handy when you're critically short of HDD real estate and want to remove junk and clutter quickly.

 

# ################
# My Section:
# ################

# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
export HISTSIZE=5000
export HISTFILESIZE=2000
export HISTLENGTH=50000

# don't put duplicate lines in the history. See bash(1) for more options
# ... or force ignoredups and ignorespace
export HISTCONTROL=ignoredups:ignorespace


# http://serverfault.com/questions/72456/stop-bash-tab-completion-from-thinking-i-want-to-cd-into-svn-directories

# Stop bash tab completion from thinking I want to cd into .svn directories
export FIGNORE=svn



alias ls='ls -hF --color=tty'                 # classify files in colour
alias dir='ls --color=auto --format=vertical'
alias vdir='ls --color=auto --format=long'
alias ll='ls -l'                              # long list
alias la='ls -A'                              # all but . and ..
alias l='ls -CF'                              #
# shortcut to see which RPMs are taking the most disk space
alias diskhoggers='rpm -qa --qf "%10{SIZE}\t%{NAME}\n" | sort -n'

# shell prompt
export PS1="[\e[2;33m\u@mintty\e[m \e[0;33m\t\e[m \w] \$ "


#function to tail the correct HA logs automatically
function tailHelpAlert
{

# NOTE: correct the following logs path for your system:
g2aLogsDir=/cygdrive/c/Users/ambars/AppData/Local/Temp/CitrixLogs/GoToAssist/

#find out the correct build number
build_dir=`cd $g2aLogsDir; ls -1t | head -n 1`

cd $g2aLogsDir/$build_dir

targetdir=`ls -1t | head -n 1`

cd $targetdir
tail -f GoToAssist*

}

awk: kill all pimgr jobs in one go

Unfortunately, the /usr/local/pi/pimgr/bin/pimgr-jobs utility doesn't provide a "kill-all" option that automagically kills all the jobs that are currently scheduled to run.

Here's some simple scriptfoo that does the trick rather well:

for i in `gawk --re-interval '$1 ~ /^[0-9]{1,4}/ {print $1}' 1`
do 
./pimgr-jobs --jobid=$i  kill-delete
done

wrt this excellent link, plain old awk doesn't support braces/curlybrackets for denoting the number of occurrences. So you have to use gawk instead, with the '--re-interval' option as shown above. Whoa.

Powered by ScribeFire.

linux: extremely useful BASH scriptfoo for investigating coretraces:

grep -E "(#0 |#1 )" [[FILENAME]]  | grep -vE "(kernel_vsyscall|pthread_cond_timedwait|nanosleep)"
[/name_of_coretrace_file][/filename]

Example:

[root@domain ~]# grep -E "(#0 |#1 )" coretrace_4.2_upgrade_crash | grep -vE "(kernel_vsyscall|pthread_cond_timedwait|nanosleep)"<br />#0  0x0040abc8 in _IO_vfscanf_internal () from /lib/libc.so.6<br />#1  0x0041a451 in vsscanf () from /lib/libc.so.6<br />#1  0x003afdeb in read () from /lib/libpthread.so.0<br />#1  0x003afdeb in read () from /lib/libpthread.so.0<br />#1  0x003afa0e in __lll_mutex_lock_wait () from /lib/libpthread.so.0<br />#0  0x00130244 in _fini () from /lib/libSegFault.so<br />#1  0x0011e5b2 in _dl_fini () from /lib/ld-linux.so.2<br />#1  0x003afdeb in read () from /lib/libpthread.so.0<br />#1  0x003afdeb in read () from /lib/libpthread.so.0<br />#1  0x003afdeb in read () from /lib/libpthread.so.0<br />#0  0x0040f948 in _IO_vfscanf_internal () from /lib/libc.so.6<br />#1  0x0041a451 in vsscanf () from /lib/libc.so.6<br />#1  0x003ea3d6 in kill () from /lib/libc.so.6<br />#1  0x003af365 in sem_timedwait () from /lib/libpthread.so.0<br />

Now you know which are the interesting threads - the ones that are not doing kernel_vsyscall() or pthread_cond_timedwait() or nanosleep() are printed as the output of this command-chain.

linux: creating and using arrays in shell scripts

Bash arrays have numbered indexes only, but they are sparse, i.e. you don't have to define all the indexes.

An entire array can be assigned by enclosing the array items in parenthesis:

arr=(Hello World)

Individual items can be assigned with the familiar array syntax (unless you're used to Basic or Fortran):

arr[0]=Hello
arr[1]=World

But it gets a bit ugly when you want to refer to an array item:

echo ${arr[0]} ${arr[1]}

To quote from the man page:

The braces are required to avoid conflicts with pathname expansion.

In addition the following funky constructs are available:

${arr[*]};  # All of the items in the array
${!arr[*]}; # All of the indexes in the array
${#arr[*]}; # Number of items in the array
${#arr[0]}; # Length of item zero

The ${!arr[*]} is a relatively new addition to bash, it was not part of the original array implementation.

source

linux: rename multiple files with one command

We're going to do this simple task with a simple regex find+replace in sed:

for i in *.avi
do
    j=`echo $i | sed 's/mymovies/myflicks/g'`
    mv "$i" "$j"
done

Can also be written on a single line as:

for i in *.avi; do j=`echo $i | sed 's/find/replace/g'`; mv "$i" "$j"; done

source

linux basics: ${var} vs $var

The "$" character introduces parameter expansion, command substitution, or arithmetic expansion. The parameter name or symbol to be expanded may be enclosed in braces, which are optional but serve to protect the variable to be expanded from characters immediately following it which could be interpreted as part of the name.

The basic form of parameter expansion is "${PARAMETER}". The value of "PARAMETER" is substituted. The braces are required when "PARAMETER" is a positional parameter with more than one digit, or when "PARAMETER" is followed by a character that is not to be interpreted as part of its name.

In other word, as a safety measure, always prefer ${var} over $var in your shell scripts.

source

linux basics: bash shortcuts

alt+f : move forward one word
alt+b : move backwards one word

ctrl+a : takes you to the begining of the line you are currently typing
ctrl+e : takes you to the end of the line you are currently typing
ctrl+b : move backward one character
ctrl+f : move forward one character

ctrl+c : kills the current command or process.
ctrl+d : kills the shell.
ctrl+z : puts the currently running process in background

ctrl+h : deletes one letter at a time from the line you are typing in
ctrl+w : delete word before cursor
ctrl+k : clears the line after the cursor
ctrl+u : clears the line before the cursor
ctrl+l : clear screen



ctrl+r : does a search in the previously given commands so that you don't have to repeat a long command

esc+b : takes you back by one word while typing a command
esc+p : like ctrl+r + lets you search through the previously given commands
esc+. : gives the last command you typed.