I was suspecting my tomcat6 test app (I3P) was guilty of a memleak.
So I installed munin and munin-node on my box so that I could get pretty memory curves. The default munin setup doesn't plot graphs for specific processes out of the box. I eventually figured out that 'multips_memory' was the plugin for the job.
I was trying to get munin's multips_memory plugin to show me the RSS (Resident Set Size, not the other one :P - see 'man ps') of tomcat6. I wasn't getting any values in munin's multips_memory for "tomcat6" because multips_memory only checks the command name (which isn't 'tomcat' in my case). Tomcat's command line is a huge mess:
/usr/lib/jvm/java-6-sun/bin/java -Djava.util.logging.config.file=/var/lib/tomcat6/conf/logging.properties -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m -Xmx512m -Djava.endorsed.dirs=/usr/share/tomcat6/endorsed -classpath /usr/share/tomcat6/bin/bootstrap.jar -Dcatalina.base=/var/lib/tomcat6 -Dcatalina.home=/usr/share/tomcat6 -Djava.io.tmpdir=/tmp/tomcat6-tmp org.apache.catalina.startup.Bootstrap start
The command name is thus
/usr/lib/jvm/java-6-sun/bin/java which doesn't contain the string "tomcat" that I'm interested in. So that's why multips_memory couldn't find tomcat6 on my system... Great, here comes another tweakathon :P
So I went about adjusting the multips_memory code to suit my purposes:
[SNIP]
.
.
.
ps -eo $monitor,args | gawk '
BEGIN { total = "U"; } # U = Unknown.
/grep/ { next; }
/'"$name"'/ { total = total + ($1*1024); }
END { print "'"$fieldname"'.value", total; }'
done
.
.
.
[/SNIP]
What the above snippet does is match against the entire command+args of the processes against the string of interest using gawk. In other words, I'm now using "ps -eo args" instead of "ps -eo comm" which is necessary to find "tomcat6" in
/usr/bin/java's arguments. Also, I changed the gawk regex match to search the entire line (not just $2 since the matching args for my search string could be $3 or $4 etc). I also made the search more inclusive by matching a substring instead of the exact string name (removed the ^ and $ from the regular expression).
It works like a charm. Munin is now reporting beautiful (and worrying) memory graphs for my selected processes:
And then it struck me that this is pretty handy code. I sometimes need to see how much RAM a particular process is taking.
Every so often I see bad stuff about the tomcat app I'm testing:
4 Nov, 2011 12:58:23 AM com.aaa.bbb.application.modules.AppLauncherModule$1 uncaughtException
SEVERE: Uncaught exception from Thread[Timer-0,5,main]
java.lang.OutOfMemoryError: Java heap space
in the tomcat logs [/var/log/tomcat6/catalina.out]. The OOM forces a "sudo service tomcat6 restart" - really it must be a memleak.
So here's the result of this effort - a shell script inspired by the multips_memory munin plugin that that tells you how much memory all the instances of Java (e.g.) are consuming. See the usage() below for more details.
#!/bin/bash
#
# meminfo: A simple utility to display the memory usage of given process(es)
#
###############################################################################
# Default values of arguments:
VERBOSE=0
MEMORY_TYPE=rss
ARGS_TYPE=args
THIS_PROGRAM=$(basename $0)
usage()
{
cat << EOF
USAGE: $THIS_PROGRAM [arguments]
SUMMARY: A simple utility to display the memory consumption of given process(es) on this machine.
ARGUMENTS: (all optional)
-h Show this message
-a The type of arguments specified in 'ps -o'. Can be either 'args' or 'comm' (default: args)
args: match against the full command name + argument
comm: match against the command name only
-m Specify memory type (default: rss) - see "man ps"
-p The process_string: can be simply a name or a regular expression.
This argument is optional: if not supplied, all processes are considered.
-v Verbose mode: show debugging information
Each line of 'ps -e' is matched against the string using gawk: so you may have to escape special characters like '/' and ':" etc for gawk regex matching.
EXAMPLES:
$THIS_PROGRAM tomcat6
[show tomcat6 RSS memory usage]
$THIS_PROGRAM java
[show total memory usage by all java processes]
$THIS_PROGRAM -m vsz \\/usr\\/bin\\/java.*eclipse.*
[show memory VSZ taken by eclipse]
$THIS_PROGRAM ".usr.bin.java.+eclipse.+"
[simpler version of the above example]
$THIS_PROGRAM "\/usr\/bin\/java -Djava.library.path=\/usr\/lib\/jni -Dosgi.requiredJavaVersion=1.5 -XX:MaxPermSize=256m -Xms40m -Xmx512m -jar \/home\/ambar\/workspace\/tools\/eclipse\/\/plugins\/org.eclipse.equinox.launcher_1.2.0.v20110502.jar"
[very-specific command and args]
EOF
}
while getopts "hvm:a:p:" OPTION
do
case $OPTION in
h)
usage
exit 0
;;
v)
VERBOSE=1
;;
a)
ARGS_TYPE=$OPTARG
;;
m)
MEMORY_TYPE=$OPTARG
;;
p)
PROCESS_STRING=$OPTARG
;;
?)
usage
exit 1
;;
esac
done
# another way to set default values, not needed here though
#: ${MEMORY_TYPE:=rss}
#: ${ARGS_TYPE:=args}
#: ${VERBOSE:=0}
if [[ -z $PROCESS_STRING ]] # if no process name, then override ARGS_TYPE to args so that we calculate the FULL memory usage of all processes
then
ARGS_TYPE=args
fi
ps -eo $MEMORY_TYPE,$ARGS_TYPE | gawk '
BEGIN { total = "U"; } # U = Unknown.
/'$THIS_PROGRAM'/ { next; }
/grep/ { next; }
/'"$PROCESS_STRING"'/ { total = total + ($1*1024); if('$VERBOSE'==1) {print "\n\t", $0; print "\tCUMULATIVE USAGE: ", total} }
END { mbs = total/(1024*1024); printf("\nTotal '$MEMORY_TYPE' memory used by all '$PROCESS_STRING' processes: %d bytes == %11.3f MB\n", total, mbs); }'
I learnt quite a bit: it's the first time I used getopts, basename, and integrated an awk script (that consumes bash variables) in a shell script. The whole endeavor seemed like a pointless digression at first, but now I think it was totally worth my time :)
And now it's time to show off the results of this little adventure:
