does I3P leak memory?

I was suspecting my tomcat6 test app (I3P) was guilty of a memleak.

So I installed munin and munin-node on my box so that I could get pretty memory curves. The default munin setup doesn't plot graphs for specific processes out of the box. I eventually figured out that 'multips_memory' was the plugin for the job.

I was trying to get munin's multips_memory plugin to show me the RSS (Resident Set Size, not the other one :P - see 'man ps') of tomcat6. I wasn't getting any values in munin's multips_memory for "tomcat6" because multips_memory only checks the command name (which isn't 'tomcat' in my case). Tomcat's command line is a huge mess:

/usr/lib/jvm/java-6-sun/bin/java -Djava.util.logging.config.file=/var/lib/tomcat6/conf/logging.properties -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m -Xmx512m -Djava.endorsed.dirs=/usr/share/tomcat6/endorsed -classpath /usr/share/tomcat6/bin/bootstrap.jar -Dcatalina.base=/var/lib/tomcat6 -Dcatalina.home=/usr/share/tomcat6 -Djava.io.tmpdir=/tmp/tomcat6-tmp org.apache.catalina.startup.Bootstrap start 
 
The command name is thus /usr/lib/jvm/java-6-sun/bin/java which doesn't contain the string "tomcat" that I'm interested in. So that's why multips_memory couldn't find tomcat6 on my system... Great, here comes another tweakathon :P

So I went about adjusting the multips_memory code to suit my purposes:

[SNIP]
.
.
.
    ps -eo $monitor,args | gawk '
BEGIN              { total = "U"; } # U = Unknown. 
/grep/             { next; }
/'"$name"'/        { total = total + ($1*1024); }
END                { print "'"$fieldname"'.value", total; }'
done
.
.
.
[/SNIP]

What the above snippet does is match against the entire command+args of the processes against the string of interest using gawk. In other words, I'm now using "ps -eo args" instead of "ps -eo comm" which is necessary to find "tomcat6" in /usr/bin/java's arguments. Also, I changed the gawk regex match to search the entire line (not just $2 since the matching args for my search string could be $3 or $4 etc). I also made the search more inclusive by matching a substring instead of the exact string name  (removed the ^ and $ from the regular expression).

It works like a charm. Munin is now reporting beautiful (and worrying) memory graphs for my selected processes:

And then it struck me that this is pretty handy code. I sometimes need to see how much RAM a particular process is taking.

Every so often I see bad stuff about the tomcat app I'm testing:

4 Nov, 2011 12:58:23 AM com.aaa.bbb.application.modules.AppLauncherModule$1 uncaughtException SEVERE: Uncaught exception from Thread[Timer-0,5,main] java.lang.OutOfMemoryError: Java heap space

in the tomcat logs [/var/log/tomcat6/catalina.out]. The OOM forces a "sudo service tomcat6 restart" - really it must be a memleak.

So here's the result of this effort - a shell script inspired by the multips_memory munin plugin that that tells you how much memory all the instances of Java (e.g.) are consuming. See the usage() below for more details.

#!/bin/bash
#
# meminfo: A simple utility to display the memory usage of given process(es)
#
###############################################################################

# Default values of arguments:
VERBOSE=0
MEMORY_TYPE=rss
ARGS_TYPE=args
THIS_PROGRAM=$(basename $0)



usage()
{
cat << EOF

USAGE: $THIS_PROGRAM [arguments]

SUMMARY: A simple utility to display the memory consumption of given process(es) on this machine.

ARGUMENTS: (all optional)
    -h      Show this message
    -a      The type of arguments specified in 'ps -o'. Can be either 'args' or 'comm' (default: args)
                args: match against the full command name + argument 
                comm: match against the command name only
    -m      Specify memory type (default: rss) - see "man ps"
    -p      The process_string: can be simply a name or a regular expression.
                This argument is optional: if not supplied, all processes are considered.
    -v      Verbose mode: show debugging information

Each line of 'ps -e' is matched against the string using gawk: so you may have to escape special characters like '/' and ':" etc for gawk regex matching.

EXAMPLES:
$THIS_PROGRAM tomcat6
[show tomcat6 RSS memory usage]

$THIS_PROGRAM java
[show total memory usage by all java processes]

$THIS_PROGRAM -m vsz \\/usr\\/bin\\/java.*eclipse.*
[show memory VSZ taken by eclipse]

$THIS_PROGRAM ".usr.bin.java.+eclipse.+"
[simpler version of the above example]

$THIS_PROGRAM "\/usr\/bin\/java -Djava.library.path=\/usr\/lib\/jni -Dosgi.requiredJavaVersion=1.5 -XX:MaxPermSize=256m -Xms40m -Xmx512m -jar \/home\/ambar\/workspace\/tools\/eclipse\/\/plugins\/org.eclipse.equinox.launcher_1.2.0.v20110502.jar"
[very-specific command and args]

EOF
}


while getopts "hvm:a:p:" OPTION
do
    case $OPTION in
        h)
            usage
            exit 0
            ;;
        v)
            VERBOSE=1
            ;;
        a)
            ARGS_TYPE=$OPTARG
            ;;
        m)
            MEMORY_TYPE=$OPTARG
            ;;
        p)
            PROCESS_STRING=$OPTARG
            ;;
        ?)
            usage
            exit 1
            ;;
    esac
done



# another way to set default values, not needed here though
#: ${MEMORY_TYPE:=rss}
#: ${ARGS_TYPE:=args}
#: ${VERBOSE:=0}


if [[ -z $PROCESS_STRING ]]  # if no process name, then override ARGS_TYPE to args so that we calculate the FULL memory usage of all processes
then
    ARGS_TYPE=args
fi

ps -eo $MEMORY_TYPE,$ARGS_TYPE | gawk '
BEGIN                   { total = "U"; } # U = Unknown. 
/'$THIS_PROGRAM'/       { next; }
/grep/                  { next; }
/'"$PROCESS_STRING"'/   { total = total + ($1*1024); if('$VERBOSE'==1) {print "\n\t", $0; print "\tCUMULATIVE USAGE: ", total} }
END                     { mbs = total/(1024*1024); printf("\nTotal '$MEMORY_TYPE' memory used by all '$PROCESS_STRING' processes: %d bytes == %11.3f MB\n", total, mbs); }'

I learnt quite a bit: it's the first time I used getopts, basename, and integrated an awk script (that consumes bash variables) in a shell script. The whole endeavor seemed like a pointless digression at first, but now I think it was totally worth my time :)

And now it's time to show off the results of this little adventure:

[02:43:27] /var/log/tomcat6 $ meminfo -p eclipse -a args -m vsz

Total vsz memory used by all eclipse processes: 5109342208 bytes ==    4872.648 MB
[02:43:36] /var/log/tomcat6 $ meminfo -p tomcat6

Total rss memory used by all tomcat6 processes: 474861568 bytes ==     452.863 MB
[02:43:41] /var/log/tomcat6 $ meminfo 

Total rss memory used by all  processes: 4642054144 bytes ==    4427.008 MB