how to add multiple ip addresses to loopback and ping an AVD's host machine

From my Android Virtual Device I was trying to figure out how to connect to a service running on the same machine running the emulator.

For example, let's say I'm running Apache on my laptop, and also running the AVD on my laptop. In a browser on my laptop I can type "http://localhost/index.html" to test the index.html present in my Apache webroot.

But if I put "http://localhost/index.html" in the Android browser in the AVD, I don't get anything. That's because 'localhost' refers to the AVD itself, not the host outside the AVD.

This stackoverflow question saved the day: it turns out that from within the AVD you can ping the host machine at 10.0.2.2 - this is a kind of hard-coded virtual IP address in Android emulators for the host box.


Next, I wanted to make 10.0.2.2 go to localhost so that when I play a URL from the logs like http://10.0.2.2:8080/sms/srs?merch_txn_id=AK0AqgCkAMIAwwB5AOMAeA%3D%3D&i3p_db=ptsqa_fortumo3&i3p_host=10.0.2.2%3A5432&country=US
I don't want to have to replace 10.0.2.2 with localhost every time in every URL.

This part of the puzzle was solved by this very helpful link : I simply had to create a virtual IP address for the loopback device via:
ip -4 addr add 10.0.2.2/32 dev lo

After this we can see that 10.0.2.2 is an alias for the loopback interface, "lo":

[02:24:26] ~ $ ip addr
1: lo:  mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 10.0.2.2/32 scope global lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

does I3P leak memory?

I was suspecting my tomcat6 test app (I3P) was guilty of a memleak.

So I installed munin and munin-node on my box so that I could get pretty memory curves. The default munin setup doesn't plot graphs for specific processes out of the box. I eventually figured out that 'multips_memory' was the plugin for the job.

I was trying to get munin's multips_memory plugin to show me the RSS (Resident Set Size, not the other one :P - see 'man ps') of tomcat6. I wasn't getting any values in munin's multips_memory for "tomcat6" because multips_memory only checks the command name (which isn't 'tomcat' in my case). Tomcat's command line is a huge mess:

/usr/lib/jvm/java-6-sun/bin/java -Djava.util.logging.config.file=/var/lib/tomcat6/conf/logging.properties -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m -Xmx512m -Djava.endorsed.dirs=/usr/share/tomcat6/endorsed -classpath /usr/share/tomcat6/bin/bootstrap.jar -Dcatalina.base=/var/lib/tomcat6 -Dcatalina.home=/usr/share/tomcat6 -Djava.io.tmpdir=/tmp/tomcat6-tmp org.apache.catalina.startup.Bootstrap start 
 
The command name is thus /usr/lib/jvm/java-6-sun/bin/java which doesn't contain the string "tomcat" that I'm interested in. So that's why multips_memory couldn't find tomcat6 on my system... Great, here comes another tweakathon :P

So I went about adjusting the multips_memory code to suit my purposes:

[SNIP]
.
.
.
    ps -eo $monitor,args | gawk '
BEGIN              { total = "U"; } # U = Unknown. 
/grep/             { next; }
/'"$name"'/        { total = total + ($1*1024); }
END                { print "'"$fieldname"'.value", total; }'
done
.
.
.
[/SNIP]

What the above snippet does is match against the entire command+args of the processes against the string of interest using gawk. In other words, I'm now using "ps -eo args" instead of "ps -eo comm" which is necessary to find "tomcat6" in /usr/bin/java's arguments. Also, I changed the gawk regex match to search the entire line (not just $2 since the matching args for my search string could be $3 or $4 etc). I also made the search more inclusive by matching a substring instead of the exact string name  (removed the ^ and $ from the regular expression).

It works like a charm. Munin is now reporting beautiful (and worrying) memory graphs for my selected processes:

And then it struck me that this is pretty handy code. I sometimes need to see how much RAM a particular process is taking.

Every so often I see bad stuff about the tomcat app I'm testing:

4 Nov, 2011 12:58:23 AM com.aaa.bbb.application.modules.AppLauncherModule$1 uncaughtException SEVERE: Uncaught exception from Thread[Timer-0,5,main] java.lang.OutOfMemoryError: Java heap space

in the tomcat logs [/var/log/tomcat6/catalina.out]. The OOM forces a "sudo service tomcat6 restart" - really it must be a memleak.

So here's the result of this effort - a shell script inspired by the multips_memory munin plugin that that tells you how much memory all the instances of Java (e.g.) are consuming. See the usage() below for more details.

#!/bin/bash
#
# meminfo: A simple utility to display the memory usage of given process(es)
#
###############################################################################

# Default values of arguments:
VERBOSE=0
MEMORY_TYPE=rss
ARGS_TYPE=args
THIS_PROGRAM=$(basename $0)



usage()
{
cat << EOF

USAGE: $THIS_PROGRAM [arguments]

SUMMARY: A simple utility to display the memory consumption of given process(es) on this machine.

ARGUMENTS: (all optional)
    -h      Show this message
    -a      The type of arguments specified in 'ps -o'. Can be either 'args' or 'comm' (default: args)
                args: match against the full command name + argument 
                comm: match against the command name only
    -m      Specify memory type (default: rss) - see "man ps"
    -p      The process_string: can be simply a name or a regular expression.
                This argument is optional: if not supplied, all processes are considered.
    -v      Verbose mode: show debugging information

Each line of 'ps -e' is matched against the string using gawk: so you may have to escape special characters like '/' and ':" etc for gawk regex matching.

EXAMPLES:
$THIS_PROGRAM tomcat6
[show tomcat6 RSS memory usage]

$THIS_PROGRAM java
[show total memory usage by all java processes]

$THIS_PROGRAM -m vsz \\/usr\\/bin\\/java.*eclipse.*
[show memory VSZ taken by eclipse]

$THIS_PROGRAM ".usr.bin.java.+eclipse.+"
[simpler version of the above example]

$THIS_PROGRAM "\/usr\/bin\/java -Djava.library.path=\/usr\/lib\/jni -Dosgi.requiredJavaVersion=1.5 -XX:MaxPermSize=256m -Xms40m -Xmx512m -jar \/home\/ambar\/workspace\/tools\/eclipse\/\/plugins\/org.eclipse.equinox.launcher_1.2.0.v20110502.jar"
[very-specific command and args]

EOF
}


while getopts "hvm:a:p:" OPTION
do
    case $OPTION in
        h)
            usage
            exit 0
            ;;
        v)
            VERBOSE=1
            ;;
        a)
            ARGS_TYPE=$OPTARG
            ;;
        m)
            MEMORY_TYPE=$OPTARG
            ;;
        p)
            PROCESS_STRING=$OPTARG
            ;;
        ?)
            usage
            exit 1
            ;;
    esac
done



# another way to set default values, not needed here though
#: ${MEMORY_TYPE:=rss}
#: ${ARGS_TYPE:=args}
#: ${VERBOSE:=0}


if [[ -z $PROCESS_STRING ]]  # if no process name, then override ARGS_TYPE to args so that we calculate the FULL memory usage of all processes
then
    ARGS_TYPE=args
fi

ps -eo $MEMORY_TYPE,$ARGS_TYPE | gawk '
BEGIN                   { total = "U"; } # U = Unknown. 
/'$THIS_PROGRAM'/       { next; }
/grep/                  { next; }
/'"$PROCESS_STRING"'/   { total = total + ($1*1024); if('$VERBOSE'==1) {print "\n\t", $0; print "\tCUMULATIVE USAGE: ", total} }
END                     { mbs = total/(1024*1024); printf("\nTotal '$MEMORY_TYPE' memory used by all '$PROCESS_STRING' processes: %d bytes == %11.3f MB\n", total, mbs); }'

I learnt quite a bit: it's the first time I used getopts, basename, and integrated an awk script (that consumes bash variables) in a shell script. The whole endeavor seemed like a pointless digression at first, but now I think it was totally worth my time :)

And now it's time to show off the results of this little adventure:

[02:43:27] /var/log/tomcat6 $ meminfo -p eclipse -a args -m vsz

Total vsz memory used by all eclipse processes: 5109342208 bytes ==    4872.648 MB
[02:43:36] /var/log/tomcat6 $ meminfo -p tomcat6

Total rss memory used by all tomcat6 processes: 474861568 bytes ==     452.863 MB
[02:43:41] /var/log/tomcat6 $ meminfo 

Total rss memory used by all  processes: 4642054144 bytes ==    4427.008 MB

python: subtitle_delay_adjuster

Motivation: Couldn't find the perfect subtitles (.srt) file for a relatively-obscure movie? Are the subs in your .srt file off by a few seconds - just enough to irritate you, but not enough to ditch the awesome flick you're watching, AND are you feeling too lazy to tweak the subtitle offset in vlc manually?

This little proggie might just save the day  ^_^

#!/usr/bin/python2.7

import re
from datetime import datetime, timedelta
from functools import partial
import argparse



def replace_time(matchobj, time_diff):
    if matchobj.group(1) not in [None, '']:
       x = (datetime.strptime(matchobj.group(1), "%H:%M:%S") + time_diff).strftime("%H:%M:%S")
    

    if matchobj.group(3) not in [None, '']:
       y = (datetime.strptime(matchobj.group(3), "%H:%M:%S") + time_diff).strftime("%H:%M:%S")
    
    return x + matchobj.group(2) + y



def main():
    parser = argparse.ArgumentParser(
        description = 'subtitle_delay_adjuster: delay or advance all subtitles in an srt file by a given number of seconds',
        epilog = 'example: "%(prog)s -2 Thank.You.For.Smoking.2005.srt" would make all subs in the srt file appear 2 seconds earlier'
    )
    parser.add_argument('seconds', action='store', type=int, help='number of seconds by which to delay subtitles: use a negative number to advance subs instead of delaying them')
    parser.add_argument('srt_file', action='store', help='name of the srt_file to process')    
    args = parser.parse_args()

    time_diff = timedelta(seconds = args.seconds)


    infile = file(args.srt_file, 'r')
    outfile = file('out.srt', 'w')

    inlines = infile.readlines()

    for line in inlines:
        #print "line before sub: " + line
        newline = re.sub('(\d\d:\d\d:\d\d)(.*-->.*)(\d\d:\d\d:\d\d)', partial(replace_time, time_diff=time_diff), line)
        #print " line after sub: " + newline
        outfile.write(newline)

    infile.close()
    outfile.close()

    print "\n\n\nNew subs are ready! File: out.srt"




if __name__ == '__main__':
    main()

It certainly did save my day when I was trying to get the right subs for one of my favorite movies, "Thank You for Smoking" :-)

python: how to fix 'fatal error: Python.h: No such file or directory'

Was trying to install PyYAML. Got the latest tarball from the PyYAML website, and ran the usual:

sudo python setup.py install

which resulted in:

running install
running build
running buildpy
creating build
creating build/lib.linux-i686-2.6
creating build/lib.linux-i686-2.6/yaml
copying lib/yaml/constructor.py -> build/lib.linux-i686-2.6/yaml
.
.
.
running buildext
creating build/temp.linux-i686-2.6
checking if libyaml is compilable
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c build/temp.linux-i686-2.6/checklibyaml.c -o build/temp.linux-i686-2.6/checklibyaml.o
checking if libyaml is linkable
gcc -pthread build/temp.linux-i686-2.6/checklibyaml.o -lyaml -o build/temp.linux-i686-2.6/checklibyaml
building 'yaml' extension
creating build/temp.linux-i686-2.6/ext
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c ext/yaml.c -o build/temp.linux-i686-2.6/ext/yaml.o
ext/yaml.c:4: fatal error: Python.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1


Google quickly yielded the solution. The missing Python.h was in the python-dev package. Since I was running python2.6.6, I simply needed to do:

sudo apt-get install python2.6-dev

after which compilation and installation was successful:

running install
running build
running buildpy
running buildext
building '_yaml' extension
.
.
.
running installegginfo
Writing /usr/local/lib/python2.6/dist-packages/PyYAML-3.10.egg-info


git: how to ignore certain file types globally

I wanted to tell git not to ever include .pyc files in any project, anywhere on my system, instead of having to specify a separate .gitignore file in every python project directory. A simple google search solved the problem; here's one way to do this:

Create a ~/.gitignore file in your home directory:

.DS_Store
*.pyc

Then run:
git config --global core.excludesfile ~/.gitignore

That's it!

 

thanks to [source]

linux: vnstat rocks!

vnstat is a superb tool to monitor monthly bandwidth usage.

I get notified by my ISP when I hit my monthly download quota (25 GB) that my speed will now be reduced to 256 kbps. Sometimes I'm left wondering how the heck I hit 25 gigs of downloading. So I started looking for a tool on Ubuntu that would monitor my bandwidth usage and show me daily/weekly/monthly download totals.

Requirements: It should work silently and unobtrusively in the background, and it should start automatically on reboot like a daemon. It would be nice to have it display graphical graphs. Also nice to have it display realtime stats (e.g. current rx and tx speeds).

There are quite a few tools that I found: bandwidthd, bwmon, bwbar etc (complete list here: http://www.ubuntugeek.com/bandwidth-monitoring-tools-for-ubuntu-users.html).

I didn't try all of them, but the most popular (or the one that appears at the top of the google search results for "bandwidth monitor tool ubuntu") is bandwidthd, and I just didn't have the patience to get it to work. There's no user manual, no real how-to and I gave up quickly on it.

vnstat saved the day. It exactly meets my requirements. You don't need to install it from the tarball either (in fact, that didn't work too well for me, I kept getting post-install configuration errors), and if you're on ubuntu, vnstat is in the standard apt repositories.

vnStat is a console-based network traffic monitor for Linux and BSD that keeps a log of network traffic for the selected interface(s). It uses the network interface statistics provided by the kernel as information source. This means that vnStat won't actually be sniffing any traffic and also ensures light use of system resources.

 

[08:17:59] ~ $ vnstat -d

eth1  /  daily
     day         rx      |     tx      |    total    |   avg. rate
 ------------------------+-------------+-------------+---------------
  08/24/11    134.81 MiB |    7.24 MiB |  142.05 MiB |   13.47 kbit/s
  08/25/11     84.21 MiB |    5.63 MiB |   89.85 MiB |   24.82 kbit/s
 ------------------------+-------------+-------------+---------------
 estimated       244 MiB |      14 MiB |     258 MiB |
[08:18:04] ~ $ vnstat -s
                  rx      /      tx      /     total    /   estimated
eth1:
 Aug '11    219.02 MiB  /   12.88 MiB  /  231.90 MiB  /  293.00 MiB
 yesterday    134.81 MiB  /    7.24 MiB  /  142.05 MiB
 today     84.21 MiB  /    5.63 MiB  /   89.85 MiB  /     258 MiB

 

python interview questions - part 1

1. Q:What is PYTHONPATH?
   A: Python uses the directories listed in PYTHONPATH to find a module.
2. Q: How can I see the PYTHONPATH on my system?
   A: Several ways:          

    

   • From the shell prompt: echo $PYTHONPATH
   • Programmatically in python itself:
     

     >>> import sys
     >>> sys.path
     ['', '', '', '/usr/lib/python2.6/dist-packages/pygame/tests', '/usr/lib/python2.6/dist-packages/pygame', '', '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', '/usr/lib/python2.6/lib-dynload', '/usr/local/lib/python2.6/dist-packages', '/usr/lib/python2.6/dist-packages', '/usr/lib/python2.6/dist-packages/PIL', '/usr/lib/python2.6/dist-packages/gst-0.10', '/usr/lib/pymodules/python2.6', '/usr/lib/python2.6/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.6/gtk-2.0']
     

3. Q: How can I add a directory to the PYTHONPATH?
   A: Programmatically, in python itself:          

    

   #!/usr/bin/python
   
   import sys
   sys.path.append('path/to/my/modules/')
   
   # now import a module/.py file that resides in "path/to/my/modules"
   import my_module
   

4. Q: What are some standard python modules?
   A: Take a look at the Python Standard Library Reference Manual: http://docs.python.org/library/index.html
   Some standard modules are: os, sys, string, re, socket, datetime, math, commands [a useful wrapper around ps.popen() for running unix commands], httplib, urllib, json, ssl, thread, getopt, optparse, random
5. Q: What are packages in python?
   A: Packages are used to organize and group modules. Packages are just folders of modules with a special init.py file that indicates to Python that this folder is special because it contains Python modules. Packages are just a convenience to hierarchically organize modules.
6. Q: What is init.py used for?
   A: Source: http://effbot.org/pyfaq/what-is-init-py-used-for.htm
   It is used in marking a directory as a package directory. If you have the files:
   
mydir/mypackage/init.py
mydir/mypackage/mymodule.py

   and mydir is on the PYTHONPATH, you can import the code in mymodule.py via from mypackage import mymodule
   or via import mypackage.mymodule          

    

   If you remove the init.py file from the mypackage directory, Python will NOT look for modules inside the mypackage directory, and you will not be able to import the mymodule module.

   The init.py file is usually empty. Furthermore, init.py is the first file to be loaded in a package, so you can use it to execute initialization code that you want to run each time a module is loaded, or specify the submodules to be exported.

7. Q: What is the init function for?
   A: init is similar to a constructor for a class in Python. It is a way to specify default values for an object's data members when you create an object. These values are set in the class definition.          

    

   The init() method is called immediately after an instance of the class is created. It would be tempting — but technically incorrect — to call this the “constructor” of the class. It’s tempting, because it looks like a C++ constructor (by convention, the init() method is the first method defined for the class), acts like one (it’s the first piece of code executed in a newly created instance of the class), and even sounds like one. Incorrect, because the object has already been constructed by the time the init() method is called, and you already have a valid reference to the new instance of the class.

   The first argument of every class method, including the init() method, is always a reference to the current instance of the class, called "self".

   Example:

   class Point:
       def __init__ (self, arg1=0, arg2=0):
           self.x = arg1
           self.y = arg2
   
       def display(self):
           print "Point::display(): x=" + str(self.x) + " and y=" + str(self.y)
   
   p1 = Point(5, 8);
   p2 = Point();
   p1.display();
   p2.display();
   

   results in:
   
Point::display(): x=5 and y=8
Point::display(): x=0 and y=0


8. Q: What is main used for?
   A: source: http://effbot.org/pyfaq/tutor-what-is-if-name-main-for.htm
   The if name == "main": ... trick exists in Python so that our Python files can act as either reusable modules, or as standalone programs.         

    

   Example:
   square.py :

   def square(x):
       return x*x
   
   if __name__ == '__main__' ;
       print "square.py | the square of 10 is: " , square(10)
   

   test.py:

   import square
   
   print "test.py | the square of 20 is: " , square(20)
   

   execution:

   $ python square.py
   square.py | the square of 10 is: 100
   $ python test.py
   test.py | the square of 20 is: 400
   

9. Q: What is a lambda function? Explain with an example. Typical usage?
   A: A lambda function is a shorthand for an anonymous function.
   The form is lambda <arguments>: <resultant-expression>
    
10. Q: What is the map function? Explain with an example. Typical usage?
    A: map applies a function to every element of a list and returns the resultant list.
    PNEMONIC: Hence it "maps" a function to every element of a list.
    Example:         

     

    >>> map ( lambda x: x*2, [1,2,3] )
    [2, 4, 6]
    

11. Q: What is the reduce function? 
    A: reduce takes as input a function and a list. The function must take exactly two arguments. reduce applies the function to the first two successive elements of the list from left to right, and then replaces the first two elements with the result of the function.
    PNEMONIC: Hence it "reduces" the number of elements in the list by one on each operation, until eventually only one element is left.Example:         

     

    >>> # this effectively sums the elements in the list by doing ((((1+2)+3)+4)+5)
    >>> reduce( lambda x,y : x+y, [1,2,3,4,5] )
    15
    

12. Q: What is the filter function?
    A: filter applies a function to every element of a list. Unlike, map, however, the function must return either True or False for each check. filter returns a list of all elements that pass this check.
    PNEMONIC: Hence it "filters" those elements that pass the test of the function.         

     

    Example:

    >>> # get those elements in the list that are divisible by 3
    >>> filter ( lambda x : x%3==0 , [1,2,3,4,5,6])
    [3,6]
    

13. Q: What is a list comprehension? Explain with an example. Typical usage?
14. Q: What is pass used for?
    A: The pass statement in Python is like a empty set of curly braces ({}) in Java or C. 
15. Q: How do you create an empty tuple?
    A: t = ()
16. Q: How do you create a tuple containing one element?
    A: t = (something, ) #note the comma at the end - this is the only way to specify a single-element tuple 
17. Q: How can you use the and-or trick safely (to simulate a C-style ternary expression)?
    A: <exprtotest> and [a] or [b])[0] 
18. Q: How can you use a much-nicer and more-readable ternary style expression in Python?
    A: Example: map(lambda x:x*2, range(0,20)) if (2 > 1) else ""
19. Q: How can you delete an element or a slice of elements from a list?
    A: Using the del statement
    
    >>> a = [-1, 1, 66.25, 333, 333, 1234.5]
    >>> del a[0]
    >>> a
    [1, 66.25, 333, 333, 1234.5]
    >>> del a[2:4]
    >>> a
    [1, 66.25, 1234.5]
    >>> del a[:]
    >>> a
    [] 

     

     

20. Q: What are ".pyc" files and when are they generated?
    A: Importing a module is a relatively costly affair, so Python does some tricks to make it faster. This .pyc file is useful when you import the module the next time from a different program - it will be much faster since a portion of the processing required in importing a module is already done. Also, these byte-compiled files are platform-independent.
    source
21. Q: What is the use of the inbuilt dir function?
    A: You can use the built-in dir function to list the identifiers that an object defines. When you supply a module name to the dir() function, it returns the list of the names defined in that module. When no argument is applied to it, it returns the list of names defined in the current module.
    source
22. Q: Explaing duck-typing wrt Python.
    A: Duck-typing can be simply explained as, "As long as you get the job done, we don't care who your parents are." 

     

    In other words, as long as an object can perform these actions:

    • quackLikeADuck()
    • walkLikeADuck()
    • swimLikeADuck()

    then that object is a Duck for Python. It doesn't matter which class the object is derived from (who its parents are). This greatly improves polymorphism and thus flexibility. Duck-typing avoids tests using type() or isinstance()

    Example: The canonical examples of duck typing in Python are file-like classes. You can use cStringIO to handle a string like a file, or GzipFile to handle a gzipped-object like a file. Similarly you can treats sockets like files too. However, sockets lack the 'tell()' method and so cannot be used everywhere that a file can be used. This shows the flexibility of duck typing: a file-like object needs to implement methods only that it is able to. Thus it can only be used in situations where it makes sense. Where the object is expected to support the 'tell()' method, and it doesn't, it can throw a "TellException" instead, which makes it easy to handle the unsupported-method-exception in the calling code and then move on. This is called as EAFP: "Easier to Ask Forgiveness than to ask for Permission" style programming.