Books to read for Python programmers

So you have completed learning Python, what's next?. Learn the Data structures and algorithms to be real programmer. here I suggest 3 books that I read and practised. 

1. Grokking algorithms, buy from  amazon here.

2. Elements of Programming interviews In Python. buy here

3. if you want deep dive into algorithms then "Introduction to algorithms" is best book

you can buy from Amazon here.

Pearson Correlation vs Spearman correlation

correlation: Say we have 2 variables X and Y. if X value changes which also change Y then these two variables said to be in correlation. if Y increases with increase in value of X then these two said to have +ve correlation. if Y value decreases with increase of X then these two are in -ve correlation. 

correlation can help in determing other value , given a value.

Lets take a data set and calculate correlation between 2 variables. 
download using link here
OR
#wget https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data

This is Cars Data set which has following description
    1. mpg:           continuous
    2. cylinders:     multi-valued discrete
    3. displacement:  continuous
    4. horsepower:    continuous
    5. weight:        continuous
    6. acceleration:  continuous
    7. model year:    multi-valued discrete
    8. origin:        multi-valued discrete
    9. car name:      string (unique for each instance)

Lets discuss 2 types of correlation here, Pearson correlation expects linear relationship between variables. Spearman correlation works on non-linear relationship. 

Linear relationship means Increase in value of X, increases/decreases Y in same ratio. 

Here is the program to determine whether weight influences the mpg factor. 

pandas has DataFrame which can give correlation between two variables. 
say if df is object of DataFrame class, then df['col1'].corr['col2'] gives correlation between col1 and col2 . df['weight'].corr(df['mpg']) gives how weight influences mpg. 

import pandas as pd
def correlation(df,col1,col2,method='pearson'):
   coeff = df[col1].corr(df[col2],method=method)
   return coeff
def main():
   df = pd.read_csv('auto-mpg.data', delim_whitespace=True)
   df.columns = ['mpg', 'cylinders', 'displacement' , 'horsepower', 'weight', 'acceleration', 'model year' , 'origin', 'car name']
   ''' lets calc correlation between mpg and weight
   as weight increases how does mpg effect '''
   coeff = correlation(df, 'weight', 'mpg', 'spearman')
   print("spearman correlation is %f" % (coeff))
   ''' now calc pearson correlation'''
   coeff = correlation(df, 'weight', 'mpg', 'pearson')
   print ("pearson correlation is %f" % (coeff))

if __name__ == '__main__':
   main()

~         

How to become Data Scientist

I have been playing with data science since few years. Here are steps to become data scientist .

1. Choose Python or R. (i am python developer since 7 years, so i am gonna recommend few ebooks here)
Python: Dive Into Python pdf. 
I would also recommend python class here from google


2. Once you are good at programming in Python , Learn statistics
ebook: Think Stats

3. Learn Numpy, Scipy, Pandas and machine learning
ebook: Practical machine learning with Python . 
you can pick up chapters related to Numpy, Pandas, Visualization techniques with pandas or seaborn and Machine learning in the above ebook.


Once you are done with these 3 ebooks, you should be able to do some data sets. try to sign up in www.kaggle.com and compete in data science problems. you can also download some data sets and practice.

Make HTTP Requests using Python + httplib

Python has very good package called httplib, which can send GET,POST,PUT,DELETE methods.

while dealing with REST API's you may want to test the API's.

Examples Here include Registering the User using POST .

POST /user creates user.
GET /user returns the list of users.
GET /user/uid return user details.
PUT /user/uid modifies the user details.
DELETE /user/uid deletes the user from server.

I am providing pseudo code here.

import httplib

# This creates the connection with server running on 5000 port.

h = httplib.HTTPConnection('127.0.0.1', 5000) 

def do_post():

   '''POST here'''

  hdrs = {'mobile':mobile,'email':email}

  h.request('POST', '/user', None, hdrs)

  resp = h.getresponse()

  print(resp.read())

  print(resp.status) # This prints HTTP status code.

    

def test_get():

  hdrs = {'mobile':mobile,'email':email}

  h.request('GET', '/user', None, hdrs)

  resp = h.getresponse()

  print(resp.read())

  print(resp.status) # This prints HTTP status code.

def test_put():

  hdrs = {'mobile':mobile,'email':email}

  #passing known user id 1234 here, you can change for your use case.

  h.request('PUT', '/user/1234', None, hdrs)

  resp = h.getresponse()

  print(resp.read())

  print(resp.status) # This prints HTTP status code.

we can further improve this program making multi-threaded/multi-processing and use it to test scalability and performance of your server.

Run Python + Flask Web app in Docker Container

Do you want to Dockerize your Python Flask web app. Well follow these steps. very often there will be difference of packages installed on Dev vs tester systems.
So running webapp using Docker can be easy for testing.
Follow this 10 step simple tutorial.

1. Create a Virtual Environment first .

# virtualenv dockerC
# cd dockerC
# source bin/activate

2. Install Flask

# pip install Flask

3. Install any other packages u want eg: memcached.

# pip install python-memcached

4. collect the requirements, so that we package all modules.

# pip freeze > requirements

5. create a DIR with webapp. move the requirements file to that dir.

# mkdir webapp

 #mv requirements webapp/

6. work on your Webapp using Flask.
app.py has code for entry points.

#cat app.py
from flask import Flask
app = Flask(__name__)


@app.route("/")
def get():
    return "Hello Docker"

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port=8000)

7. Now create Docker File called "Dockerfile" in same webapp dir
The contents has to be like this.

FROM fedora:latest
MAINTAINER Maintainer redara
RUN dnf install -y python python-pip
COPY .   /webapp
RUN pip install -r /webapp/requirements
EXPOSE 8000
WORKDIR /webapp
CMD python app.py
#eof

The Structure of our webapp dir is here.

(dockerC) [vedara@localhost webapp]# ls -ltr
total 12
-rw-r--r--. 1 vedara vedara 151 Apr 10 16:18 requirements
-rw-r--r--. 1 vedara vedara 180 Apr 10 16:46 app.py
-rw-r--r--. 1 vedara vedara 187 Apr 10 16:49 Dockerfile

description of Docker file is here
FROM uses the fedora image from docker hub.
RUN means we are installing required packages . here we are installing python,pip first
COPY .   /webapp means we are copying our current dir(. dir) which is "webapp" to /webapp dir in destination
once we copied our webapp dir, lets install required packages for Flask using RUN pip install -r /webapp/requirements
EXPOSE will expose the port on which container listens to connections. our webapp uses 8000 as port.
WORKDIR sets the working dir , so that you dont need to specify absolute paths everytime
CMD python app.py shows the cmd needed to run the flask web app.

8. Lets Build the Docker Image now. give -t which can be used to refer your image. "." means looks for Dockerfile in curr dir.

# docker build -t flask_webapp .

9. Lets run the Docker Container now by using the cmd here. giving -p tells docker to publish the containers port to host.

   #docker run  -p 8000:8000 flask_webapp 

  If you want to run as daemon specify -d option in above cmd.

10. Lets hit the URL now using curl

  #curl -i http://0.0.0.0:80

 HTTP/1.0 200 OK
 Content-Type: text/html; charset=utf-8
 Content-Length: 12
 Server: Werkzeug/0.12.1 Python/2.7.13
 Date: Mon, 10 Apr 2017 12:15:39 GMT
 Hello Docker

Thats it. your web app is running. Push it your Docker Hub using your username/password

#docker push username/tagname

Note: This demo is based on Just "Hello Docker". copy your entire webapp files using COPY . if you use database you can ship different container for Database server or Ship in Same Container.

Use the Dockerfile to copy required packages and run in same container.

Contact me for any queries: smart.ram856@gmail.com

Use C Library In Python using C Types

Do you have a C library which needs to be used for Python Devs. well, you can have python bindings which loads the C library and use those functions using C types.

Here I am giving small demo, my C library has add and multiply functions like this.
demo.c
=======

#include

int add(int,int);

float mul(float,int);

int add(int a,int b)

{

return a+b ;

}

float mul(float a,int b)

{

return a*b ;

}

Lets Compile this program as Shared Library

 # gcc -c -Wall -Werror -fpic demo.c 

The above step produces demo.o, Lets have shared lib now

On Linux

 # gcc -shared -o libcalc.so demo.o

On Mac

# gcc -shared -o libcalc.dylib demo.o

Lets Use this libcalc.so or libcalc.dynlib in Python now, using the C types. 

api.py

=========================

import ctypes

from ctypes.util import find_library

from ctypes import sizeof

'''this will find library name . In Linux you have libcalc.so , in Mac libcalc.dynlib'''

so_file_name = find_library("calc")

if so_file_name is None:

    raise Exception("libcalc.so not found")

try:

    client = ctypes.CDLL(so_file_name, ctypes.RTLD_GLOBAL, use_errno=True)

except OSError:

    raise ImportError("cannot load {0}. please set LD_LIBRARY_PATH \                              variable".format(so_file_name))

def calc_func(method,rettype,*argtypes):

    '''

    create a func belonging to calc.so

    '''

    return ctypes.CFUNCTYPE(rettype,*argtypes,use_errno=True)((method,client))

''' Define python functions for each C function in libcalc.so  

we have add,mul functions in our libcalc.so.  '''

py_add = calc_func('add',ctypes.c_int, ctypes.c_int,ctypes.c_int)

py_mul = calc_func('mul',ctypes.c_float,ctypes.c_float,ctypes.c_int)

Now we have api.py which loads the C library and bridges to python.

main.py which uses api.py in python style

==================

import api

add_output = api.py_add(2,3)

print(add_output)

mul_output = api.py_mul(float(5.5),10)

print(mul_output)

#python main.py 

Note: What if you have C++ library, well its better you use extern "C" and write a C wrapper which intern calls C++.

Generate subsets of an array using Python

Problem: If you given a problem to generate subsets of given array , we can use PowerSet concept.

lets say Array = ['Red','Orange','Blue']

the power set is defined as 2 to power of len(Array). 2^3 here.

Run the binary counter from 0 to pow(2,len(Array))

Value of Counter            Subset
    000                    -> Empty set
    001                    -> Red
    011                    -> Red,Orange
   100                     -> Blue
   101                     -> Red,Blue
   110                     -> Orange,Blue
   111                     -> Red,Orange,Blue

def gen_subsets(list_attribs):
    len_set = len(list_attribs)
    power_set = 2 ** len_set
    for i in range(power_set): # this generates the binary sequence
        subsets = ""
        for j in range(len_set): #iterate through the list_attribs positions and check bit is set
            if i & (1 << j):
                subsets +=  list_attribs[j] + ","
        print(subsets[0:-1])
        

My 1st App on App Store - VCalendar

Finally after practicing the iOS dev from many sources, I developed an app called VCalendar which is Indian Calendar app. This app shows the Indian Vedic timings for the current year. This Calendar is based on VenkatRama & Co calendar Ap, India.

I published this app on April 21st on app store. This app shows the sunrise/sunset, Tidhi, Star, Festivals, Rahukalam,Masam,Gulika Kalam etc. There are timezone conversions for these vedic timings also .

If you are an Indian staying abroad, download this calendar.

Here is the link:
https://itunes.apple.com/in/app/vcalendar/id1104625702

Here are few screenshots of the app. more screen shots are there in App Store.

Note: App updated with Telugu language support for 2017 calendar.

Getting hands on Apple ios dev.

This year 2015, I didnt launched any products , but making myself exposed to mobile app dev.
As a  apple fan I have chosen to be ios dev and launch cool apps.

Actually I started getting hands on Objective C, but by the time I completed Obj C , Apple launched
Swift which is faster , reliable than Obj C. then i started with Swift again which is very cool and powerful. I always learn any tech in 2 phases .

Phase 1: Get the basics right.
Phase 2: Advanced by building up cool apps.

I am still in Phase1 , I bought the udemy ios dev app course which gave me the basics, but to be expert I recommend the http://www.raywenderlich.com/ ios apprentice and other advanced tutorials.

My advice to newbies who want to learn ios dev is do 2 tutorials what i specified above, build some stuff using CoreData, CoreLocation, Tableviews, ImageViews, gestureRecognizer etc. which are used frequently in most of apps.

I started making a calendar app with some features like holiday reminders, diary etc. I hope to release the app in 2016.

my 1st failure as Entrepreneur

Me and my friend Eswar always discuss about ideas whenever we meet. we both are passionate to start some venture . we had an idea to make drinks online, we thought of making a drinks store online every 5 kms in Bangalore city and then spread to other cities. techies, software companies can buy drinks online to party at their home or at office.

we did coding for ecommerce site with sample data of drinks . we are new to web programming stuff. as we are already know core python programming, we learnt Django stuff and we built up ecommerce site on Django,Jquery,Html,CSS3 . we bought domain name called Drinkybee.com and we hosted it even though we dont have any customers . we hosted for beta testing and for showing demo to our customers who wants to host their store in our drinkybee.

we are new to marketing and learnt how to market our site. many store vendors rejected us as they dont know how much online business can do . but we got 2 customers who has 6 stores. we thought of going online and came to know the risk that liquor drinks cant be delievered to home and its Illegal according to karnataka excise dept. Our customers has called us saying lets stop going online as it has legal problems.

Before Setting up business check the legal front of it and then invest on it. By gones are bygones, we do have ecommerce site now and we can sell atleast that now. but our dream to launch drinkybee site didnt fade away . may be we have to approach other city like Mumbai or hyderabad.

our vision is just make drinkybee as synonym to drinker and at same time being responsible to society by reducing drink and drive .

permuations of list of lists for ecommerce attributes

Q)  In a ecommerce product a product can have many attributes like size,color,gender. and again each attribute can have multiple values like size ['s,'m','L','XL','XXL'].
and color as ['red,'blue','brown'] and gender as ['M','F'].

A ecommerce product want to display all possible combinations of attributes. each attribute has multiple values and attribute list vary in length. ex: if color ['red,'blue','green','brown'] and gender as ['M','F']
then possible combos are [red,M],[red,F],[blue,M],[blue,F],[green,M], [green,F], [brown,M] , [brown,F]

Solution1: Just a recursive algo, which takes list as argument. for each element in list , get the permutations of remaining lists.

l = [['red', 'brown', 'black'], ['S','M','L', 'XL'], ['M','F']]

def permu(lists, prefix=''):

    if not lists:

        print prefix

        return

    first = lists[0]

    rest = lists[1:]

    for letter in first:

        permu(rest, prefix + letter +",")

permu(l)

Solution:2 make a m way tree. like root having children for 1st set of attributes. and for each node of attribute has next set of attributes as children.

            [root]
[red] [blue] [green] [brown]

[M] [F]

. Then doing a dfs till node and print the path to leaf will give different combinations of product.

In Python:

class Node(object):

    def __init__(self):

        self.data=''

        self.children=[]

                

class mway(object):

    def __init__(self):

        self.root=None

    def insert(self,list_string):

        if self.root is None:

            root=Node()

            root.data=''

            root.children=[]

            self.root=root            

        

        node=self.root

        list_nodes=[]    

        for i in list_string:

            node=Node()

            node.data=i

            node.children=[]

            list_nodes.append(node)

        #stack for insertion

        li_bt=[self.root]

        while len(li_bt) > 0:

            node=li_bt.pop()

            #print "popping %s"%(node.data)

            if len(node.children) == 0:

                for n in list_nodes:

                    node.children.append(n)

                    #print "appending %s to %s"%(n.data,node.data)

            else:

                for chi in node.children:

                    if chi not in list_nodes:

                        li_bt.append(chi)

                    

    def printcomb(self,node,string=""): #dfs to leaf and print path

        if len(node.children) == 0:

            print string

        for i in node.children:          

            self.printcomb(i,string+ i.data + ",")

            

    def getroot(self):

        return self.root

        

print "enter number of attributes"

li_all=[]

attr=int(raw_input())

while attr > 0:

    print "enter attributes seperated by ,"

    list_attr=str(raw_input()).split(",")

    li_all.append(list_attr)

    attr=attr-1

    

mtree=mway()

for iattr in li_all:

    mtree.insert(iattr)

    

node=mtree.getroot()

mtree.printcomb(node)

stringroot.com going live

Finally my sringroot.com is live now :). The one and only social networking site where you can share photos,music,pdf,ppt,mp4 videos in just few clicks.

busy in fixing bugs for www.stringroot.com

after working for 6 months on my social content site, I registered name stringroot.com for my site. Finally launched the site, but under vigorous testing phase. my friends , collegues helping out in their free time by identifying bugs by doing thier way of testing.

Testing is very important phase for any product or website. I sometimes break existing things by fixing up new bugs becoz of over confidence. Now I learnt many things from this journey being zero to founder of stringroot.com. developing server side middleware, geting things done for UI(css) from others, learning jquery ,css and especially learning scalability things for my website to satisfy thousands of requests.

I hope by end of June, I will launch a complete www.stringroot.com .

stringroot is Social Content site with minimal Social networking features. you can follow people who upload their content like music,photos,pdf , power point presentations . If you have content which should be shared among 100's ,1000's of members stringroot is your place. once your content reaches thousands of people you will be very famous and people will start recognizing you.

Stop emailing your content to your friends every time you wanna share ebooks,music,photos,flash videos. tell them to follow you on stringroot.com and people will get your content displayed on their home page.

We dont encourage any pirated,hatred,morphed content on our site .

I Thank my wife for allowing me to work on my idea, my friend eswar who is also vice president for my venture and his roomates, my collegues and ravinder pagidi . 

mini python search engine

many web developers use database for search. does it scale?. I dont think so. every tried to search using LIKE or CONTAINS operators in database. did it return results which are closer to search string .

when you search for RAM , you may get  RAMARAO, RAM EDARA, RAMGOPAL VERMA, VenkataRam  etc.. bunch of results... but which result  is closer to your search string. did you ever cared about that!!.

many use sphinix or apache lucene..did you ever try to write your own search algorithm !!

ever wondered how google search works...when your search has many words?.

1) gets  list of document id's for each word u searched and will do intersection of doc ids.
2) uses some algorithms which relate ur search to context of search
3) will use ranking algorithms based on number of times string appeared on web page or number of hits on the page for that search string
4) displays results to you.

I am developing a mini text search algorithm which would be useful for many websites to search their users based on email,firstname, last name etc.

I am going to use python dictionaries and also edit distance algorithm, this may take little bit of memory for search, but it takes just O(1) to retrieve the user details...results will be based on how string is closer to your search string. to save memory we can also use trie data structure with some modifications , but it would take O(k) time where k is length of your search string.

watch this space for more info on my search algorithm. I would compare python dictionary vs python trie for searching...

coming up with social content site

all these days busy in building up new social content site. using python django as my framework.

learnt many things abt software project management. Things I have learnt are

1) you will never finish enhancements . once u come up with idea and do coding daily...new ideas come up and its upto you to decide which features to ship to satisfy your clients.

2) things you think easy may take more time..and some other things you  think hard may get finished
in lesser time.

3) modules that are not doable by you can be outsourced but not to cost of wasting money and time.

4) for any kind of product , people hate the word 'slow'. along with functional requirements make sure performance requirements are fulfilled.

5) technical things I have learnt are .
a) python , django framework
b) hitting db always doesnt scale. so used memcached for caching
c) divide project into subproblems..abstract the core module so that other developers can work on that and enhance the project later.
d) write wrapper for modules which are used frequently rather than duplicating the code.

and finally
e) do not re-invent the wheel again just to show that you can code complex things...I came across open source project memcached for using in memory cache to save DB hits. it saved lot of time for my project and many famous social portels like facebook use it. so its okay for me to use memcached as my in-memory cacheing

finally looking ahead for front end developer who can add colors to my website and look it attractive like heroine ileana :) lolz just kidding .

will blog again once I complete my social content site :) . 

find least common ancestor in Binary search tree

Q)least common ancestor for binary search tree

recursive way of doing is here. In BST, all values lesser than root will be on left subtree, values greater than root will be on right subtree.

node *ancestor=NULL

node * LCA(node *root,node *n1, node*n2)
{

if(n1 == NULL && n2 == NULL)
  return root ;

if(n1 == NULL)
 return n2;

if(n2==NULL)
  return  n1 ;

return findLCA(root,n1,n2) ;
}

node *findLCA(node *root,node *n1,node *n2)
{
ancestor=root;

if(n1->data < root->data && n2->data < root->data)
 return LCA(root->left,n1,n2)
else if(n1->data > root->data && n2->data > root->data )
return LCA(root->right,n1,n2)
else
return ancestor;

}

Binary tree Questions

1) how to merge two Binary search trees

A) convert the each binary tree to Single LL using inorder traversal
merge two LL and copy to array.

construct binary tree with mid element as root. if you want to balance tree then we can use AVL tree rotations.

node *prev=NULL;
node *head=NULL:

converttoLL(Treenode *root,node **head)
{
if(root !=NULL)
{
converttoLL(root->left,&head)

if( *head==NULL )
*head=prev=(node *)root;
else{
prev->next=root;
prev=root;
}

convertoLL(root->right,&head)

}//..end if(root !=NULL)
}

constructroot(Treenode *&root,int mid)
{
Treenode *temp=new Treenode();
temp->data=mid;
temp->left=null;
temp->right=null;
root=temp;

}

constructtree(Treenode *&root, int element)
{
Treenode *curr=root;

while(curr!=NULL)
{
 if(curr->data <= element )
curr=curr->left;
else
curr=curr->right;
}

Treenode *temp=new Treenode();
temp->data=mid;
temp->left=null;
temp->right=null;

curr=temp;

}

int main(){

// main method
node *head1=NULL ;
node *head2=NULL;

ConverttoLL(Tree1,&head1);
ConverttoLL(Tree2,&head2);

node *n2=head2;
while(n2!=NULL)
{
node *save=n2->next ;
merge( &head1 , n2 ) ;
n2=save;
}
code for merge is in
http://techie-builder.blogspot.in/2012/06/linked-lists-questions.html

node *n1=head1;
int a[100];
int i=0;

while(n1!=NULL)
{
a[i]=n1->data ;
i++;
n1=n1->next ;
}

constructroot(a[i/2]);

for(int j=0;j < i ; j++ )
{
if(j!=i/2) // already we constructed root
constructtree(a[j]);
}


}

Merge Sort iterative way

recursive way is very easy to write for merge sort. generally interviewers now a days asking for iterative way..here is mergesort in iterative way. this prints elements in ascending manner.

NOTE: some of greater , lesser operators could be missing due to bloging issues .


#include
#include
#include
using namespace std  ;

void merge(int [],int,int,int);
int main()
{
int a[100];

int j=0,size=0;
for(int i=10;i > =0; i--){
++size;
a[j++]=i;
}

int mergelen=2,start=0;
int i=0;
int mid=0;
int lastmerge=0;
while(mergelen < size)
{

for(i=0;i < size; i=i+mergelen)
{
if((i+mergelen) < size){
mid=i+mergelen/2 ;
merge(a,i,i+mergelen-1,mid);
}
else{
mid=(i+(size))/2;
merge(a,i,size-1,mid+1);
}
lastmerge=i;
}
mergelen=mergelen * 2 ;
}
merge(a,0,size-1,lastmerge);
cout<<"final"<
//print elements here.
getchar();return 0;
}


void merge(int a[],int start,int mergelen,int size)
{
cout<<"\n pass "<
int temp[100];
//cout<<"\n pivot is "<
int i=start,j=size,k=0;
while(i < size && j <= mergelen)
{
//cout<<"\n exchanging "<
if(a[i]>a[j]){
temp[k]=a[j];
k++;j++;
}
else{
temp[k]=a[i];
k++;i++;
}
}
while(i < size )
temp[k++]=a[i++];

while(j < = mergelen)
temp[k++]=a[j++];


for(int index=0;index < k ; index=index+1)
a[start++]=temp[index];

}

binary semaphore vs mutex

binary semaphore take 0 and 1 values. so we may think only 1 thread can access the resource.
but in semaphore there is nothing like ownership of resource and also other thread can
increment the semaphore . In mutex only the thread which acquired the lock can release the lock and the
thread is owner of the lock. developers uses mutex for  mutually exclusive lock instead of binary semaphore.

data structures - interview questions

I got few requests to post questions on linked lists,trees,graphs , sorting algorithms etc. i will be blogging when i have time for this.

first linked lists questions are here

http://techie-builder.blogspot.in/2012/06/linked-lists-questions.html

sortings:

http://techie-builder.blogspot.in/2012/07/merge-sort-iterative-way.html

Arrays:
https://techie-builder.blogspot.in/2016/06/generate-subsets-of-array-using-python.html