Dtrace to find memory leaks - Solaris 10

The dbx and libumem utilities are pretty slow in finding leaks. in my experience I didnt get callstacks for memory  leaks.  so I used Dtrace tool which can print malloc/free callstacks and size to outputfile. From the output file we can extract malloc addresses which are not present in free ptr address. these addresses and callstacks will be memory leaked ones.

Step 1 ) use this dtrace script to print malloc/free statistics . usage is ./memleak.d pid  > output.txt

#!/usr/sbin/dtrace -s
#echo " usage ./memleak.d pid "
pid$1:libc.so.1:malloc:entry
{
        self->trace = 1;
        self->size = arg0;
}
pid$1:libc.so.1:malloc:return
/self->trace == 1/
{
        printf("Ptr=0x%p Size=%d", arg1, self->size);
        ustack();
        self->trace = 0;
        self->size = 0;
}

pid$1:libc.so.1:free:entry
{
        printf("Ptr=0x%p ", arg0);
}

for example if the process id is 1234 then
/memleak.d  1234 > output.txt

 step 2 )
copy this python script to showleaks.py and run  python ./showleaks.py output.txt 6 > leaks.txt 
output.txt is the output file  you got from dtrace 
6 is the depth of callstack. make sure your output.txt has minimum of callstack depth 6. 

#showleaks.py


import sys
import re

if (len(sys.argv) > 1 ):
    allocsfile = open(sys.argv[1],'r')
    allocs_lines=list(allocsfile.readlines())
    linenums=len(allocs_lines)
    i=0
    poin_size=" "
    ''' str1='12  66201                    malloc:return Ptr=0x10244cb30 Size=73' '''
    allocs_map={}
    stacks_map={}
    final_map={}
    depth=int(sys.argv[2])

    malloc_set=set(" ")
    free_set=set(" ")
    while i < linenums:
   
        malloc_match=re.search(r"malloc:return Ptr=\b0[xX][0-9a-fA-F]+\b Size=[0-9]*", allocs_lines[i])
        free_match=re.search(r"free:entry Ptr=\b0[xX][0-9a-fA-F]+\b",allocs_lines[i])
        if malloc_match:
            malloc_line=str(malloc_match.group())
            pos_ptr=malloc_line.find('=')
            ptr_address=str(malloc_line[pos_ptr+1:pos_ptr+12])
            malloc_set.add(ptr_address)
            allocs_map[ptr_address]=i # store ptr addr and line number
        elif free_match:
            free_line=str(free_match.group())
            pos_ptr=free_line.find('=')
            freeptr_address=str(free_line[pos_ptr+1::])  
            free_set.add(freeptr_address)
        i=i+1
else:
    print("usage: ./showleaks.py ")

leak_set=malloc_set - free_set


for i in leak_set:
        j=allocs_map[i]
        formatstr="malloc:return Ptr=%s Size=[0-9]*" % i
        malloc_match=re.search(formatstr,allocs_lines[j])
        if malloc_match:
            malloc_line=str(malloc_match.group())
            pos_ptr=malloc_line.find('=')
            rest_of_line=str(malloc_line[pos_ptr+1::])
            pos_Size=rest_of_line.find('=')
            Size = int(rest_of_line[pos_Size+1::])
            callstack=str(allocs_lines[j+1:j+depth])

            if callstack in stacks_map:
                value=stacks_map[callstack]
                stacks_map[callstack] = value + Size
            else:
                stacks_map[callstack] = Size

keys=stacks_map.keys()

for k in keys:
    print("leaked bytes: %d" %stacks_map[k])
    stacks=str(k)
    stacks1=stacks.split(r'\n')
    for sts in stacks1:
        print(sts)
   
print("\n DONE ")
   
   
   

for any queries, pls do post here. I will respond to it .

9 comments:

Anonymous said...

I have followed the steps mentioned above but iam getting a error

dtrace: failed to initialize dtrace: DTrace requires additional privileges

Venkat Edara said...

we should use dtrace as root user.

Anonymous said...

running the second script doesn't provide any input... its just print DONE in output file.

But I can see in my memory graph for that process that there is memory leak...
Can you please elaborate the working of second script?

Venkat Edara said...

did u give dtrace output as input to python script?.

Venkat Edara said...

u can also email ur dtrace output if u want to smart.ram856@gmail.com i can run python script and give output

Sonal said...

I have followed the steps, but the first script memleak.d is not providing any output.
The output file generated is blank.
Please advice.

Venkat Edara said...

u need to run memleak.d script as root user and try. also give pid as argument.

Venkat Edara said...

did u give the output of dtrace to python script ?. just email me your dtrace output if you can. i will look into that . this script is being used in oracle for finding leaks.

my email: smart.ram856@gmail.com

Transform DBA said...

the scripts worked, great job. now I just need understand what actually happened :)