Memory leaks in a native library?

Everybody should be proficient with examining Java heaps. However, if your Java process uses native libraries (Tibco in our case), diagnosing memory leaks becomes harder.

These are a few notes I've made when trying to ascertain what is going on when our 4g Java heap looks fine but our off-heap usage (as shown by the pmap Linux command) has now approached 12g!

Say we wanted to know what our process was linked to. We can do this with:

[phenry@localhost MemLeaker]$ ldd leaker 
linux-gate.so.1 =>  (0x00bf0000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x47007000)
libm.so.6 => /lib/libm.so.6 (0x46903000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x468cc000)
libc.so.6 => /lib/libc.so.6 (0x46710000)
/lib/ld-linux.so.2 (0x466ee000)

where leaker is an executable I created in C++.

Say, my program creates a thread using pthread_create. Where does this thread live? Well, going through those above files that my executable links, I found the one by doing this:

[phenry@localhost MemLeaker]$ nm -D /usr/lib/libstdc++.so.6 | grep thread | grep create

         w pthread_create

         w pthread_key_create

The nm command lists the symbols from the object files.

Now, say I was to dump the memory of a process (it need not be a JVM but let's say this one is) I would do this:

[henryp@corsair DumpMemory]$ gdb -p YOUR_PID_HERE

.

.

(gdb) gcore /hdda1/java.core

(The gdb is the GNU debugger. Think of it as attaching a Java debugger to a JVM).

Note: this file is big. It was 17g for a small Java program so make sure you have disk space.

Now examine it with:

[henryp@corsair DumpMemory]$ gdb --core=/hdd1/java.core /usr/java/jdk1.7.0_51//bin/java

.

.

(gdb) bt

#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:218

#1  0x00007f4656e4d2f8 in os::PlatformEvent::park(long) () from /usr/java/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so

#2  0x00007f4656e4e044 in os::sleep(Thread*, long, bool) () from /usr/java/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so

#3  0x00007f4656cb6612 in JVM_Sleep () from /usr/java/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so

#4  0x00007f464d012cd8 in ?? ()

#5  0x00007f46566377d0 in ?? ()

#6  0x00007f4656637820 in ?? ()

#7  0x00007f46566377c0 in ?? ()

#8  0x0000000000000000 in ?? ()

Picking a thread at random, we see it's in JVM_Sleep in the libjvm shared object. Let's just check:

[henryp@corsair DumpMemory]$ objdump -t /usr/java/jdk1.7.0_51/jre/lib/amd64/server/libjvm.so | grep JVM_Sleep

000000000067d2d0 g     F .text 000000000000045c              JVM_Sleep

"The text section of a program is where the program instructions live." [1]

If you want to look for text at these addresses, you need something like this:

(gdb) x/1000c 0x00007fb22c000000
0x7fb22c000000: 32 ' ' 0 '\000' 0 '\000' 44 ',' -78 '\262' 127 '\177' 0 '\000' 0 '\000'
0x7fb22c000008: 0 '\000' 0 '\000' 0 '\000' 0 '\000' 0 '\000' 0 '\000' 0 '\000' 0 '\000'
0x7fb22c000010: 0 '\000' 16 '\020' 2 '\002' 0 '\000' 0 '\000' 0 '\000' 0 '\000' 0 '\000'
0x7fb22c000018: 0 '\000' 16 '\020' 2 '\002' 0 '\000' 0 '\000' 0 '\000' 0 '\000' 0 '\000'

The syntax is:

x/nfu a

where n is the number of u units with format f starting at address a.

You can also search the memory for a particular string with:

(gdb) find /b 0x00601000, 0x7FFF175E900, 'P', 'h', 'i', 'l', 'l'

Where the syntax can be found here.

Alternatively, you can dump the memory to a file and then run it through the strings command to make it  easier to read with:

(gdb) dump binary memory FILE_NAME START_ADDRESS END_ADDRESS

More information on core dumps can be found here.

[1] Programming from the Ground Up.