Debugging Production Environments

Have you ever been in the position where you have to look after a legacy app that has insufficient logging and non-deterministic problems? Whatsmore, it's running in production and that rare non-deterministic bug has just occurred. How do you diagnose it?

This task faced me last week. Looking at the code on my machine, I just couldn't see how the problem could manifest itself. "Afterall, there is only one instance of this object in the entire system, right?" I found myself saying. "It must be a bizarre and rare threading issue!"

So, I ran this on the production Linux box (where PID is the Java process ID):

$JAVA_HOME/jmap -histo:live PID | grep OUR_CLASS

and saw something like this (I've added the column headings for clarity):

num #instances #bytes class name

----------------------------------------------

9030: 2 200 OUR_CLASS

So, there were 2 instances of our class! All my recent investigation of odd threading issues was for naught!

This is one reason why the factory pattern is so nice - you know exactly where the objects are being instantiated (tip: make the class's constructor package protected and put the factory in the same package with no other classes in that package). As an added benefit, it makes writing test code easier.

Still, this isn't going to help you if some smart alec decides to introspectively instantiate an object of this class. But such a practice should be strongly discouraged.