I tried adding the following JVM args when I ran the JMeter performance tests:
-XX:+AggressiveOptsUses performance improvements in the JVM that are still experimental in a given release .
UseFastAccessorMethodsInlines getXXX() methods for primitives .
Controls when HotSpot compiles code into native bytecode. High numbers mean faster start up time, low numbers mean the code is optimized sooner. For a server, a low number may be desirable .
Allows threads to have an affinity with a lock they frequently use. Makes attaining that lock faster .
There are many others but from reading the literature, these looked likely candidates.
Unfortunately I saw no improvements on my machine. This last point is important. Currently, we have a shortage of QA machines so I had to try the tests on my Windows machine. I protested that this was not an accurate way of doing things. For instance, adding the –server JVM arg massively improved performance. I was quite excited until I realized that for multi-core Linux machines such as our production boxes this is set as default. On 32-bit Windows machines, it is not .
So, I gave up on JVM tuning and looked at Tomcat config. The obvious first choice was to use NIO (non blocking IO). This is easy. You just make your connector in server.xml look like this:
But although this massively reduced the number of threads from roughly one thread per connection to about half a dozen, it didn't seem to speed things up.
Time to fire up the profiler of choice, YourKit. Under normal load, the threads look like this:
So, I looked at the thread the third party library gave us and it looks like this:
So, we have a bottle neck that's nothing to do with Tomcat. Tomorrow, I am going to rip out this in-house library as it's caused us nothing but pain and go back to Java RMI. Then, let's see how much we can tune Tomcat.