Sunday, August 12, 2012

JConsole and Firewalls

JConsole is a funny little beast. We had problems connecting it to our remote application from our development machines. While it tried to connect, a quick glance at the current state of active sockets on our desktop showed:

15:59:15 solo:~/trunk/hermes dev$  netstat -na | grep
tcp4       0      0    SYN_SENT

SYN_SENT "represents waiting for a matching connection request after having sent a connection request" [1]. In a healthy situation, this state should transition to ESTABLISHED upon receiving a SYN/ACK and subsequently sending an ACK packet (see p22 of the RFC).

However, nothing came back. This may indicate a firewall problem yet the system administrators confirmed that they had opened the port that we specified on the command line [2] with:

Having said that, upon SSHing into the remote box, we'd run something like:

09:43:46 solo:~/trunk/sns-conduit dev$ lsof -p `jps | grep CustomAgent | sed -e s/\ .*//g` -P | grep TCP 

java    2388 dev    5u  IPv6 0x731d19c       0t0       TCP *:51652 (LISTEN)
java    2388 dev    6u  IPv6 0x731cf38       0t0       TCP localhost:51651->localhost:51650 (TIME_WAIT)
java    2388 dev    7u  IPv6 0x731d664       0t0       TCP *:3000 (LISTEN)
java    2388 dev    8u  IPv6 0x731db2c       0t0       TCP *:51653 (LISTEN)

(where, say, CustomAgent was the name of our application and the -P flag to lsof tells it not to resolve port numbers in our environment [3]).

So, the port we defined is up and LISTENing. But not only is the port we specified in our command line LISTENing, there appear to be other ports LISTENing too.

By putting breakpoints in the constructor of ServerSocket, we could find out who was opening these ports.

1. The JMX RMIServer for Remote Connections

The first comes from the JVM bootstrap classes staring a JMX RMIServerImpl object. In turn, this tries to export itself via RMI on port 0 (that is, asking the operating system to choose an arbitrary free port for itself). By using lsof, we can see the value of this OS-assigned port - 51652 in the example I am currently playing with.

2. The RMI Registry

The second hit of the breakpoint indicates it is about to start listening on the port we defined in the command line (that is, 3000). This call is originating in Sun code, the source code of which does not come with the JDK although you can see it if you go to the OpenJDK site (or check it out with something like hg clone But it is apparent from the stack that something RMI related is being started. It turns out that it is an RMI Registry as executing this code with the port of 3000 as an argument demonstrates:

package com.henryp.rmi;

import java.rmi.Remote;
import java.rmi.RemoteException;
import java.rmi.registry.LocateRegistry;
import java.rmi.registry.Registry;
import java.util.HashSet;
import java.util.Set;

public class RmiRegistryLister {
     * @param args
    public static void main(String[] args) {
        RmiRegistryLister app = new RmiRegistryLister();

    private void listRegistryOnLocalPort(int port) {
        try {
            Registry registry = LocateRegistry.getRegistry(port);
            String[] entries = registry.list();
            for (String entry : entries) {
                Remote value = registry.lookup(entry);
                System.out.println("Key = " + entry + ", value = " + value);
        } catch (Exception e) {

    private void printAllInterfaces(Object obj) {
        Set alreadSeen = new HashSet();
        printAllInterfaces(obj, alreadSeen);

    private void printAllInterfaces(Object obj, Set alreadSeen) {
        Class[] interfaces = obj.getClass().getInterfaces();
        for (Class anInterface : interfaces) {
            if (!alreadSeen.contains(anInterface)) {
                printAllInterfaces(anInterface, alreadSeen);

The output from this looks something like:

Key = jmxrmi, value = RMIServerImpl_Stub[UnicastRef2 [liveRef: [endpoint:[,javax.rmi.ssl.SslRMIClientSocketFactory@8e3e60](remote),objID:[-196ac51:1391aafa8b6:-8000, -8281973287725425859]]]]

So, it looks like the registry contains a stub (remote proxy) to a JMX RMIServer where calling methods on it would be routed through to port 51652 - the port that we started LISTENing to in step 1.

3. Local Connector Server

The final time we hit the ServerSocket breakpoint reveals a stack that looks a lot like the first time we hit it. Indeed, we're exporting another JMX RMIServer. This particular instance is JMX's "local connector server". Connecting jConsole to a JVM running locally uses this RMI server. You can demonstrate this by doing exactly that and then using lsof to show that both the jConsole process and the application are connected to each other on this port. We're not terribly interested in this port.

So, this explains why we can't connect through our firewall. We can see the RMI Registry in which we find the JMX RMIServer stub but on using it, the firewall blocks us.

There are ways around this, but nothing terribly simple. One good idea is here. This requires an appreciation of the JMXServiceURL class which gives us a good idea of how to read these JMX URLs. Take:


No ports are defined so starting any servers will use OS-assigned, undefined ports.

Adding the RMI port so:


is a start but note that (i) you need to have started an RMI Registry on port 3000 with something like:


And note (ii) that the port from calling jmxServiceURL.getPort() is still 0! To demonstrate:

    public void testNoJmxPortGivesPortOf0() throws Exception {
        JMXServiceURL jmxServiceURL = new JMXServiceURL("service:jmx:rmi://solo/jndi/rmi://solo:3000/jmxrmi");
        assertEquals(0, jmxServiceURL.getPort());

This is because we have only defined the registry port. You can define on which port the RMIServer is actually servicing requests with, say:


And, to demonstrate:

    public void testWhichPort() throws Exception {
        int             jmxPort         = 2000;
        int             rmiPort         = 3000;
        String          url             = String.format("service:jmx:rmi://solo:%d/jndi/rmi://solo:%d/jmxrmi", jmxPort, rmiPort);
        JMXServiceURL   jmxServiceURL   = new JMXServiceURL(url);
        assertEquals(jmxPort, jmxServiceURL.getPort());

It's the second part of the URL that is used first. It defines where we find the registry that points to the RMIServer in the first part. Again, to illustrate:

    public void testUrlPath() throws Exception {
        String          urlPath                  = "/jndi/rmi://solo:3000/jmxrmi";
        String          fixedUrlPartPlusProtocol = "service:jmx:rmi://solo:2000";
        JMXServiceURL   jmxServiceURL            = new JMXServiceURL(fixedUrlPartPlusProtocol + urlPath);
        assertEquals(urlPath, jmxServiceURL.getURLPath());

Together, you have the means to connect to a JMX Server via RMI but both have to be accessible if you want to use jConsole.

[Aside: somewhat annoyingly, it does not appear to be possible to start jConsole on the command line with a username and password as part of the jmxUrl. The JavaDocs say: "This address uses a subset of the syntax defined by RFC 2609 for IP-based protocols. It is a subset because the user@host syntax is not supported". Instead, you have to type it into the GUI when it starts.]

[1] RFC-793
[2] Monitoring and Management Using JMX
[3] the environment I was working on can be described as below:

16:53:57 solo:~/trunk/sns-conduit dev$ java -version
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode)
17:01:51 solo:~/trunk/sns-conduit dev$ uname -a
Darwin solo 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr  9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64

No comments:

Post a Comment