JConsole is a funny little beast. We had problems connecting it to our remote application from our development machines. While it tried to connect, a quick glance at the current state of active sockets on our desktop showed:
SYN_SENT "represents waiting for a matching connection request after having sent a connection request" [1]. In a healthy situation, this state should transition to ESTABLISHED upon receiving a SYN/ACK and subsequently sending an ACK packet (see p22 of the RFC).
However, nothing came back. This may indicate a firewall problem yet the system administrators confirmed that they had opened the port that we specified on the command line [2] with:
-Dcom.sun.management.jmxremote.port=3000
Having said that, upon SSHing into the remote box, we'd run something like:
09:43:46 solo:~/trunk/sns-conduit dev$ lsof -p `jps | grep CustomAgent | sed -e s/\ .*//g` -P | grep TCP
java 2388 dev 5u IPv6 0x731d19c 0t0 TCP *:51652 (LISTEN)
java 2388 dev 6u IPv6 0x731cf38 0t0 TCP localhost:51651->localhost:51650 (TIME_WAIT)
java 2388 dev 7u IPv6 0x731d664 0t0 TCP *:3000 (LISTEN)
java 2388 dev 8u IPv6 0x731db2c 0t0 TCP *:51653 (LISTEN)
(where, say, CustomAgent was the name of our application and the -P flag to lsof tells it not to resolve port numbers in our environment [3]).
So, the port we defined is up and LISTENing. But not only is the port we specified in our command line LISTENing, there appear to be other ports LISTENing too.
By putting breakpoints in the constructor of ServerSocket, we could find out who was opening these ports.
1. The JMX RMIServer for Remote Connections
The first comes from the JVM bootstrap classes staring a JMX RMIServerImpl object. In turn, this tries to export itself via RMI on port 0 (that is, asking the operating system to choose an arbitrary free port for itself). By using lsof, we can see the value of this OS-assigned port - 51652 in the example I am currently playing with.
2. The RMI Registry
The second hit of the breakpoint indicates it is about to start listening on the port we defined in the command line (that is, 3000). This call is originating in Sun code, the source code of which does not come with the JDK although you can see it if you go to the OpenJDK site (or check it out with something like hg clone http://hg.openjdk.java.net/jdk7/jdk7). But it is apparent from the stack that something RMI related is being started. It turns out that it is an RMI Registry as executing this code with the port of 3000 as an argument demonstrates:
package com.henryp.rmi;
import java.rmi.Remote;
import java.rmi.RemoteException;
import java.rmi.registry.LocateRegistry;
import java.rmi.registry.Registry;
import java.util.HashSet;
import java.util.Set;
public class RmiRegistryLister {
/**
* @param args
*/
public static void main(String[] args) {
RmiRegistryLister app = new RmiRegistryLister();
app.listRegistryOnLocalPort(Integer.parseInt(args[0]));
}
private void listRegistryOnLocalPort(int port) {
try {
Registry registry = LocateRegistry.getRegistry(port);
String[] entries = registry.list();
for (String entry : entries) {
Remote value = registry.lookup(entry);
System.out.println("Key = " + entry + ", value = " + value);
printAllInterfaces(value);
}
} catch (Exception e) {
e.printStackTrace();
}
}
private void printAllInterfaces(Object obj) {
Set alreadSeen = new HashSet();
printAllInterfaces(obj, alreadSeen);
}
private void printAllInterfaces(Object obj, Set alreadSeen) {
Class[] interfaces = obj.getClass().getInterfaces();
for (Class anInterface : interfaces) {
if (!alreadSeen.contains(anInterface)) {
System.out.println(anInterface.getName());
alreadSeen.add(anInterface);
printAllInterfaces(anInterface, alreadSeen);
}
}
}
}
The output from this looks something like:
Key = jmxrmi, value = RMIServerImpl_Stub[UnicastRef2 [liveRef: [endpoint:[192.168.0.3:51652,javax.rmi.ssl.SslRMIClientSocketFactory@8e3e60](remote),objID:[-196ac51:1391aafa8b6:-8000, -8281973287725425859]]]]
javax.management.remote.rmi.RMIServer
java.io.Serializable
java.lang.reflect.GenericDeclaration
java.lang.reflect.Type
java.lang.reflect.AnnotatedElement
So, it looks like the registry contains a stub (remote proxy) to a JMX RMIServer where calling methods on it would be routed through to port 51652 - the port that we started LISTENing to in step 1.
3. Local Connector Server
The final time we hit the ServerSocket breakpoint reveals a stack that looks a lot like the first time we hit it. Indeed, we're exporting another JMX RMIServer. This particular instance is JMX's "local connector server". Connecting jConsole to a JVM running locally uses this RMI server. You can demonstrate this by doing exactly that and then using lsof to show that both the jConsole process and the application are connected to each other on this port. We're not terribly interested in this port.
So, this explains why we can't connect through our firewall. We can see the RMI Registry in which we find the JMX RMIServer stub but on using it, the firewall blocks us.
There are ways around this, but nothing terribly simple. One good idea is here. This requires an appreciation of the JMXServiceURL class which gives us a good idea of how to read these JMX URLs. Take:
service:jmx:rmi://solo/jndi/rmi://solo/jmxrmi
No ports are defined so starting any servers will use OS-assigned, undefined ports.
Adding the RMI port so:
service:jmx:rmi://solo/jndi/rmi://solo:3000/jmxrmi
is a start but note that (i) you need to have started an RMI Registry on port 3000 with something like:
LocateRegistry.createRegistry(3000);
And note (ii) that the port from calling jmxServiceURL.getPort() is still 0! To demonstrate:
@Test
public void testNoJmxPortGivesPortOf0() throws Exception {
JMXServiceURL jmxServiceURL = new JMXServiceURL("service:jmx:rmi://solo/jndi/rmi://solo:3000/jmxrmi");
assertEquals(0, jmxServiceURL.getPort());
}
This is because we have only defined the registry port. You can define on which port the RMIServer is actually servicing requests with, say:
service:jmx:rmi://solo:2000/jndi/rmi://solo:3000/jmxrmi
And, to demonstrate:
@Test
public void testWhichPort() throws Exception {
int jmxPort = 2000;
int rmiPort = 3000;
String url = String.format("service:jmx:rmi://solo:%d/jndi/rmi://solo:%d/jmxrmi", jmxPort, rmiPort);
JMXServiceURL jmxServiceURL = new JMXServiceURL(url);
assertEquals(jmxPort, jmxServiceURL.getPort());
}
It's the second part of the URL that is used first. It defines where we find the registry that points to the RMIServer in the first part. Again, to illustrate:
@Test
public void testUrlPath() throws Exception {
String urlPath = "/jndi/rmi://solo:3000/jmxrmi";
String fixedUrlPartPlusProtocol = "service:jmx:rmi://solo:2000";
JMXServiceURL jmxServiceURL = new JMXServiceURL(fixedUrlPartPlusProtocol + urlPath);
assertEquals(urlPath, jmxServiceURL.getURLPath());
}
Together, you have the means to connect to a JMX Server via RMI but both have to be accessible if you want to use jConsole.
[Aside: somewhat annoyingly, it does not appear to be possible to start jConsole on the command line with a username and password as part of the jmxUrl. The JavaDocs say: "This address uses a subset of the syntax defined by RFC 2609 for IP-based protocols. It is a subset because the user@host syntax is not supported". Instead, you have to type it into the GUI when it starts.]
[1] RFC-793
[2] Monitoring and Management Using JMX
[3] the environment I was working on can be described as below:
16:53:57 solo:~/trunk/sns-conduit dev$ java -version
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode)
17:01:51 solo:~/trunk/sns-conduit dev$ uname -a
Darwin solo 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64
15:59:15 solo:~/trunk/hermes dev$ netstat -na | grep 10.244.144.67
tcp4 0 0 10.240.224.177.53124 10.244.144.67.11191 SYN_SENT
SYN_SENT "represents waiting for a matching connection request after having sent a connection request" [1]. In a healthy situation, this state should transition to ESTABLISHED upon receiving a SYN/ACK and subsequently sending an ACK packet (see p22 of the RFC).
However, nothing came back. This may indicate a firewall problem yet the system administrators confirmed that they had opened the port that we specified on the command line [2] with:
-Dcom.sun.management.jmxremote.port=3000
Having said that, upon SSHing into the remote box, we'd run something like:
09:43:46 solo:~/trunk/sns-conduit dev$ lsof -p `jps | grep CustomAgent | sed -e s/\ .*//g` -P | grep TCP
java 2388 dev 5u IPv6 0x731d19c 0t0 TCP *:51652 (LISTEN)
java 2388 dev 6u IPv6 0x731cf38 0t0 TCP localhost:51651->localhost:51650 (TIME_WAIT)
java 2388 dev 7u IPv6 0x731d664 0t0 TCP *:3000 (LISTEN)
java 2388 dev 8u IPv6 0x731db2c 0t0 TCP *:51653 (LISTEN)
(where, say, CustomAgent was the name of our application and the -P flag to lsof tells it not to resolve port numbers in our environment [3]).
So, the port we defined is up and LISTENing. But not only is the port we specified in our command line LISTENing, there appear to be other ports LISTENing too.
By putting breakpoints in the constructor of ServerSocket, we could find out who was opening these ports.
1. The JMX RMIServer for Remote Connections
The first comes from the JVM bootstrap classes staring a JMX RMIServerImpl object. In turn, this tries to export itself via RMI on port 0 (that is, asking the operating system to choose an arbitrary free port for itself). By using lsof, we can see the value of this OS-assigned port - 51652 in the example I am currently playing with.
2. The RMI Registry
The second hit of the breakpoint indicates it is about to start listening on the port we defined in the command line (that is, 3000). This call is originating in Sun code, the source code of which does not come with the JDK although you can see it if you go to the OpenJDK site (or check it out with something like hg clone http://hg.openjdk.java.net/jdk7/jdk7). But it is apparent from the stack that something RMI related is being started. It turns out that it is an RMI Registry as executing this code with the port of 3000 as an argument demonstrates:
package com.henryp.rmi;
import java.rmi.Remote;
import java.rmi.RemoteException;
import java.rmi.registry.LocateRegistry;
import java.rmi.registry.Registry;
import java.util.HashSet;
import java.util.Set;
public class RmiRegistryLister {
/**
* @param args
*/
public static void main(String[] args) {
RmiRegistryLister app = new RmiRegistryLister();
app.listRegistryOnLocalPort(Integer.parseInt(args[0]));
}
private void listRegistryOnLocalPort(int port) {
try {
Registry registry = LocateRegistry.getRegistry(port);
String[] entries = registry.list();
for (String entry : entries) {
Remote value = registry.lookup(entry);
System.out.println("Key = " + entry + ", value = " + value);
printAllInterfaces(value);
}
} catch (Exception e) {
e.printStackTrace();
}
}
private void printAllInterfaces(Object obj) {
Set
printAllInterfaces(obj, alreadSeen);
}
private void printAllInterfaces(Object obj, Set
Class[] interfaces = obj.getClass().getInterfaces();
for (Class anInterface : interfaces) {
if (!alreadSeen.contains(anInterface)) {
System.out.println(anInterface.getName());
alreadSeen.add(anInterface);
printAllInterfaces(anInterface, alreadSeen);
}
}
}
}
The output from this looks something like:
Key = jmxrmi, value = RMIServerImpl_Stub[UnicastRef2 [liveRef: [endpoint:[192.168.0.3:51652,javax.rmi.ssl.SslRMIClientSocketFactory@8e3e60](remote),objID:[-196ac51:1391aafa8b6:-8000, -8281973287725425859]]]]
javax.management.remote.rmi.RMIServer
java.io.Serializable
java.lang.reflect.GenericDeclaration
java.lang.reflect.Type
java.lang.reflect.AnnotatedElement
So, it looks like the registry contains a stub (remote proxy) to a JMX RMIServer where calling methods on it would be routed through to port 51652 - the port that we started LISTENing to in step 1.
3. Local Connector Server
The final time we hit the ServerSocket breakpoint reveals a stack that looks a lot like the first time we hit it. Indeed, we're exporting another JMX RMIServer. This particular instance is JMX's "local connector server". Connecting jConsole to a JVM running locally uses this RMI server. You can demonstrate this by doing exactly that and then using lsof to show that both the jConsole process and the application are connected to each other on this port. We're not terribly interested in this port.
So, this explains why we can't connect through our firewall. We can see the RMI Registry in which we find the JMX RMIServer stub but on using it, the firewall blocks us.
There are ways around this, but nothing terribly simple. One good idea is here. This requires an appreciation of the JMXServiceURL class which gives us a good idea of how to read these JMX URLs. Take:
service:jmx:rmi://solo/jndi/rmi://solo/jmxrmi
No ports are defined so starting any servers will use OS-assigned, undefined ports.
Adding the RMI port so:
service:jmx:rmi://solo/jndi/rmi://solo:3000/jmxrmi
is a start but note that (i) you need to have started an RMI Registry on port 3000 with something like:
LocateRegistry.createRegistry(3000);
And note (ii) that the port from calling jmxServiceURL.getPort() is still 0! To demonstrate:
@Test
public void testNoJmxPortGivesPortOf0() throws Exception {
JMXServiceURL jmxServiceURL = new JMXServiceURL("service:jmx:rmi://solo/jndi/rmi://solo:3000/jmxrmi");
assertEquals(0, jmxServiceURL.getPort());
}
This is because we have only defined the registry port. You can define on which port the RMIServer is actually servicing requests with, say:
service:jmx:rmi://solo:2000/jndi/rmi://solo:3000/jmxrmi
And, to demonstrate:
@Test
public void testWhichPort() throws Exception {
int jmxPort = 2000;
int rmiPort = 3000;
String url = String.format("service:jmx:rmi://solo:%d/jndi/rmi://solo:%d/jmxrmi", jmxPort, rmiPort);
JMXServiceURL jmxServiceURL = new JMXServiceURL(url);
assertEquals(jmxPort, jmxServiceURL.getPort());
}
It's the second part of the URL that is used first. It defines where we find the registry that points to the RMIServer in the first part. Again, to illustrate:
@Test
public void testUrlPath() throws Exception {
String urlPath = "/jndi/rmi://solo:3000/jmxrmi";
String fixedUrlPartPlusProtocol = "service:jmx:rmi://solo:2000";
JMXServiceURL jmxServiceURL = new JMXServiceURL(fixedUrlPartPlusProtocol + urlPath);
assertEquals(urlPath, jmxServiceURL.getURLPath());
}
Together, you have the means to connect to a JMX Server via RMI but both have to be accessible if you want to use jConsole.
[Aside: somewhat annoyingly, it does not appear to be possible to start jConsole on the command line with a username and password as part of the jmxUrl. The JavaDocs say: "This address uses a subset of the syntax defined by RFC 2609 for IP-based protocols. It is a subset because the user@host syntax is not supported". Instead, you have to type it into the GUI when it starts.]
[1] RFC-793
[2] Monitoring and Management Using JMX
[3] the environment I was working on can be described as below:
16:53:57 solo:~/trunk/sns-conduit dev$ java -version
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode)
17:01:51 solo:~/trunk/sns-conduit dev$ uname -a
Darwin solo 11.4.0 Darwin Kernel Version 11.4.0: Mon Apr 9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64 x86_64
No comments:
Post a Comment