tomcat - How do I fix a 'Failed to retrieve RMIServer stub' JMX error?

I just completed a migration from Windows server 2008 R2 to Ubuntu 10.04. I manage a java application (Java 6, Tomcat) that is having some performance issues. I'd like to use JMX to try and troubleshoot, but I can't seem to get jvisualvm to connect.

If I do a ps -ef | grep "java", I see the following parameters.

-Dcom.sun.management.jmxremote.port=8084 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false

Netstat shows that port 8084 is listening on 0.0.0.0. In my config, JMX is setup to bind to the FQDN of the server (we use a private DNS server). My firewall (IPTABLES/UFW) is setup to allow all outgoing traffic, and to allow incoming traffic on port 8084.

The server itself is virtual with two NICs, a public and a private. The public NIC's gateway is disabled so that connections can only come in on the private side.

When I try to connect jvisualvm to my app server using JMX, I get the following error in jvisualvm.

Cannot connect to [FQDN OMITTED]:8084 using server:jmx:rmi:///jndi/rmi://[FQDN OMITTED]:8084/jmxrmi

If I look in the jvisualvm log, I see the following trace.

NFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(service:jmx:rmi:///jndi/rmi://[FQDN OMITTED]:8084/jmxrmi)
java.io.EOFException: SSL peer shut down incorrectly

                at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
Caused: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:808)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1120)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:623)
                at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
                at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                at java.io.DataOutputStream.flush(DataOutputStream.java:106)

                at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:211)
Caused: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
                javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
                at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
                at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
                at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
                at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
                at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
Caused: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
                javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]

                at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
                at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
                at javax.naming.InitialContext.lookup(InitialContext.java:392)
                at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
                at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
                at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
Caused: java.io.IOException: Failed to retrieve RMIServer stub
                at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
                at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:451)
[catch] at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:395)

                at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:216)
                at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.(JmxModelImpl.java:205)
                at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:61)
                at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:42)
                at com.sun.tools.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:111)
                at com.sun.tools.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69)
                at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.addJmxApplication(JmxApplicationProvider.java:267)
                at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.createJmxApplication(JmxApplicationProvider.java:185)
                at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationImpl(JmxApplicationsSupport.java:283)
                at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationInteractive(JmxApplicationsSupport.java:261)

                at com.sun.tools.visualvm.jmx.impl.AddJMXConnectionAction$1.run(AddJMXConnectionAction.java:80)
                at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:577)
                at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1030)

Does anyone have any ideas?

Answer

The problem is that even if JMX is configured to listen on port 8084, after the initial connection is made, the JMX host will dynamically pick a different port for the rest of the "conversation". My firewall allowed traffic through 8084, but was blocking the subsequent traffic.

There are two options.

1. Allow inbound traffic on all ephemeral ports (yikes!).


2. Develop a JMX wrapper that uses a single port for communication.