JBoss – Changing RMI Remote Client Callback Address

Recently during a JBoss production deployment (4.2.3.GA) that I had to carry on, I came across a problem with RMI Remoting (EJB3), which gave an exception when remote EJBs are invoked. The exception I got was ‘java.lang.IllegalArgumentException: port out of range:-1′, whenever a Remote EJB call was made.

12:57:58,354 WARN  [ServiceExceptionTranslatorAspect] Unable to Translate Exception : org.jboss.remoting.CannotConnectException
12:57:58,402 ERROR [[default]] Servlet.service() for servlet default threw exception
java.lang.IllegalArgumentException: port out of range:-1
at java.net.InetSocketAddress.<init>(InetSocketAddress.java:118)
at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:183)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:827)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:569)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
at org.jboss.remoting.Client.invoke(Client.java:1634)
at org.jboss.remoting.Client.invoke(Client.java:548)
at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:74)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
at org.jboss.ejb3.stateless.StatelessRemoteProxy.invoke(StatelessRemoteProxy.java:107)

The application was working fine in our local environments, and this problem happened only in production. This deployment consisted of two JBoss AS instances running two applications, which communicates with each other via EJB Remoting.

Googling did not give me any positive lead, and I was stuck on this problem for a while. So I tried out the only option I was left with, trying to go to detailed log output to see what’s happening under the hood. After changing the log level for CONSOLE and org.jboss category to DEBUG level (if you are new to JBoss, you can change the logging mechanism by modifying jboss-log4j.xml in conf directory of server profile).

This gave me the following output.

12:21:41,049 DEBUG [AuthenticationContextInterceptor] AuthenticationContextInterceptor Intercepting EJB Invocation...
12:21:41,049 DEBUG [AuthenticationContextInterceptor] Setting the authenticated user details in Authentication Context: XXXX
12:21:41,061 DEBUG [AuthenticationContextInterceptor] AuthenticationContextInterceptor Intercepting EJB Invocation...
12:21:41,061 DEBUG [AuthenticationContextInterceptor] Setting the authenticated user details in Authentication Context: XXXX
12:21:43,994 DEBUG [MicroSocketClientInvoker] SocketClientInvoker[b874d2, socket://192.168.12.4:-1] constructed
12:21:43,994 DEBUG [MicroRemoteClientInvoker] SocketClientInvoker[b874d2, socket://192.168.12.4:-1] connecting
12:21:43,994 DEBUG [MicroSocketClientInvoker] Creating semaphore with size 50
12:21:43,994 DEBUG [MicroRemoteClientInvoker] SocketClientInvoker[b874d2, socket://192.168.12.4:-1] connected
12:21:43,995 DEBUG [InvokerRegistry] removed SocketClientInvoker[b874d2, socket://192.168.12.4:-1] from registry
12:21:43,995 DEBUG [MicroSocketClientInvoker] SocketClientInvoker[b874d2, socket://192.168.12.4:-1] disconnecting ...
12:21:44,946 ERROR [[default]] Servlet.service() for servlet default threw exception
java.lang.IllegalArgumentException: port out of range:-1

As the highlighted line shows, JBoss was trying to communicate to the remote EJB via port -1, which is of course invalid. Also, an interesting observation was that the IP address which JBoss tries to use is not the one I expected it to use. This, was actually due to the network setup of the production environment. The production server in question had two network cards, each having local IPs  192.168.12.4 and 192.168.12.5. The 192.168.12.4 NIC was used to expose the server to Internet via NAT. 192.168.12.5 was used to internally connect to the machine via VPN for maintenance, etc. Since 192.168.12.4 was used for NAT, it was restricted to HTTP traffic on port 8080 only.

So the problem that I was facing was that JBoss was using the 192.168.12.4 to refer to the remote EJB, where as I expected it to use 192.168.12.5 (note that we have to start JBoss bound to all addresses using -b 0.0.0.0 because we access it via multiple NICs). Since JBoss could not get a free port on 192.168.12.4, it was falling back to port -1.

So I wanted to find out a way to force JBoss to use the IP address I wanted for remote EJB calls (without binding it specifically to one IP using -b). I was already referring to the remote server’s JNDI Registry via the IP I expected (192.168.12.5). After trying out various options, finally I found the following in the deploy folder of JBoss.

File: <JBOSS_HOME>/server/xxxx/deploy/ejb3.deployer/META-INF/jboss-service.xml

<mbean code="org.jboss.remoting.transport.Connector"
name="jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3">
<depends>jboss.aop:service=AspectDeployer</depends>
<attribute name="InvokerLocator">socket://${jboss.bind.address}:13873</attribute>
<attribute name="Configuration">
<handlers>
<handler subsystem="AOP">org.jboss.aspects.remoting.AOPRemotingInvocationHandler</handler>
</handlers>
</attribute>
</mbean>

I changed it to the following, so that instead of dynamically resolving the IP address, JBoss will use the IP I wanted when remote client callbacks are created.

<mbean code="org.jboss.remoting.transport.Connector"
name="jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3">
<depends>jboss.aop:service=AspectDeployer</depends>
<attribute name="InvokerLocator">socket://192.168.12.5:13873</attribute>
<attribute name="Configuration">
<handlers>
<handler subsystem="AOP">org.jboss.aspects.remoting.AOPRemotingInvocationHandler</handler>
</handlers>
</attribute>
</mbean>

With this in place, the client was able to invoke the remote EJBs without any issues. This is a rare situation, but if anyone else face the same issue, I guess this post would help to get it sorted out.

From http://blog.yohanliyanage.com/2011/05/jboss-changing-rmi-callback-address/