Over a million developers have joined DZone.

Using Akka Remoting for Building a Distributed Application: Addendum

DZone's Guide to

Using Akka Remoting for Building a Distributed Application: Addendum

· DevOps Zone ·
Free Resource

Learn more about how CareerBuilder was able to resolve customer issues 5x faster by using Scalyr, the fastest log management tool on the market. 

This post was originally written by Byron Shelden.

A couple weeks ago I wrote a blog about my experience using Akka remoting for building a distributed application titled: Why blocking is bad in Akka remoting … really bad. To my surprise people actually read my blog (crazy, I know!). One of those people was none other than Dr. Roland Kuhn (@rolandkuhn), Akka Tech Lead at Typesafe. Yep, Mr. Akka himself read my blog and sent me an e-mail.

He asked what precisely the problems were which brought me to the conclusion that “Akka is not suitable for communications over an untrusted network because it’s less reliable once latency gets involved”. He noted that: “One of the design goals is precisely that actor communication fits the network very well, so it would help us to understand what exactly the difficulties were that your team encountered”. Though I was in a coding crunch making sure we got our Enterprise (on-prem) Edition out the door on time for our customers, I was a bit terrified that everything I had said was wrong, and I just sucked at using Akka, so I replied with a hurried brain dump mid bug fix.

I realized that I had in fact dropped the ball in the editing process. The statement he referenced, “However, it’s not suitable for communications over an untrusted network because it’s less reliable once latency gets involved” should actually have been in regards to why we didn’t use it for the remote agents, not as an overall thought (which I corrected). We did, however, find it to be true, but it was only part of the picture. We found Akka remoting to be basically unsuited to remote connections to networks we don’t control. For context, Akka was in 1.x at the time these decisions were made, and I wasn’t sure how Akka had evolved in the mean time. The reasons were:

1) Without using ask, I couldn’t find a way to get information on if a message was delivered. So we were left with the situation where we needed an infinite number of ACK type messages.

Roland pointed out that this is fundamental: without the other side replying you cannot possibly be sure that the message has arrived. This aspect can be factored out, though, if you are willing to trust local delivery—which can also fail but for which the probability to do so is the same as for not actually executing a method call. In Akka 2.1 they introduced the ReliableProxy in the akka-contrib package; this is currently being revamped and will probably be moved into akka-persistence. The purpose is to do the ACKing for you, giving you a reliable tunnel from one system to another.

2) We cannot control the network environment for the remote agents. No promise that we could open sockets in, and no promises we didn’t have NAT involved. Akka demands bi-directional communications, and for IPs to be mutually known and bindable.

Roland offered that Akka remoting has been completely rewritten for 2.2 and is quite a bit more reliable, supports proper DeathWatch and so on, but it still is designed for peer-to-peer networking: everybody needs to be able to directly talk to everybody else, otherwise ActorRef would lose its location transparency.

3) The round trips could easily take longer than expected, and if I used ask to avoid (1), timeouts became an issue.

Not much else to say about this one. It is what it is.

So how much of this still applies? Can modern Akka handle this use case at all? According to Roland Akka is being used in such deployments despite the caveats, but he admits that it is not optimal, because it was never designed for the asymmetric client–server case. A better solution would be to use a transport protocol which is optimized for this, like HTTP (REST; they’re in the process of absorbing Spray as akka-http) or rolling your own based on the akka.io package.

I feel pretty good about the decision not to use it, and very much appreciate Roland getting in touch with me (thanks Roland!). Hopefully this will end up being a helpful blog for someone out there considering using Akka for the asymmetric client–server case.

Oh, and we did release our fancy new Enterprise Edition. The marketing guy wants me to say that if you’d find an on-prem tool that makes it quick and super easy to diff files and packages: across the servers in a cluster; between dev, test, and prod environments; etc., you should check it out. Don’t mind him, but it is chock-full of Scala goodness!

Find out more about how Scalyr built a proprietary database that does not use text indexing for their log management tool.


Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}