Message ordering in disconnected Remote Actors

I spotted a curious little Remote Actors behavioural bug recently, mostly by chance.

Consider two remote actors, a source sending messages to a sink. If the source is alive prior to the sink, then the source can still send (asynchronous) messages. When the sink is started, the source will finally be able to connect, and messages will be passed. Personally, I think it would be entirely reasonable for messages prior to the sink’s startup to be dropped. Scala’s Remote Actors queues the messages instead.

If messages are to be queued, however, I would fully expect those messages to arrive at the sink in FIFO order. What actually happens is that they arrive in LIFO order, i.e., entirely in reverse. This goes against my assumptions as a developer, given the behaviour of Scala message passing in all other circumstances.

The reason the messages arrive in reverse order is quite simple: the buffer is a Scala list, constructed by attaching each new item as the head of the list. When the sink comes alive, the TcpService running deep within the source is able to successfully connect(), and the message that prompted the connect() is sent. Then, the buffer is drained, by iteratively sending the head of the list until it is empty. Efficient, but surely backwards.

The fix is simple: when a connect() succeeds, reverse and then drain the buffer, and then send the message that prompted the successful connect. Thus, messages are buffered and arrive in FIFO order.

It’s a trivial little behavioural bug, but it could catch somebody out. The fix is simple; I submitted a bug report and a patch.

This post is most closely related to my other posts on Remote Actors: 1 2.