Assembling requests from multiple actors

Akka actor systems are built on groups of actors that communicate via messages in an asynchronous way, while each actor (or group of actors) has its own responsibility. But what if an actor not only depends on the results of another actor, but on that of two or more?

In one of our optimization systems in Kenshoo, we have a reporting module that, based on input from the user, calculates parameters, plots a trajectory, and generates a prediction report which is then available to the user.

So we have:

1. An actor that calculates the parameters
2. An actor that plots the trajectory (independent of the calculated parameters)
3. An actor that generates the report based on the results of the first two actors
4. An actor that orchestrates this flow

In order to generate the report, we need an actor that waits for both the parameters and for the trajectory. The actor that produces our desired result — the report — depends on the results of two other actors, which isn’t trivial.

The actor model does not guarantee message order. In addition, one of the key features of the model is that an actor can process only one message at a time. These two features can be great for concurrent computation (for example, we don’t need to apply any locks on any resource), but assembling requests from multiple actors becomes non-trivial. In a simple procedural system, we’d call one service to get the parameters and then (if completed successfully) call another service to get the plotted trajectory. When that’s complete, we know we have everything we need to generate the report. But if we want to compute the two pieces independently using actors — we can’t (or shouldn’t) wait for one actor’s response before calling the other.

So how can we create dependency between our actors?

Solving the problem with ask pattern

The ask pattern (also known as ?) lets the sender actor wait for a response from a receiving actor asynchronously. This is a fancy way of saying that Akka would wrap the response in a Future.

One way of defining the dependency between our actors is by adding a condition that will force the actor to wait for the CalculateDataActor and DrawPlotActor responses, which it can then send to the CreateReportFromTemplateActor.

It looks like this:

This looks pretty simple, and it does get the job done, but there are some problems with this code.

First of all, the ? method returns Future[Any] which is not type-safe, which means that we’ll have to use .mapTo[ResponseCalculateData] on its result. Although this is a known problem in Akka (solved by Akka Typed), it’s magnified when using the ask pattern. When you are solving the type problem as part of the receive behavior, you can control the actor behavior (you can log it, you can throw an exception, you can wrap it with another message and send it to another actor, and much more…) but when using ask, you can either use the concise but unsafe mapTo[A], or end up with some boilerplate-heavy code to handle failures.

The second problem is the infamous AskTimeOutException. If the Future result times out, Akka will throw an AskTimeOutException which gives you no clues as to what went wrong — you can’t tell if you had an exception along the way, or if it just took longer than expected. And good luck debugging that!

Solving the problem with a dedicated actor

The dedicated actor pattern allows us to create a new actor at runtime that is dedicated to handle one request. We used this pattern, and the tell method to override the default sender, to gather one result from the CalculateDataActor and one result from the DrawPlotActor into one dedicated actor. Since each request has its own dedicated actor that waits for its relevant results, the flow isn’t blocking.

The dedicated actor looks something like this:

And the ReportGeneratorActor looks something like this:

The ReportGeneratorActor creates the ReportDedicatedActor, and uses the tellmethod to explicitly set the sender -

calculateDataActor.tell(CalculateData(message.userData), reportDedicatedActor)

and then the request is handled.

The ReportDedicatedActor waits for input from two different actors. When a message from the first actor arrives (it doesn’t matter if it’s the ResponseCalculateData or the ResponseDrawPlot), it will use the become pattern to wait for a message from the second, and once both messages arrive, it will wait for the GenerateReport message using become once again. Because the actor was created to serve just one request, it never sends the report to the wrong sender or misses any message thanks to the become pattern.

So what are the advantages of this pattern?

1. We can now handle the type issue by adding a default case to every behavior.
2. No timeouts on any Future hiding bugs.
3. The number of reports the system can handle is as permitted by our resources. Once the ReportGeneratorActor creates the ReportDedicatedActor, it is free to receive other GenerateReport messages instead of handling one report at a time.
4. This pattern actually creates a scale up/down mechanism inside the machine. If the system gets a lot of report requests, a proportional number of actors are spawned. If there aren’t many requests, the number of ReportDedicatedActor actors decreases and that in turn frees up system resources.

This pattern solved our issue simply and with very little changes to code. I can recommend you use it, but there are a couple of things you should keep in mind:

1. Because the system frequently creates and destroys actors, it gets pretty hard to debug and understand errors. You can solve this easily by employing some good logging.
2. Testing is tricky too: we can’t mock the ReportDedicatedActor because it’s created within the tested code. To get around this, inject some trait that extends ((ActorContext,Props)=> ActorRef) into the ReportGeneratorActor, and use that instance to generate the ReportDedicatedActor actors. Then, you can mock this in unit tests. This should give you full coverage for this component.

Creating actors at runtime is cheap, and although it has some disadvantages, it’s a powerful tool.