Asynchronous Communication


What Is The Business Need?

Business needs agility i.e. bring disruptive and creative ideas to life very quickly. Adding new features quickly to the existing set of features is possible if that can be done easily.

When any idea floats, it goes through multiple phases like brainstorming, prototyping, building hypothesis, feasibility study etc and then is picked for implementation. When it comes to implementation stage then where to do the changes in the system, which components are to be changed are the usual things to be addressed.

How Monolith Troubled Idea?

In the monolithic system, its easy as there is one box which has everything. Due to lack of modularity and unclear segregation of responsibility, there are multiple places where the changes need to be copied. Also there could be changes in different unrelated systems to make the change happen.

The same code is usually copy pasted without understanding complete context make the system fragile, brittle and hard to understand/debug. Without automation any feature implementation takes quite long to build, test, and release. Ideas are killed in implementation complexity or during the long wait due to long release delays.

How Does MicroServices Enable Business Agility?

Micro-services make this possible and easier with its two characteristics, independent and autonomous.

Since each service is cohesive and has a specific business function and role to play in an entire echo system, identifying the responsible component is easy.

Asynchronous communication makes the services independent and autonomous. If any service needs to be changed and deployed (released) independently, then it has to be autonomous, cohesive and independent. Lets looks at this in detail.

Autonomous means having the freedom to govern itself or control its own affairs. In the enterprise it means the function can change itself without impacting other business function and a service can change without other component getting impacted. Along with Asynchronous communication, it is essential that every service announces its function, responsibilities and the message structure (aka contract or request/response). This way the consumer know what messages to pass and expect. Since micro-services has distributed transaction it is far more important to have standardized contracts among all services since they consume and serve each other.

Defining standardized contracts make the services decoupled. Once a standardized contract is agreed then the respective teams are free to go ahead and implement the services with their appropriate choice. Teams have autonomy to design and choose technology. E.g. payment service team can build the service in Java or .NET and authentication service team may prefer Scala.

Cohesion is an important aspect. Cohesion is the action or fact of forming a united whole. You can imagine the water pouring out of a glass because it has high cohesion of droplets. Micro-service should serve a single business purpose and should have that purpose as its only focus. This is analogous to the Single Responsibility Principle (SRP) of SOLID design principle.

E.g. For a retail website, Login function is handled by Authentication service, Search products function is handled by Search or Discovery service, show details is handled by the specific product sub category, payment, send order email etc should be handled by respective services.

Since the overall transaction is distributed across different services there a better co-ordination and communication required amongst services.

The communication must be clear, complete and crisp. For this, the elements/attributes within the service contract should be explained with the purpose of attribute and how do they play a role in the function. E.g. user credentials in the authentication service makes sense but in search product or payment process those are irrelevant. This helps in having only needed elements in the service contract. Also helps consumer build clear understanding on the significance each element and contract overall. Service must be provided with all i.e. complete details to operate on and service maintain all required details with it for its function so it does not depend on other parties.

Communication Types

There are two major types of communication between the services.

Synchronous means existing or occurring at the same time. This means services must be talking to each other through out the transaction and waiting for each other during the process. E.g. a telephone conversation. In essence, the service is dependent on the calling service to process its own function. This leads to dependency. Examples of synchronous communication are service API calls are several e.g. RPC, an HTTP endpoint or a REST API.

REST API is a preferred mode of communication among all for synchronous communication.

Asynchronous means not existing or occurring at the same time. E.g. sending an email which is responded at a later point in time.

Working through asynchronous communication is a paradigm shift and needs different way to look at consumer experience and technology too.

In our comparison of SOA and micro-services, we briefly touch-based that SOA uses the (communication mechanism) ESB which holds the business logic and does , transformation, routing etc. However micro-services use the communication mechanism as dumb messenger, meaning having neither the knowledge about the message nor an ability to act or interpret the message.

Within the asynchronous mechanism micro-services operate on producer and consumer relationship. Every service has an act or a role to perform hence service works as actor. This actor registers itself for events. Once those events occur the actor is notified with a message. Actor then consumes the message and acts on that. This message has all needful details for the service/actor to complete its function. Once the service has completed the operation it generates the notification and sends on the channel. Respective services listening to the channel grab the message and chain of actions continues.

There are couple of things to be noticed.

Role Of Actor And Queue Framework

One we have referred service as an actor, this was for two reasons 1. in the entire orchestration of distributed transaction service is expected to perform a role 2. an actor framework can be leveraged for asynchronous communication mechanism and when we do that we would already know which all actors we would create. Will have an elaborated view on this in actor framework discussion.

Two, When the service is working on the message other services are not waiting for it. Whenever service completes the operation it posts message on a channel. This makes communication inverted as supposed to a synchronous manner. In asynchronous way, producer can work on the message that it has consumed, complete the function and post the message. Consumer once received the message can start working on it. Producers and consumers are working independently.

Third, this requires a channel which works as a (dumb) messenger, to which producer and consumer.

With this you have a systematic orchestration in place where.

You can imagine any automatic auto pilot manufacturing machine where each process does its own job and moves through the rolling belt to reach the next process.

All the processes are unaware of the other process’s job and context.

There are few scenarios where synchronous communication is needed e.g. database queries. Decisions of when to choose synchronous communication are crucial.

There are multiple frameworks available for producer-consumer pattern and many option available for implementing communication channel.

Let us explore distributed parallel processing frameworks. There are two key choices Akka and Storm.

Comparison FactorAkkaStorm
What is it?Akka is a toolkit for building highly concurrent, distributed, and resilient message-driven applications.Storm is a free and open source distributed real-time computation system.
Message PatternActor can send back a reply to the sender of the message that is being handled.Actor can send back a reply to the sender of the Storm has unidirectional message flow so it can not send message to caller again.
Message DeliveryAkka although does not have its own mechanism, it can use Kafka for the same.Storm does provide guaranteed message delivery.
Communication PatternCan be changed dynamicallyStorm works on a fixed and pre-decided pattern.
Cluster ManagementAkka can leverage Kafka for this purpose.Storm advances in this segment has abilities like managed fail over, load balancing etc.
Preferred Use CasesParallel, Concurrent and Distributed applicationsStreaming and Analytics

What to choose from depends on the use case, however speaking broadly, as Akka is generic it could be used to orchestrate and decouple business process and Storm is for batch and stream processing. Above illustration should help decide.

For communication channel we can use RabbitMQ, Kafka, MSMQ, AMPQ. Here is quick summary of the comparison between RabbitMQ and Kafka.

Comparison FactorRabbitMQKafka
What is it?RabbitMQ is an open source message broker software that originally implemented the AMQP and has since been extended with a plug-in architecture to support various (MQTT,STOMP) protocols.Kafka an open-source stream-processing software platform to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
ArchitectureGeneral purpose message broker with Exchanges and QueuesKafka leverages Zookeeper which co-ordinates between primary and replica brokers. Kafka is designed for high volume and scalable transactions.
Ease of use & installationFairly simple. No learning curve required.Needs Zookeeper as well and has learning curve.
DocumentationComprehensive and up to date documentation available.Kafka documentation is just adequate and is evolving.
Processing speed (Performance)20K Per Second100K Per Second
Supported languagesJava, C#, Erlang (Suports 15+ other client implementations like Go, Elixir, Python, Ruby)Java and supports less than 12 client implementations ( PHP, Python, Nodejs)
Message replay capabilityNoYes
Routing, ACL & SSL Support, Monitoring InterfaceYesNo

Both of the frameworks are very powerful and competitive. RabbitMQ is simple and ready to use, on the other side Kafka comes from the Apache family and has great performance. The illustration above and its mapping to your need should drive the choice.

Lets communicate asynchronously and live happily!