Event Forwarding Pattern

In an event driven system there are different points at which message loss could happen resulting in data loss. The Event Forwarding pattern is one of the solutions to avoid this.

Like every other architectural pattern this has its own pros and cons.


While this pattern ensures data integrity in all scenarios, the overall performance and throughput of the system decreases by a third, according to some estimates.


Contents:


1. Challenge 

2. Concept

3. Points of Failure

4. Code Implementation

 
Let's understand the problem and proposed solution.


Challenge:


Make sure that no messages are lost when passing data from one system to another.



Pattern used: Event Forwarding Pattern



Concept :


For illustrating the problem and pattern lets consider two Services A and B, along with a event broker and database.

Service A publishes events into Event Broker. Service B subscribes to the said events, processes them and then stores the result in a database. 

Before applying any pattern visualise the usual flow.



Illustration : A sample flow of a simple distributed system


In the above example there are 3 places where events could be lost. Let's examine each of these one by one and solution for such scenarios.


Possible Point of Failure


Scenario 1:


Service A has published but message did not reach Event Broker, or broker did receive the message but was not able to persist it. Service A has no way of knowing if something went wrong.



Illustration : Scenario 1 failure




Solution :

Synchronous send to broker : Service A sends the event to broker and waits for an acknowledgement that the event has been received and persisted by the broker.

Incase the broker goes down, and ack is not received then there is a retry mechanism at Service A to resend the same message.


Illustration : Scenario 1 Solution


Scenario 2:


Service A has published the message successfully to broker, Service B has subscribed and read the message for processing.

But while processing something goes wrong and the service goes down. Now as far as the broker is concerned, the message is read successfully so it will mark it as read. But in reality the message is not processed successfully and there is no way to re process the message.


Illustration : Scenario 2 failure

Solution:

Client Acknowledge mode : Broker marks the message with a client id of service currently processing. 

Incase of crash, broker erases the  client id for some other service instance to pick up.


Illustration : Scenario 2 solution

Scenario 3:

Service B has sent the data for commit to DB but commit fails. As far as the broker is concerned the message is processed but in reality the message is not committed.

Service B cannot pick the message for reprocessing.


Illustration : Scenario 3 failure

Solution:


Last Participant Support : Post commit by service B on database, the next line of code sends ack to broker stating message is persisted.

the broker waits till it gets an ack that the message has been persisted before discarding the message.


Illustration : Scenario 3 solution


The above 3 problems and solutions cater to most of the scenarios where messages can be lost.


It is not mandatory to implement all the 3 stages, but decision needs to be taken on a case to case basis where some amount of data loss is acceptable while gaining on throughput of the system.


Code Implementation:


Traditionally if implemented using JMS ques or internal queues, all the above features need to be implemented individually.

Modern event brokers like kafka has all these features available out of the box and we just have to use it appropriately.

The sample implementation for above can be found here.

This implementation is kafka specific, with bonus implementation of dead letter queue too in the code


Comments

Post a Comment

Popular posts from this blog

AWS Lambda with Java : Starting the journey

Image
 The majority of courses out there cater to a teaching Lambda (or Serverless in general)  in combination with some scripting language like python or javascript based. Although this has its own advantages,  Java developers find little solace seeing their favourite language being left out. This series is an attempt to understand the AWS lambda model in combination with java and the amazon sdk for java. I personally enjoy the java lambda combination A love-hate relationship Before divulging into the details let us have a look at what are the advantages and disadvantages of using java for Lambda functions. After all we are going against opinion, so better be prepared with the facts. Fact 1: JVM based applications have higher start up times which has an impact on event based firing of lambda functions. This has a major impact on cold start time of lambda events. There are a couple of ways around it like provisioned concurrency with its own downsides with cost or reduci...
Read more

AWS Lambda, Kinesis and Java : The Producer
 (Part two of a three part blog.)

Image
This is the second post of the series in which we look at the different Kinesis Producers and their usage in a Java based SDK environment. For knowing details of Kinesis or the consumer you can visit below links A kinesis stream : Kinesis Fundamentals and the design we will be following in the post Kinesis Producer : creating a stream and publishing into the stream using both SDK API and KPL and also the kinesis agent. A kinesis consumer : A lambda function which runs whenever a message is published into the stream The code for this post can be found here . Kinesis Producer:  This is a spring boot application which is meant to run as an independent application. It could be deployed into the EC2 instance/ECS or run as a third party. For our purpose we are running this locally to publish messages into the producer. T here are 4 different mechanisms of producing messages. 2 using SDK APIs and another 2 using KPL libraries.All 4 are covered here.Let us dive into the code...
Read more

AWS Lambda, Kinesis and Java : Streams (Part one of a three part blog.)

Image
  Previous post deals with AWS lambda and java. Before we go to the hands on of lambda part I would like to introduce Kinesis. The reason being Lambda is almost always used in conjunction with another services. And the other service in our case will be Kinesis. Because Kinesis is a bigger topic we will be covering this in in 3 parts. Links to subsequent blogs   A kinesis stream : Kinesis Fundamentals and the design we will be following in the post Kinesis Producer : creating a stream and publishing into the stream using both SDK API and KPL and also the kinesis agent. A kinesis consumer : A lambda function which runs whenever a message is published into the stream Kinesis streams:  A managed message streaming service provided by AWS, comparable to Apache Kafka with subtle differences. It comes in variants of provisioned and on-demend variants. Provisioned service is selected when we know and we can control the amount of data coming in per second while on-demand is ...
Read more