Panda Insights

  Previous post deals with AWS lambda and java. Before we go to the hands on of lambda part I would like to introduce Kinesis. The reason being Lambda is almost always used in conjunction with another services. And the other service in our case will be Kinesis. Because Kinesis is a bigger topic we will be covering this in in 3 parts. Links to subsequent blogs   A kinesis stream : Kinesis Fundamentals and the design we will be following in the post Kinesis Producer : creating a stream and publishing into the stream using both SDK API and KPL and also the kinesis agent. A kinesis consumer : A lambda function which runs whenever a message is published into the stream Kinesis streams:  A managed message streaming service provided by AWS, comparable to Apache Kafka with subtle differences. It comes in variants of provisioned and on-demend variants. Provisioned service is selected when we know and we can control the amount of data coming in per second while on-demand is ...

This is the second post of the series in which we look at the different Kinesis Producers and their usage in a Java based SDK environment. For knowing details of Kinesis or the consumer you can visit below links A kinesis stream : Kinesis Fundamentals and the design we will be following in the post Kinesis Producer : creating a stream and publishing into the stream using both SDK API and KPL and also the kinesis agent. A kinesis consumer : A lambda function which runs whenever a message is published into the stream The code for this post can be found here . Kinesis Producer:  This is a spring boot application which is meant to run as an independent application. It could be deployed into the EC2 instance/ECS or run as a third party. For our purpose we are running this locally to publish messages into the producer. T here are 4 different mechanisms of producing messages. 2 using SDK APIs and another 2 using KPL libraries.All 4 are covered here.Let us dive into the code...

 The majority of courses out there cater to a teaching Lambda (or Serverless in general)  in combination with some scripting language like python or javascript based. Although this has its own advantages,  Java developers find little solace seeing their favourite language being left out. This series is an attempt to understand the AWS lambda model in combination with java and the amazon sdk for java. I personally enjoy the java lambda combination A love-hate relationship Before divulging into the details let us have a look at what are the advantages and disadvantages of using java for Lambda functions. After all we are going against opinion, so better be prepared with the facts. Fact 1: JVM based applications have higher start up times which has an impact on event based firing of lambda functions. This has a major impact on cold start time of lambda events. There are a couple of ways around it like provisioned concurrency with its own downsides with cost or reduci...

There is an external configuration server which contains configuration details of services. Now whenever there is a change in any configuration the services need to be actively listening for configuration changes, This creates an always on connection consuming resources where not necessarily required. Or a broker is required to continuously publish the changes to different services making the overall infrastructure heavy. Problem : Notify services whenever configuration value changes Solution : Watch notification pattern. A broker less approach to publish changes across services Prelude to watch notification service : Before we divulge into the watch notification pattern lets have a look at attempts already done to solve the problem of notifying services Service Broadcast Pattern Actors:  1. A group of services 2. External config server 3. A broker Concept : All services are subscribed to a broker. The external configuration service publishes  a ch...

In an event driven system there are different points at which message loss could happen resulting in data loss. The Event Forwarding pattern is one of the solutions to avoid this. Like every other architectural pattern this has its own pros and cons. While this pattern ensures data integrity in all scenarios, the overall performance and throughput of the system decreases by a third, according to some estimates. Contents: 1. Challenge  2. Concept 3. Points of Failure 4. Code Implementation   Let's understand the problem and proposed solution. Challenge: Make sure that no messages are lost when passing data from one system to another. Pattern used :  Event Forwarding Pattern Concept : For illustrating the problem and pattern lets consider two Services A and B, along with a event broker and database. Service A publishes events into Event Broker. Service B subscribes to the said events, processes them and then stores the result in a database.  Before applying any pat...