Message Queue Driven Architecture in MuleSoft using Amazon SQS.

Whenever we are building complex systems using microservice architecture, we often encounter need of some asynchronous mechanism for communication between microservices(Mule API’s).

In an asynchronous system, events are produced and consumed independently, which means that components do not need to wait for each other to complete their tasks.

Message Queue Driven Model

• In a message-queue model, the publisher pushes messages to a queue where 1 or more subscriber can listen to a particular queue.
• Generally each message is processed by only one consumer which then deletes it from the queue.
• Due to this behavior, it is also sometimes called as Pull Model.
• FIFO queues in Amazon SQS is one of the ways to implement this model.

Event Driven Architecture using Pub/Sub Model

• Event-Driven Architecture (EDA) is a software design pattern or architectural style where the flow of the system’s operation is driven by events or notifications.
• Event-Driven Architecture (EDA) is generally implemented using the Publish-Subscribe (Pub/Sub) messaging pattern as a foundational communication mechanism.
• We generally deal with event emitters (or agents), event consumers (or sinks), and event channels.
• SQS and SNS can work together to form a powerful combination for EDA. We can use SNS to publish events to SNS topics, and use SQS to subscribe to those SNS topics and receive messages in queues which are then published in queues and received asynchronously by consumers using Pub/Sub Model
• A publisher publishes a message to a topic and all the subscribers of that topic receives at least 1 copy of that message.

In the current scope of our blog, we will be dealing with implementing Message Queue Driven Architecture using Amazon SQS in Mulesoft.

I will be assuming that you already have your AWS Account and MuleSoft setup.

Step 1 : Open Amazon SQS page from AWS console and click on Create Queue.

Step 2 : Select FIFO Queue as the type and name your queue as any_name.fifo.

We are using FIFO queue here not only because the order of messages is maintained but because the messages are deleted automatically after successful consumption by a single consumer, ensuring exactly-once processing.

Step 3 : Set the queue configuration as shown in the image.

Step 4 : Amazon SQS provides in-transit encryption by default. To add at-rest encryption to your queue, enable server-side encryption.

Step 5: Open AWS IAM in a new tab and create a user if not exists.

Step 6: Click on the user and copy the ARN to your notepad.

Step 7: Generate Access Key and Secret Access Key for your user and store it securely.

Step 8: Switch back to the create queue tab and select Advanced in Access Policy

Step 9: Embed the following statement in the statement array carefully to prevent connection errors from Mulesoft.

Step 10: Set the queue not to be used as a Dead Letter Queue and click on Create Queue Button.

Dead Letter Queue is a specialized queue used in messaging systems to manage messages that cannot be successfully delivered or processed

Step 11: Open Anypoint Studio and create a demo project.

Step 12: Add Amazon SQS Connector as a dependency from Exchange.

Step 13: Create a global.xml file and create a configuration for SQS.

You need to add your copied ARN to Test Queue ARN to test your connection with AWS SQS.

It is advisable to externalise these credentials in resources folder.

Step 14: Now we are ready to go with SQS and now its time to create a simple POC to demonstrate its usage.

Here we have scheduled a CRON job which will run after every 1 minute to add data into the queue but we need to make sure that the data should be sent in java format only.

We have added a receive message connector which will listen for the messages and log them to the console to validate our POC.

Kudos you have successfully built a POC with Mulesoft and Amazon SQS.

Stay tuned for more such exciting blogs!