Message Queue 101: Your Ultimate Guide to Understand Message Queue
Introduction
Message Queue is a metaphysical conveyor belt of ethereal dispatches, where intangible parcels of digital essence…
Wait, we’re always excited to learn new concepts, but the complex, bookish definition always ruins our interest. Let’s ditch complex bookish knowledge of Message Queue. We’ll try to understand Message Queue with simple and easy-to-understand real-life examples, and then we’ll connect the dots with the Message Queue.
Imagination is more important than knowledge. Knowledge is limited. Imagination encircles the world. — Albert Einstein.
We’ll imagine more to encapsulate the knowledge of anything we learn new. This blog post will be your only need to understand the theory of Message Queue. Take some popcorn, and with due respect to Albert Einstein, Lego.
Message Queue in Real Life
Let’s begin with a story. Assume you have gone to a supermarket. You have bought some products and come to the payment desk. It has a long line, and while waiting in the line, you have decided to eat something at the fast food shop in front of the payment desk. You ordered some food, and the waiter asked you to wait for the food. Suddenly, you remembered you had to call customer support of an online shop. Your call to customer support is in waiting, and in the meantime, your number on the payment desk has been reached, and you have paid the payment. After some time, the waiter has served the food you ordered, and while eating, the call to customer support is connected, and you start talking to the agent.
Later, you measured that all three tasks took a total of 30 minutes to complete. Remember this story as we’ll recall this story to understand the use cases of message queue.
Now, let’s connect the dots with Message Queue.
In the story, there are three distinguishable sub-stories.
In the first sub-story, after buying the products, you (a producer) stand at the end of a line (a queue) to pay (a message) the amount of the products. The cashier (a consumer) is taking money one by one from the line (consuming/processing), and eventually, you get your turn to pay.
This “line” in the computer system is known as a queue where an element is inserted at the end of it and receives the element from the front of it resulting in the first element pushed into the queue, which will be the first one to be taken out. This behavior is called FIFO (first-in, first-out).
In the second sub-story, you (the producer) have placed an order for some food, and the waiter (a channel) takes the order (a message) to the kitchen (the queue). The chef (a worker/consumer) prepares the food (processes the message) based on the FIFO principle. Once the food is ready, the waiter delivers the food (processed message) to you.
In the third sub-story, you (the producer) phone-called (a channel) customer service of an online shop, the call is in waiting (a queue), while the agent (a consumer) is talking (serving) to someone else, you are waiting (in a queue) until all the customers (messages) called the support before you finish talking to the agent.
What are the common patterns you have seen in these three stories?
There’s always someone who is triggering an event. The one who creates an event is a producer. Here, you are acting as a producer as you have triggered some tasks by standing at the end of a line, placing a food order, and calling customer support.
The tasks you have produced have a chain of actions — by producing (triggering), you are passing a message (data) through a medium (queue) to the other end of the medium — someone somewhere is taking your tasks (consumers. cashier is processing payment, chef is cooking, and agent is resolving issues with an online order) and acting on it to bring a result.
We always use Message Queue in everyday life to make our lives easier.
How does it make our life easier, did you ask? Let’s assume the cashier tells you no one can stand in line because he is taking payment from another person, or the waiter tells you the fast food shop cannot take any more orders because the chef is currently cooking food for someone else, or the phone call to the online shop continuously drops because the agent is busy talking with someone else. What Message Queue does is that it acts like a buffer or a big basket. If A is busy doing a task x, and B comes to A with task y, A will place B’s task y in a basket allowing A to continue what he was doing, and B knows that the task will eventually be processed. A can process the task y at its pace and convenience. That basket changed the whole scenario of processing tasks. It is known as asynchronous task processing. More on this later.
I hope you have connected the dots between the Message Queue with real-life incidents. Now, it’s time to learn the technical aspects of a Message Queue.
Architecture and Components of Message Queue
The message queue is rooted in queuing theory, a mathematical discipline that analyzes waiting lines and resource allocation. — Wikipedia.
A Message Queue is an architectural pattern or a way of structuring communication commonly between distributed applications. Hence, a Message Queue acts like a bridge between two distributed applications to exchange data between them. The application that initiates data exchange is called a producer, and the application or set of applications that respond to the act of the producer and process the data are called consumers. Let us explore the important terminologies of Message Queue.
To simplify the understanding the concepts, I will use an analogy of a bank. Let’s assume your father has sent you to a bank with a cheque to withdraw money.
Message Broker
A message broker is a software component that sits between a producer and a consumer. It acts as the intermediary entity (more like a manager) between them and manages the routing, queuing, and delivery of messages received from the producer and to the consumer according to a set of predefined rules.
- You are going to a bank to withdraw some money. The bank is the message broker as it is the central entity that will facilitate financial transactions with a set of predefined rules.
Producer
An entity (application/applications or service/services) that is responsible for creating messages for the consumer to process further.
- Your father has sent you to a bank with a cheque. Here, your father is a producer of a task.
Channel
A communication medium to interact with the broker by the producer. This is the only communication entity that a producer can talk to the broker.
- Your father has sent you to the bank. You are in the bank. Your father could influence the banking facility he wants through you. You are the channel or medium of communication for your father to interact with the bank (Message Queue).
Consumer
An entity that is responsible for consuming the messages from the queue and processing them for a result.
- The banker who is seated at the withdrawal desk is the consumer of the withdrawal cheque (a message).
Queue
A queue is a buffer (storage/basket/data structure) that inserts messages from the end of it and empties messages from the front side of it (first-in, first-out). The messages produced by the producer are temporary (or persisted, depending on use cases) until the consumer is ready to consume (process) the message.
- As you enter the bank, you see there are many long lines — some are for deposits, some are for withdrawals, some are for money drafting, and even some are for disputes regarding financial transactions. The lines are divided based on the nature of the task — these lines are queues. In a message queuing system, many queues can be created based on the type of message (data) that is stored in it.
Exchange
An exchange is an entity that is responsible for routing the messages to different queues based on predefined rules. Any message is received by the exchange first, and the exchange determines to which queue the message should be deposited.
- As you enter the bank, you see a bank representative standing at the door. He asks you what is the purpose of your visiting the bank, and as you tell him you have come to withdraw some money, the bank representative tells you the line (a queue) to stand. Here, the bank representative acts as a routing exchange and decides which customer (a message) stand (is inserted) in which queue based on his predefined knowledge.
Message
The message is the data or signal that has been passed or created by the producer and sent over to the consumer for further processing.
- You have been given a cheque by your father to withdraw money from the bank. Here, the cheque is the message, which will be processed by the consumer (the banker) for a result (money).
Binding Key
Binding keys are the association between exchange and queue. A binding key helps an exchange to establish a relationship with the queue to uniquely distinguish it while delivering messages.
- As you have entered the bank, and once the bank representative has heard you have come to “withdraw” money, he directs you to the correct line/queue to stand.
- How did the bank representative know which queue to stand?
- He has to know it somehow as he is acting as an exchange of routing of customer’s tasks (messages). Here, the bank manager (it’s the programmer) has instructed him to route any customer with withdrawal requests (a key) to a specific line, deposit customers to another specific line, and so on. This is the binding that lets the exchange know if “withdraw” is the keyword, sends the customer to the ABC line, and so on. In practical life, the binding is done only once, as it is funny that the bank manager has to say ten times a day to the bank representative to forward deposit customers to a particular line! (but we do this each time in respect of tech, it’s considered good practice. Funny right!?)
In short, the binding key is the instruction on how an exchange should identify a queue.
Routing Key
Routing keys are the values used by exchanges to determine the queue to which the incoming messages should be delivered.
- Back to our example. As you enter the bank, you inform the bank representative that you want to withdraw, and by this information, the bank representative identifies you as a withdrawal customer and directs you to the withdrawal line/queue. Another customer comes, and he says he wants to report a dispute, upon hearing the keyword, the bank representative routes him to the correct queue.
- Although it is not required for the bank manager to tell the bank representative each time the binding key, all the incoming customers with a task (messages) should have (but not mandatory in some cases, discussed later) a routing key to correctly end up in a queue.
In reality, both the keys are the same. The differences between the routing key and the binding key are that the binding key lets the exchange know which queue is responsible for what kind of task and the routing key lets the exchange know to which queue the incoming message should be inserted based on the nature of the task.
Message Attributes
Metadata associated with each message such as timestamps, some ID, TTL (time to live), and message headers. Message Attributes provide additional information regarding the message.
- As you have come to the bank with a cheque (a message), the cheque has printed the bank account number, the account holder’s name, the bank name, the amount to withdraw, and the signature of the account holder. These are the metadata that provide additional information to correctly process the message by the consumer.
Acknowledgements (ACKs)
Mechanisms that is used to inform the producer that the message has been successfully delivered and the state of processing to the consumer by the message broker. Consumers send ACKs to the producer. The state of the consumption of the message could be successful or failure, and consumers inform this state of the task to the producer by ACKs.
- After the withdrawal is completed, the banker who has processed your message has handed over a piece of paper with the bank’s seal and stamp indicating that the withdrawal has been successful. Seeing this, your father could confirm the cheque has been accepted and the money has been withdrawn successfully.
Message Persistence
Option to store messages to disk to ensure messages are durable (not deleted) in case of system failures or restarts. These messages remain in the queue until they are explicitly consumed or expired.
- Assume in the bank, you have seen someone standing in the dispute queue for a dispute that a fraudster has fooled him and withdrawn money from his account. As this is a sensitive case, the banker has decided to keep the documents and evidence the customer brought to the bank in a secure place so that the bank can investigate and decide later.
RabbitMQ
RabbitMQ is a popular open-source message broker that sits between a producer and consumer for efficient message delivery. Some other message brokers are Redis, Apache Kafka, and Amazon SQS.
- Message Queue is the architectural pattern, while software like RabbitMQ, and Apache Kafka implement this architectural pattern.
Virtual Hosts
This is almost self-explanatory. Virtual hosts are isolated environments that allow the segregation of resources and configuration within RabbitMQ. Virtual hosts enable multi-tenancy and resource isolation for different applications or users sharing the same RabbitMQ instance.
- Assume the bank you have visited is on the second floor and the whole building has five floors. The building has branches of a few different banks. They all have their workplace and space to work without hampering the workflow of the other branches of different banks located on different floors but still grouped in the same building. The building is acting as RabbitMQ, and the floors are acting as isolated environments to provide separate spaces for different isolated environments to work independently.
AMQP Protocols
Protocols are sets of predefined rules and regulations on how a set of processes should be entertained.
- For example, think of a post office as a message queue. Messages are like letters. AMQP protocol would be the standardized format for writing addresses, stamps, and the whole lifespan of the letter from the post office to the receiver. The post office (Message Queue) uses this set of procedures (AMQP Protocol) to ensure efficient letter (a message) delivery. More on the protocols and technical exploration in my next post, for sure!
Types of Message Queue
The Message Queue comes with different varieties of protocols and algorithms to ensure message delivery, persistence, security, scalability, and reliability. However, there are mainly two types of Message Queues — Point-to-Point (P2P) and Publish/Subscribe (pub/sub).
Point-to-Point (Queue)
The Point-to-Point communication pattern (also known as the queue messaging pattern) ensures that a single message has a single consumer. The message is consumed by only one consumer (a receiver). This pattern is highly efficient and faster in message processing.
More than one consumer may be attached to a single queue, but it is guaranteed that once a consumer has started processing a message, no other consumer can process the same message.
This pattern ensures that a single message (product payment, food order, or call to customer support) has only one consumer (cashier, chef, or agent).
Publish/Subscribe (Topic)
The Publish/Subscribe communication pattern (also known as the topic or pub/sub messaging pattern) ensures that a single message can be consumed by multiple consumers. The message is broadcast to multiple consumers (receivers) that are subscribed to that topic. This pattern is highly scalable and reliable in message delivery.
This pattern ensures that a single message (advertisement notification or social media post) can be consumed by multiple consumers (subscribers).
If you remember while describing routing kye I pointed out that most of the messages should have a routing key, but in the case of pub/sub model, there is no need for a routing key. Once the publisher (a producer) publishes (produces) a topic (a message), all subscribers (consumers) subscribed (a kind of binding you can assume) to the topic get the message.
Use Cases and Application
Message Queue and Asynchronous Processing
The first popular choice for losing customers is slow response! The synchronous processing of requests in traditional web servers creates a huge overhead for the application if the request is involved in a long-running process. Let us take the below example to understand it better.
Assume a financial platform allows emailing transaction reports in PDF format. The possible workflow would be like below:
1. User selects start date, and end date and clicks submit.
2. An HTTP request has been sent to the server.
3. The server validates the form as per business logic.
4. The server starts generating the PDF.
5. The server prepares for email send, for which it might use 3rd party services like SendGrid or Twilio.
6. The 3rd party service sends the email.
7. The server receives confirmation from the 3rd party service.
8. The server responds with a 200, email has been sent response.
As the entire workflow is synchronous, the whole task is tightly coupled.
What do you think if the server does not hear back from the 3rd party email service, would it be able to send the user the 200, email has been sent response? In this case, the server might send 500, Internal Server Error as the server does not know what exactly went wrong. The whole workflow is dependent on each other.
Did you notice there are two noticeable downsides to this design?
The first one is the PDF generation. The server is handling HTTP requests from the users, and it is also responsible for generating a PDF, interacting with a 3rd party service to make sure the PDF is emailed to the user. The server is acting like a Jack Of All Trades! Would the server be able to handle if many users are sending concurrent requests to generate their PDFs?
The second downside is the server has to wait for the response from the email service. The server has nothing to do with sending the email to the user, still, it is waiting. Instead of waiting for a response from the 3rd party service, as the server has no responsibility for sending email, it could directly send a response to the user with 200, email has been sent. It would have saved a lot of time and the server could handle more concurrent requests to serve other users.
Let us solve this problem.
Here, you will use a Message Queue. You will divide the responsibility of the task between services. Once the server reaches point number 4, it will act as a producer and put the PDF generation and email sending task into a queue. The consumer (another remote service), which is consuming messages from the queue, will handle the 4, 5, and 6 number tasks.
The backend server will not wait for any response back from the consumer, so it will ignore the 7th point. Hence, the server will directly jump to the 8th point and send the user the 200, email has been sent response. The entire workflow is much faster, and the backend server can now handle many more concurrent PDF generation requests.
Here the backend server is acting as Master Of One. The heavy and long-running tasks have been assigned to some other application, and they will generate the PDF and send the email. The backend server and the consumer are not tightly coupled. The backend server can handle as many requests and keep pushing the time-bound task to the queue and return an instant response to the user. It is asynchronous processing.
Let us take another example. You began the understanding of Message Queue with a story. The story is an example of asynchronous processing. If you were doing it synchronously, then you would have done all the tasks one by one, not juxtaposing all three tasks. See the below images to understand it better.
Message Queue and Microservices
In simple terms, a microservices architecture is a combination of small sub-applications that are self-reliant on their own and do not directly depend on other external processes. You can assume all the services in microservices are Master Of One. The goal of microservices is availability and mitigate a single point of failure to name a few. The services need to communicate within them, and an easy approach to communicate between services via API calls. Let’s see what happens when you communicate within the services via API calls.
Tightly Coupling
Imagine there are two services, service A and Service B. If service A communicates via an API call with service B, it creates a dependency for service A on service B. How? Let’s see:
1. If service B fails, service A will also be unable to fulfill its request as it hasn’t heard back from service B.
2. If service A receives so many concurrent requests, service B will also be hampered as service A will keep forwarding the requests to service B for some response.
Loosely Coupling
If you introduce a Message Queue here, you will be able to mitigate this problem. In this example, you are using a message queue. Service A pushes the task to the queue and immediately returns a response to the user. On the other hand, service B, being a consumer, consumes the messages at its own pace. Any message remains in the queue until it is consumed by the consumer or expires if TTL (time to live) has been set. Hence, the consumer, service B could process the requests as per its pace and processing power.
Hence the solution is loosely coupled as per the following:
1. Service A keeps pushing the tasks to the queue does not wait for any response from the consumer and responds to the user immediately.
2. If service A experiences unexpected surge in traffic, it would not hamper service B, as service B is not directly connected with service A.
In this case, service A will keep on pushing the requests to the queue, and service B will keep on processing the request without knowing there is any surge in traffic in service A.
This creates a loosely coupled, asynchronous processing using a Message Queue increasing the efficiency of the overall platform. You will encounter this concept in many places like task distribution, event-driven architecture, microservices communication, job queue, workflow orchestration, and cross-application integration. All of them are more or less related to the same concept. They all focus on loosely coupled architecture, and if you understand loosely coupled, you almost know all of these terminologies!
Conclusion
In today’s fast-paced tech world, we should keep learning. However, the learning experience is not always easy and straightforward. When I tried to grasp the concepts of message queuing, I struggled with the terminologies. Hence, I decided to write an in-depth blog describing the responsibilities of all the terminologies in an easy-to-understand manner so that anyone attempting to comprehend it for the first time can do so in their initial reading.
As it was a purely theoretical one, you will explore the technical implementation of a message queue with microservices architecture in my next blog. But before that, you should have a basic understanding of the Message Queue, and my job is done if this blog has provided even a tiny portion of that understanding.
Oh, did I mention that I write in-depth blogs on microservices, Django, Docker, System Design, and many more? You can find more of my similar writing here on Hashnode. Here’s my LinkedIn and X if you’re, in case, interested to say hi.
Where to Go Next
In the meantime, you can read more on RabbitMQ documentation (remember, RabbitMQ is an open-source software that implements Message Queue) here.
Here are two interesting post read if you want to explore more —
Why do we need a Message Queue? from ByteByteGo.
Beginners Guide to Message Queue by Hevo.
As this was a purely introductory blog on Message Queues, you can try to implement it in your favorite language. In my next blog, we will together implement a microservices architecture and incorporate Message Queue in Python.
Until then, keep queuing, and happy messaging! 😊
