Message routing

Previous Tutorial: Introduction

Note : While you may come across a few Python code snippets in this tutorial, its main focus will be on concepts discussed at a theoretical level, making it mostly independent of any specific programming language.

That being said, let’s kick things off here, by correcting a wrong impression that we’ve created in the previous tutorial.

Messages in LavinMQ, always go to an exchange first

In the previous tutorial, we declared queues and published messages directly to them - creating the impression that messages move from a producer and straight to a queue.

channel.queue_declare(queue="hello_world") # Declare a queue

channel.basic_publish(
    exchange='',
    routing_key=routing_key,
    body=body
)

In practice messages first go to an exchange, before they are then routed to the appropriate quque.

How is this possible in our case since we didnt declare exchanges in the previous tutorial? Well, in place of an exchange, we passed an empty string - see code snippet below:

channel.basic_publish(
    exchange='',
    routing_key=routing_key,
    body=body
)

The empty string passed in place of an exchange name, essentially, tells LavinMQ to use the default exchange. But to begin, what are exchanges?

understanding exchanges

When a producer sends a message, it specifies the exchange to which the message should be published and provides a routing key for the message.

In our case, we inadvertently specified the default exchange by passing an empty string in place of an exchange name.

Exchanges in LavinMQ act as intermediaries between producers and queues - the exchange receives messages from producers and decides what queue to forward the received messages to - the exchange has to make this decision for every message received. Exchanges are fundamentally message routing agents.

What queue an exchange decides to send messages to usually depends on the type of the exchange - different exchanges have different message routing strategies.

However, in general, all exchanges use a combination of queue bindings and and a message’s routing key to decide what queue to send the message to. Let’s explore the concepts of bindings and routing keys.

A quick note on bindings and routing keys

Here, let’s make sense of bindings and routing keys in LavinMQ.

Bindings

Bindings in LavinMQ are connections established between exchanges and queues. They define the relationship between the two, specifying which queues should receive messages from a particular exchange. In other words, bindings tell LavinMQ what queues are interested in the messages coming to a given exchange.

When you create a queue, you can bind it to one or more exchanges using a binding key. The binding key acts as a filter or criteria that determines whether a message should be routed to a particular queue or not.

Routing Keys

Routing keys are properties assigned to messages by producers. They are essential for routing messages from exchanges to queues. A routing key is a string value that carries information about the intended destination or purpose of the message.

At this point, we can re-define the steps taken to publish a message to a queue: - First, we need to create a connection to our LavinMQ instance from our Python code - Next, we create a channel within the connection we’ve created - Next we declare exchange(s) - Next, we declare queue(s) - Next, we bind queues to exchanges - Then publish messages, passing an exchange and a routing key, for appropriate routing.

A queue could be bound to an exchange in the manner shown in the code snippet below:

# Create an exchange called "test_exchange"
channel.exchange_declare("test_exchange", "direct")

# Create a queue called "test_queue"
channel.queue_declare("test_queue")

# Bind the queue to the exchange: queue_bind(name, exchange, binding_key)
channel.queue_bind("test_queue", "test_exchange", "test")

With a better understanding of these concepts, we can now picture the flow of messages from a producer and eventually to a consumer to be what’s shown in the image below.

LavinMQ messaging architecture

How exchanges use bindings and routing keys

When a message arrives at an exchange, the exchange would examine its routing key. It then compares the routing key with the binding keys of all queues bound to that exchange. The goal is to find a match between the routing key and the binding key.

Once a match is found between the routing key and the binding key, LavinMQ delivers the message to the associated queue. However, what is considered a match between a message’s routing key and a queue’s binding key entirely depends on the exchange type.

Thus let’s look at the different exchange types and how they match a routing key with a binding key - let’s call this matching mechanics the routing rules.

Types of exchanges

We will explore the four commonly used LavinMQ exchanges: fanout, direct, topic, and default exchanges - there are more.

Fanout Exchanges

Fanout exchanges are the simplest type, as they broadcast every incoming message to all bound queues. The routing key is ignored in this case, and the message is replicated to all queues that are bound to the exchange.

Let’s consider a use case.

To understand this, let’s consider an exampple.

Use case: A dumb notfication system

Let’s imagine we’re developing a Slack clone for a team that uses three channels: - HR - Marketing - And Support

The purpose of our application is to notify channel members about new activities happening within each channel.

In an ideal scenario, users would only receive notifications for activities in the channels they prioritize. However, our current notification system is not very smart, and users receive notifications for activities across all channels.

To improve this, we create a fanout exchange called slack_notifications . We also create three queues: hr, marketing and support, and bind them to the exchange.

Ideally, each queue should only receive and store data from its corresponding channel. For example, the “hr” queue should only receive and store data from the HR Slack channel.

However, because our notification system is not sophisticated, all messages published to the “slack_notifications” exchange are broadcasted to all the bound queues, regardless of their routing keys. In other words, there are no specific rules for routing messages to different queues based on the channel.

The image below provides a visual representation of how messages are distributed in fanout exchanges.

LavinMQ fanout exchange

Direct Exchanges

A direct exchange routes messages to queues based on a direct match between the routing key of the message and the binding key of the queue. Remember, the routing key is a property assigned to the message by the producer, and the binding key is defined when the queue is bound to an exchange. If the routing key and the binding key match exactly, the message is delivered to the corresponding queue.

To understand this, let’s consider an exampple.

Use case: An improved notfication system

Let’s make our Slack clone even better. We currently have three channels:

HR
Marketing
And Support

Now, we want to add a new feature that allows users to customize their notification settings and choose which channels they receive notifications from.

To implement this, we’ll create three queues named hr_queue, marketing_queue, and support_queue. We’ll also set up a direct exchange called slack_notifications . Each queue will be bound to the slack_notifications exchange using the corresponding binding key: hr, marketing, and support.

By switching from a fanout exchange to a direct exchange, notifications will be directed to the specific queues and subsequently to the consumers subscribed to each queue. This means marketing notifications will go to the marketing_queue, and the same applies to the other queues.

The image below provides a visual representation of the exchanges, queue declarations, and bindings in our notification system.

LavinMQ direct exchange

Giving the bindings shown in the image above, if the application publishes a message with the routing key hr to the slack_notifications exchange, the message will be routed to the hr_queue. Similarly, a message with the routing key marketing or support will be routed to the marketing_queue, or support_queue respectively.

Topic Exchanges

A topic exchange is more flexible in routing messages based on patterns in the routing key. The routing key is typically a string that consists of multiple words or segments, separated by dots.

Topic exchanges route messages using routing rules that are pretty similar to that of direct exchanges - there has to be an exact match between the routing key and the binding key.

However, topic excahnges differ from direct exchanges in one major way - queues could be bound to an exchange using binding keys that can contain wildcard characters.

The wildcard characters used in topic exchanges are: - Asterisk (“”): The asterisk matches exactly one word in the routing key. For example, there will be a match between the routing keys _countries.nigeria_, _countries.sweden_ and the binding key _countries._ - And hash (“#”): The hash can match zero or more words.For example, there will be a match between the routing keys countries.nigeria, countries.sweden and the binding key #

Use case: Extending the notfication system

Let’s extend our Slack clone use case for more granular and flexible notifications.

In the direct exchanges use-case we allowed users to customize their notifications settings to specify exactly what channels out of the three they’d like to receive notifications from.

Taking that a step further, here, we will also allow users to customize their notification settings to only receive notifications for very specific activities within the channel they’ve subscribed to. To keep things simple, we are only going to have two categories of activities: new_messages and new_member_joined.

Even though in reality we are likely to have a lot more scenarios, for brevity’s sake, here, let’s assume that we are going to have these categories of users: - Persons that would want to receive notifications for messages in the marketing channel - Persons interested in notifications for when new members join the support channel - Persons interested in notifications for new messages in all the channels - And the last set of persons that would want to receive notifications for everything.

To do that, under the hood, the application sets up four queues: marketing_messages, support_new_members, all_messages, and all_notifications.

marketing_messages: to only receives notification for new messages in the marketing channel
support_new_members: to only receive notfications for when new members join the support channel
all_messages: to receive notifications for new messages across all channels
all_notifications: to receive notifications for everything across all channels

Additionally, a topic exchange named slack_notifications is created. The four queues are then bound to the slack_notifications exchange using the respective binding keys: notifications.marketing.messages, notifications.support.new_member_joined, notifications.*.messages and notifications.#

The image below visaulaizes the exchanges and queue declarations as well as the bindings.

LavinMQ topic exchange

Giving the bindings shown in the image above, if the application publishes a message with the routing key notifications.marketing.messages to the slack_notifications exchange, the message will be routed to the marketing_messages, all_messages, and all_notifications queues. Similarly, a message with the routing key notifications.support.new_member_joined will be routed to the support_new_members, and the all_notifications queues.

Default exchange

The default exchange in LavinMQ is a predefined direct exchange that is commonly represented by an empty string - something that we’ve done in the hello world tutorial.

When utilizing the default exchange, your message is delivered to a queue that shares the same name as the routing key of the message. Each queue is automatically associated with the default exchange, where the routing key matches the queue name.

Here, we’ve explored the most commonly used exchanges in LavinMQ. To paint a better picture, we looked at a use-case - the Slack notification system. in the susbsequent tutorials we will focus on seeing how task queues could be implemented with default exchanges and subsequently, what our notification system will look like in code.

Note: Here, we merely covered the common exchanges in LavinMQ - there are other exchange types: - Dead Letter Exchange - Consistent Hash Exchange - Alternate Exchange - Header Exchange

Before we proceed to the next section, here’s a little something for you to do:

Learning Lab: A simple email system

Use case: You are building a simple email system using LavinMQ for message routing. The system consists of three components: a sender, a receiver, and a mail server. The sender sends email messages to the mail server, which then delivers the messages to the appropriate receiver. How would you design the message routing system using LavinMQ to ensure that each email reaches the correct recipient?
Challenge: Design the message routing system using LavinMQ’s exchange types, routing keys, and bindings to ensure that each email message is delivered to the correct receiver. Describe the exchange type you would use, how the receivers’ queues would be bound to the exchange, and how the routing keys would be assigned.
Note: - you don’t have to build this, you can just think abdout it.
Hints:
- Consider using a direct exchange to handle the routing of email messages.
- Each receiver can have its own queue bound to the direct exchange with a unique binding key.
- When publishing an email message, use the routing key that corresponds to the recipient’s binding key.

What’s next?

Next Tutorial: Task queue with default exchange
Previous Tutorial: Introduction

Introduction

Configuration

Language Support

AMQP 0-9-1 Overview

More Exchange Types

More Consumer Features

Queue Deep-dive

Other Features

Reliable Message Delivery

High Availability and Backup

Monitoring

Management HTTP API

Tutorials

Networking

LavinMQ CLI