Consistent Hash Exchange
The Consistent Hash Exchange routes messages using the publisher's routing key and a hashing process. This ensures that messages related to the same identifier are always sent to the same consumer, keeping a predictable order and maintaining causal relationships.
Manually distributing messages can be complex, as publishers often lack visibility into the number of queues and their bindings. Messages are typically distributed sequentially, but to prevent long queues, data is often spread across multiple queues or handled by several consumers. However, this distribution can lead to the loss of order.
In contrast, messages sent to a consistent hash exchange are distributed to bound queues based on the hash computed from the routing key or header value. This method ensures that messages sharing the same computed hash—such as those linked to a specific booking ID or client ID—are directed to the same queue, thereby maintaining causal order for that ID as long as the bindings remain unchanged.
The routing key value determines how messages are distributed across bound queues based on their computed hash. Queues can be dynamically added or removed at runtime, allowing flexible scaling while maintaining message order for specific keys.
- Declare queues. In this case, two queues (
q1andq2) are declared with durability. - Define the exchange. Declare a consistent hash exchange (
ce), which will handle the routing of messages based on the hash of the routing key. -
Bind each queue to the exchange using a routing key. The routing key on the binding must be set as a numeric value, which determines how many times the queue is added to the hash ring. This acts like a weight, impacting the likelihood of a queue receiving messages. A queue with a higher binding key value is present more frequently in the hash ring and is therefore more likely to be selected.
Here,
q2is added to the hash ring twice as many times asq1, meaning it has a higher chance of receiving messages—but the actual distribution depends on the hash function applied to the published routing keys. -
Publishing messages: When a message is published, its routing key is hashed, and the exchange selects a queue based on its position in the hash ring.
Since
user123always hashes to the same value, this message will be routed to the same queue, ensuring that messages for a given user remain ordered. If more queues are added dynamically, the distribution of messages may shift, but messages for the same routing key will continue to be routed consistently.
Example: Set up a consistent hash exchange in LavinMQ to route messages based on the routing key’s hash.
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='q1', durable=True)
channel.queue_declare(queue='q2', durable=True)
channel.exchange_declare(exchange="ce", exchange_type="x-consistent-hash", durable=True)
channel.queue_bind(exchange="ce", queue='q1', routing_key="1")
channel.queue_bind(exchange="ce", queue='q2', routing_key="2")
channel.basic_publish(
exchange="ce",
routing_key="user123", # Routing key for hashing
body="Message for user123",
)
connection.close()
Hashing algorithms
LavinMQ provides two consistent hash algorithms: Ring and Jump. Both ensure that messages with the same routing key are consistently routed to the same queue, and that adding or removing queues causes minimal redistribution of keys.
Ring (hash ring)
The ring algorithm uses a hash ring with virtual nodes. Each bound queue is placed on the ring multiple times (determined by its weight), and messages are routed to the nearest queue on the ring by hash value.
This is the default algorithm.
Jump (jump consistent hash)
The jump algorithm is based on the paper “A Fast, Minimal Memory, Consistent Hash Algorithm” by John Lamping and Eric Veach. It maps keys to buckets in O(ln n) time with no memory overhead for storing a ring.
Jump consistent hash provides perfectly even distribution across buckets and minimal disruption when adding or removing a bucket (only 1/n keys move when the nth bucket is added).
Selecting an algorithm
The algorithm can be selected per exchange using the x-algorithm argument when declaring the exchange:
channel.exchange_declare(
exchange="ce",
exchange_type="x-consistent-hash",
durable=True,
arguments={"x-algorithm": "jump"}
)
Allowed values are ring and jump.
The server-wide default algorithm can be configured in lavinmq.ini:
[main]
default_consistent_hash_algorithm = ring
If no x-algorithm argument is provided when declaring the exchange, the server default is used.
Ready to take the next steps?
Managed LavinMQ instance via CloudAMQP
LavinMQ has been built with performance and ease of use in mind - we've benchmarked a throughput of about 1,000,000 messages/sec . You can try LavinMQ without any installation hassle by creating a free instance on CloudAMQP. Signing up is a breeze.
Get started with CloudAMQP ->
Help and feedback
We welcome your feedback and are eager to address any questions you may have about this piece or using LavinMQ. Join our Slack channel to connect with us directly. You can also find LavinMQ on GitHub.