Introduction
Configuration
Language Support
AMQP 0-9-1 Overview
More Exchange Types
More Consumer Features
Queue Deep-dive
Other Features
Reliable Message Delivery
High Availability and Backup
Monitoring
Management HTTP API
Tutorials
Networking
LavinMQ CLI
AMQP Queues
When discussing message queuing and message brokers, the focus is usually on the queue itself. The queue is the key to unbound yet connected services like a temporary storage area waiting for services to handle the messages. This guide provides an overview of queues in LavinMQ.
What is a Message Queue?
A message queue is a place where messages are stored until they are consumed by the consumer or removed from the queue in other ways.
What features do queues hold?
Message queues have properties that define how they behave. These properties are passed to the broker when the queue is declared. Queues can be marked as durable, exclusive, have auto-delete, and can be defined further with plugins and broker-specific properties. Using different properties provides different instructions on how messages should be handled within the broker depending on the use case.
How do messages enter the queue?
A message is sent to an exchange. The exchange routes these messages to queues, depending on associated bindings.
Setting up and sending messages thru queues
Before using a message queue, it must be named and declared. If a queue doesn’t already exist, declaring it will cause it to be created. A queue declared with no name is given a random name by most client libraries. After all, a name is needed for services connected to the queue in order to refer to it.
The sending service creates (declares) the message queue and the server confirms the declaration by responding “declare-ok”. The service then connects a consumer to the message queue. Messages can now be sent from one service (producer), via the queue, to another service (consumer). The consumer disconnects from a queue either explicitly or by closing the channel and/or connection.
Code example of how to declare a durable queue:
Queue Properties
A queue can be created with some given queue properties, like durable and exclusive.
Durable queues
A queue can be marked as durable, which specifies if the queue should survive an
LavinMQ restart. Setting a queue as durable only means that the queue
definition will survive a restart, not the messages in it. Create a durable
queue by specifying durable as True
during the creation of the queue.
Code example of how to declare a durable queue:
Durable queues and persistent messages are not one and the same. When publishing
messages, either persistent or transient delivery mode can be specified. The
delivery mode should be set to persistent
when publishing the message if it
should remain in the durable queue during server restart.
Exclusive queues
Setting the properties to exclusive
is useful when limiting a queue to only one
consumer. Exclusive queues can only be used by one connection, meaning that all
actions to and from that queue happen over, and follow the preset settings of
one specific connection. When the connection closes, the exclusive queue is
deleted.
Code example of how to declare an exclusive queue:
Auto-Delete property Queues
A queue can be declared with the auto-delete
property. This is useful when
queues should not stay open when the final connection to the queue closes.
Note: For the auto delete property to succeed, it requires at least one connected consumer, otherwise it will not be deleted.
Further reading: Using different methods for consumer and broker communication can affect the impact of the auto_delete property since the basic.get method gives direct access to a queue instead of setting up a consumer.
Passive Queues
The client can declare a queue as passive, just to check whether a queue exists
or not - without modifying the server state. If the queue already exists with
the same name, the server will reply with Declare-Ok
, if not, it will raise
an error.
Temporary Queues
A temporary queue is a queue that lives for a short period of time. A temporary queue is deleted when its last consumer is canceled or disconnected.
This queue type can be useful when you are creating a lot of queues for many different consumers. It ensures that queues are actually deleted once the consumer has finished consuming messages. On top of that, client connections can fail, potentially leaving unused resources (queues) behind.
A non-durable queue is not a temporary queue by default. The durability property is related to if the queue services a restart or not.
There are three ways to make queue deleted automatically:
1. Set a TTL
policy in the queue; e.g. a TTL policy of 28 days deletes queues
that haven’t been consumed for 28 days.
2. When its last consumer is canceled, the TCP connection with the server is
lost, or when the channel or connection is closed, an auto-delete queue will
be deleted.
3. An exclusive queue can only be used by its declaring connection. Exclusive
queues are deleted when their declaring connection is closed or gone.
Exclusive Queues
An exclusive queue can only be used by the same connection as the connection that declared the queue. This queue type is deleted when the declaring connection is closed or gone.
Example code in Ruby, bunny:
Queue Arguments (Optional Properties)
The specifications of the AMQP protocol (0.9.1) enables support for various features called arguments. Depending on which argument you implement, changes can be made to their settings after the queue declaration.
To set queue arguments, the use of policies is recommended. Policies make it possible to configure arguments for one or many queues at once, and the queues will all be updated when you’re updating the policy definition.
Read more about policies.
Maximum number of priorities
To apply message priority, the queue must be defined as a priority queue. Setting a queue’s priority range must be done when the queue is created. After declaring a queue, the number of priorities cannot be changed. Declared through the x-max-priority, a priority queue argument should, as with messages, be between 1 and 255.
Setting a maximum number of priorities is a property that cannot be changed after the queue declaration since it is not dynamically changeable. In other words, properties cannot be applied with policies and have to be enabled per-queue.
Read more about message priority.
Managing message and queue TTL
Time To Live (message-ttl) defines for how long a message exists before it’s considered to be dead. TTL can be configured for queues, groups of queues or messages. It it recommended to set message-ttl via policies.
LavinMQ sort messages on ttl, which may change the head of the queue, meaning that a high ttl message wont block a low ttl message.
Example of Queue TTL: If a queue TTL policy of 28 days is defined, queues that have not had messages consumed from them in the last 28 days will be deleted. Message TTL can also be specified during queue declaration or with a given argument/policy.
Example Message TTL: If a message TTL policy of 28 days is defined, messages have not been consumed for 28 days will be deleted.
Setting queue length limit
Another recommendation for applications that often get hit by spikes of messages, or where throughput is more important than anything else, is to set a max-length (x-max-length) on the queue. This keeps the queue short by discarding messages from the head of the queue so that it never becomes larger than the max-length setting.
Pausing a queue
Pausing a queue will pause all consumers (acting like there is no new messages in the queue). All clients will remain connected but they will not receive any messages. The “pause queue” option can be found in queue details in the LavinMQ management interface.
Message storage
LavinMQ adds messages to a specific queue by adding the segment number and segment position to each queue’s index. The message is stored in the message store, and a reference to the message is placed in the queue.
Read more about message storage.
Conclussions
Some guidelines to take into consideration when setting up and managing queues in LavinMQ:
-
Limit queue size with TTL or max-length, if possible. Applications that get hit by spikes of messages, and where throughput is a priority, should have the max-length set on the queue. This keeps the queue short by discarding messages from the head of the queues so that it’s never larger than the max-length setting.
-
Use multiple queues and consumers Achieve better throughput on a multi-core system with multiple queues and consumers. Optimal throughput is achieved when the number of queues matches the number of cores.
-
Persistent messages and durable queues for a message to survive a server restart To avoid message loss, make sure that the queue is declared as
durable
and the messages are sent with delivery modepersistent
(delivery_mode=2). -
Don’t set your own names on temporary queues Queue names are important when you want to share the queue between producers and consumers while naming temporary queues is less relevant. Instead, spend time naming temporary queues, let the server choose a random name, or modify the LavinMQ policies.
-
Auto-delete queues you are not using To avoid performance setbacks as a result of failing client connections that might leave unused resources (queues) behind, make sure to automatically delete unused queues. This can be accomplished by:
- Set a TTL policy in the queue; e.g. a TTL policy of 28 days deletes queues that haven’t been consumed from for 28 days.
- An auto-delete queue is deleted when its last consumer has canceled or when the channel/connection is closed (or when it has lost the TCP connection with the server).
- An exclusive queue can only be used by its declaring connection. Exclusive queues are deleted when their declaring connection is closed or gone.
Ready to take the next steps? Here are some things you should keep in mind:
Managed LavinMQ instance on CloudAMQP
LavinMQ has been built with performance and ease of use in mind - we've benchmarked a throughput of about 1,000,000 messages/sec. You can try LavinMQ without any installation hassle by creating a free instance on CloudAMQP. Signing up is a breeze.
Help and feedback
We welcome your feedback and are eager to address any questions you may have about this piece or using LavinMQ. Join our Slack channel to connect with us directly. You can also find LavinMQ on GitHub.