AMQP Queues

When discussing message queuing and message brokers, the focus is usually on the queue itself. The queue is the key to unbound yet connected services like a temporary storage area waiting for services to handle the messages. This guide provides an overview of queues in LavinMQ.

What is a Message Queue?

A message queue is a place where messages are stored until they are consumed by the consumer or removed from the queue in other ways.

What features do queues hold?

Message queues have properties that define how they behave. These properties are passed to the broker when the queue is declared. Queues can be marked as durable, exclusive, have auto-delete, and can be defined further with plugins and broker-specific properties. Using different properties provides different instructions on how messages should be handled within the broker depending on the use case.

How do messages enter the queue?

A message is sent to an exchange. The exchange routes these messages to queues, depending on associated bindings.

Setting up and sending messages thru queues

Before using a message queue, it must be named and declared. If a queue doesn’t already exist, declaring it will cause it to be created. A queue declared with no name is given a random name by most client libraries. After all, a name is needed for services connected to the queue in order to refer to it.

The sending service creates (declares) the message queue and the server confirms the declaration by responding “declare-ok”. The service then connects a consumer to the message queue. Messages can now be sent from one service (producer), via the queue, to another service (consumer). The consumer disconnects from a queue either explicitly or by closing the channel and/or connection.

Code example of how to declare a durable queue:

channel.queue_declare(queue='test', durable=True)

Queue Properties

A queue can be created with some given queue properties, like durable and exclusive.

Durable queues

A queue can be marked as durable, which specifies if the queue should survive an LavinMQ restart. Setting a queue as durable only means that the queue definition will survive a restart, not the messages in it. Create a durable queue by specifying durable as True during the creation of the queue.

Code example of how to declare a durable queue:

channel.queue_declare(queue='test', durable=True)

Durable queues and persistent messages are not one and the same. When publishing messages, either persistent or transient delivery mode can be specified. The delivery mode should be set to persistent when publishing the message if it should remain in the durable queue during server restart.

Exclusive queues

Setting the properties to exclusive is useful when limiting a queue to only one consumer. Exclusive queues can only be used by one connection, meaning that all actions to and from that queue happen over, and follow the preset settings of one specific connection. When the connection closes, the exclusive queue is deleted.

Code example of how to declare an exclusive queue:

channel.queue_declare(queue='', exclusive=True)

Auto-Delete property Queues

A queue can be declared with the auto-delete property. This is useful when queues should not stay open when the final connection to the queue closes.

Note: For the auto delete property to succeed, it requires at least one connected consumer, otherwise it will not be deleted.

Further reading: Using different methods for consumer and broker communication can affect the impact of the auto_delete property since the basic.get method gives direct access to a queue instead of setting up a consumer.

Passive Queues

The client can declare a queue as passive, just to check whether a queue exists or not - without modifying the server state. If the queue already exists with the same name, the server will reply with Declare-Ok, if not, it will raise an error.

Temporary Queues

A temporary queue is a queue that lives for a short period of time. A temporary queue is deleted when its last consumer is canceled or disconnected.

This queue type can be useful when you are creating a lot of queues for many different consumers. It ensures that queues are actually deleted once the consumer has finished consuming messages. On top of that, client connections can fail, potentially leaving unused resources (queues) behind.

A non-durable queue is not a temporary queue by default. The durability property is related to if the queue services a restart or not.

There are three ways to make queue deleted automatically: 1. Set a TTL policy in the queue; e.g. a TTL policy of 28 days deletes queues that haven’t been consumed for 28 days. 2. When its last consumer is canceled, the TCP connection with the server is lost, or when the channel or connection is closed, an auto-delete queue will be deleted. 3. An exclusive queue can only be used by its declaring connection. Exclusive queues are deleted when their declaring connection is closed or gone.

Exclusive Queues

An exclusive queue can only be used by the same connection as the connection that declared the queue. This queue type is deleted when the declaring connection is closed or gone.

Example code in Ruby, bunny:

queue = channel.queue("", :exclusive => true)

Queue Arguments (Optional Properties)

The specifications of the AMQP protocol (0.9.1) enables support for various features called arguments. Depending on which argument you implement, changes can be made to their settings after the queue declaration.

To set queue arguments, the use of policies is recommended. Policies make it possible to configure arguments for one or many queues at once, and the queues will all be updated when you’re updating the policy definition.

Maximum number of priorities

To apply message priority, the queue must be defined as a priority queue. Setting a queue’s priority range must be done when the queue is created. After declaring a queue, the number of priorities cannot be changed. Declared through the x-max-priority, a priority queue argument should, as with messages, be between 1 and 255.

Setting a maximum number of priorities is a property that cannot be changed after the queue declaration since it is not dynamically changeable. In other words, properties cannot be applied with policies and have to be enabled per-queue.

Managing message and queue TTL

Time To Live (message-ttl) defines for how long a message exists before it’s considered to be dead. TTL can be configured for queues, groups of queues or messages. It it recommended to set message-ttl via policies.

LavinMQ sort messages on ttl, which may change the head of the queue, meaning that a high ttl message wont block a low ttl message.

Example of Queue TTL: If a queue TTL policy of 28 days is defined, queues that have not had messages consumed from them in the last 28 days will be deleted. Message TTL can also be specified during queue declaration or with a given argument/policy.

Example Message TTL: If a message TTL policy of 28 days is defined, messages have not been consumed for 28 days will be deleted.

Setting queue length limit

Another recommendation for applications that often get hit by spikes of messages, or where throughput is more important than anything else, is to set a max-length (x-max-length) on the queue. This keeps the queue short by discarding messages from the head of the queue so that it never becomes larger than the max-length setting.

Pausing a queue

Pausing a queue will pause all consumers (acting like there is no new messages in the queue). All clients will remain connected but they will not receive any messages. The “pause queue” option can be found in queue details in the LavinMQ management interface.

Message storage

LavinMQ adds messages to a specific queue by adding the segment number and segment position to each queue’s index. The message is stored in the message store, and a reference to the message is placed in the queue.

Conclussions

Some guidelines to take into consideration when setting up and managing queues in LavinMQ:

Limit queue size with TTL or max-length, if possible. Applications that get hit by spikes of messages, and where throughput is a priority, should have the max-length set on the queue. This keeps the queue short by discarding messages from the head of the queues so that it’s never larger than the max-length setting.
Use multiple queues and consumers Achieve better throughput on a multi-core system with multiple queues and consumers. Optimal throughput is achieved when the number of queues matches the number of cores.
Persistent messages and durable queues for a message to survive a server restart To avoid message loss, make sure that the queue is declared as durable and the messages are sent with delivery mode persistent (delivery_mode=2).
Don’t set your own names on temporary queues Queue names are important when you want to share the queue between producers and consumers while naming temporary queues is less relevant. Instead, spend time naming temporary queues, let the server choose a random name, or modify the LavinMQ policies.
Auto-delete queues you are not using To avoid performance setbacks as a result of failing client connections that might leave unused resources (queues) behind, make sure to automatically delete unused queues. This can be accomplished by:
- Set a TTL policy in the queue; e.g. a TTL policy of 28 days deletes queues that haven’t been consumed from for 28 days.
- An auto-delete queue is deleted when its last consumer has canceled or when the channel/connection is closed (or when it has lost the TCP connection with the server).
- An exclusive queue can only be used by its declaring connection. Exclusive queues are deleted when their declaring connection is closed or gone.

Introduction

Configuration

Language Support

AMQP 0-9-1 Overview

More Exchange Types

More Consumer Features

Queue Deep-dive

Other Features

Reliable Message Delivery

High Availability

Monitoring

Management HTTP API

Tutorials

Networking

LavinMQ CLI