Introduction
Configuration
Language Support
AMQP 0-9-1 Overview
More Exchange Types
More Consumer Features
Queue Deep-dive
Other Features
Reliable Message Delivery
High Availability
Monitoring
Management HTTP API
Tutorials
Networking
LavinMQ CLI
Shovel
Shovels are used to move messages from a source to a destination, typically from one server to another. It can be used to balance the load of a queue or when you need to take messages out of one LavinMQ server and insert them into another.
What is a shovel?
A shovel is used to move messages from one place to another, for example between servers. A shovel can be set up to move messages between:
- An exchange to another exchange
- An exchange to a queue
- A queue to an exchange
- A queue to another queue
How does a shovel work?
First, the shovel connects to the source and destination, so it must successfully authenticate and be authorized to be able to move messages. When a connection is established it will start to consume from the source and re-publish to the destination. In the event of a failure, the shovel will attempt to reconnect.
When can I use shovels?
- If you have a server with a very high load on queue A you can create a shovel and configure it to consume the messages from queue A and republish them to an exchange in another server to help with the load.
- When a new instance is created and you need to move all messages from a queue on an old instance to a queue on a new instance.
Configure a shovel
A shovel can be created from the Shovel section in the LavinMQ Management Interface. The image shows a simple example where all the messages that are published to an exchange on the source will be moved to the exchange on the destination.
- Virtual host: Any vhost chosen
- Name: The name of the shovel
- URI: connection string to the source and destination clusters.
The URI used to connect can be in any of the formats:
- amqp://user:password@server-name/my-vhost
- connect to server-name, with credentials and specified virtual host
- amqps://user:password@server-name?cacertfile=/path/to/cacert.pem&certfile=/path/to/cert.pem&keyfile=/path/to/key.pem&verify=verify_peer
- connect to server-name, with credentials and SSL
- amqps://server-name?cacertfile=/path/to/cacert.pem&certfile=/path/to/cert.pem&keyfile=/path/to/key.pem&verify=verify_peer&fail_if_no_peer_cert=true&auth_mechanism=external
- connect to server-name, with SSL and EXTERNAL authentication
- amqp://user:password@server-name/my-vhost
- Type: You can set both source and destination as either a queue or an exchange. If you choose “queue”, it will be declared beforehand; if you choose “exchange” it will not, but an appropriate binding and queue will be created when the source is an exchange.
- Endpoint: Exchange or queue to be shoveled from/to.
- Routing key: only applicable when the type is exchange.
- Prefetch: The maximum unacked message count that can be pending for a shovel at any given time. The default is 1000.
- Auto delete:
- Never: The shovel never deletes itself; it will persist until it is explicitly removed.
- Queue length: The shovel will check the length of the queue when it starts up. It will transfer that many messages and then delete itself.
- Reconnection delay: Time in seconds to wait after a shovel goes down before attempting a reconnection. Defaults to 1 if not set.
-
Ack mode:
- On-confirm: Messages are acknowledged at the source after they have been confirmed at the destination. Without losing messages, this handles errors experienced over the network and broker failures. It is the default and the slowest option.
- On-publish: Messages are acknowledged at the source after they have been published at the destination. Without losing messages, this handles errors experienced over the network and broker failures.
- No-ack: Message acknowledgements are not used. Messages may be lost in the event of a network failure, though this is the fastest option.
Read more
HTTP API documentation for shovels: HTTP API
Ready to take the next steps? Here are some things you should keep in mind:
Managed LavinMQ instance on CloudAMQP
LavinMQ has been built with performance and ease of use in mind - we've benchmarked a throughput of about 1,000,000 messages/sec. You can try LavinMQ without any installation hassle by creating a free instance on CloudAMQP. Signing up is a breeze.
Help and feedback
We welcome your feedback and are eager to address any questions you may have about this piece or using LavinMQ. Join our Slack channel to connect with us directly. You can also find LavinMQ on GitHub.