Introduction
Configuration
Language Support
AMQP 0-9-1 Overview
More Exchange Types
More Consumer Features
Queue Deep-dive
Other Features
Reliable Message Delivery
High Availability
Monitoring
Management HTTP API
Tutorials
Networking
LavinMQ CLI
Message Deduplication
What is message deduplication?
Message deduplication prevents duplicate messages from being enqueued, reducing unnecessary processing and storage.
When to enable message deduplication?
Enable deduplication when you need to prevent redundant processing, reduce storage usage, or ensure messages are delivered only once.
Message deduplication
LavinMQ supports message deduplication at both exchange and queue levels:
- Exchange-level deduplication: When enabled, duplicate messages are not forwarded to bound queues.
- Queue-level deduplication: Messages are routed as usual but are not stored in the queue if already seen. This ensures correct routing while avoiding increased “Unroutable message” statistics.
Using message deduplication
To enable deduplication, set the following arguments:
x-message-deduplication(bool) – Enables deduplication.x-cache-size(int) – Defines how many messages to store for deduplication checks. A larger cache uses more memory.
Optional settings:
x-cache-ttl(int) – Cache entry time-to-live in milliseconds (default: unlimited).x-deduplication-header(string) – The message header used for deduplication (default:x-deduplication-header).
The deduplication cache is stored in memory, meaning:
- A larger cache increases memory usage.
- The cache is lost on broker restarts or leadership transfers in a cluster.
This allows LavinMQ to filter out duplicate messages while maintaining proper routing behavior.
Ready to take the next steps? Here are some things you should keep in mind:
Managed LavinMQ instance on CloudAMQP
LavinMQ has been built with performance and ease of use in mind - we've benchmarked a throughput of about 1,000,000 messages/sec. You can try LavinMQ without any installation hassle by creating a free instance on CloudAMQP. Signing up is a breeze.
Help and feedback
We welcome your feedback and are eager to address any questions you may have about this piece or using LavinMQ. Join our Slack channel to connect with us directly. You can also find LavinMQ on GitHub.