LavinMQ streams: an interactive guide to offsets and retention

A queue forgets. A stream remembers. In this post, we’ll look at what happens after a message lands in the stream.

A standard queue deletes a message as soon as a consumer acknowledges it. A stream keeps it as long as you tell it to, turning the queue into a log that any number of consumers can read, reread, and replay.

In LavinMQ, a stream is a queue type, not an exchange type. You declare it like any other queue, with the argument x-queue-type: stream, and exchanges route messages to it exactly as they route to a standard queue.

Queue or stream: same messages, different outcome

The simulation below publishes every message twice: once to a standard queue and once to a stream. Two consumers read from each. Publish a few messages and compare what each consumer gets.

This interactive simulation needs JavaScript.

The standard queue splits the work. Each message goes to one consumer, and once it’s acknowledged, it’s gone. That’s exactly what you want in a task queue, where every job should run once.

The stream hands every message to every consumer, and the message stays in the log after it’s read. Watch the slower analytics consumer: it falls behind, then catches up without losing a thing. To get the same result with standard queues, you’d need one queue per consumer and a fanout exchange copying each message into all of them.

A stream is an append-only log

Three ideas explain almost everything a stream does:

Append-only: producers only add messages to the end. Nothing in the middle changes, and reading never removes anything.
Offset: every message gets a number that marks its position in the log, starting at 1.
Segments: on disk, we split the log into 8 MiB segment files. New messages go into the newest one, the active segment.

The segments aren’t an accident. LavinMQ writes every message to disk segments by design, and streams build directly on that. Appending to the end of a file is one of the cheapest things a disk can do.

Each consumer keeps its own position in the log, like a bookmark. A fast consumer never blocks a slow one because nobody takes anything away.

Pick a starting point with offsets

A consumer chooses where to start reading with the x-stream-offset argument when it subscribes. In the simulation, each segment holds only four messages, so the boundaries are easy to see. Click an offset to see where the consumer starts and which messages it reads.

This interactive simulation needs JavaScript.

Each offset value answers a different question:

first starts at the oldest message still in the stream. Use it to rebuild state or train on the full history.
last starts at the beginning of the last segment, not at the last message. When the simulation starts, that’s offset 9, the first message in the active segment.
next skips the history and waits for new messages only.
A number such as 6 starts at that exact offset. If the offset no longer exists, the consumer moves to the start or end of the stream.
A negative number such as -3 starts that many messages before the end, so -3 gives you the last three messages.
A timestamp starts at the first message published after that point in time. Perfect for questions like “what happened at 3 AM last night?”

Our guide to replay with offsets walks through each value with code.

Retention trims whole segments

A stream that keeps everything grows until the disk fills up. With retention off, the simulation stops at 40 messages to make that point. Retention settings stop the growth:

x-max-length keeps a target number of messages.
x-max-length-bytes keeps a target size on disk.
x-max-age drops segments whose newest message is older than a given age, such as 7D for seven days.

This interactive simulation needs JavaScript.

We never delete single messages from a stream. LavinMQ drops whole segment files, oldest first, and only when the remaining messages still meet the target. Deleting a file is much cheaper than picking messages out of one, so retention costs almost nothing.

Set x-max-length to 8 in the simulation and keep publishing. The stream never drops below 8 messages, and it grows to 12 before the oldest segment goes. Length retention runs when a new segment opens, and the active segment always stays, even if all its messages have expired.

So think of the limits as targets, not hard maximums. The stream truncation guide explains the trade-offs between size-based and time-based retention.

Let the broker remember the position

When a consumer crashes, it needs to know where it stopped. With server-side offset tracking, LavinMQ stores that position for you, so the consumer can reconnect and carry on without keeping its own bookmark.

Tracking needs a consumer tag that stays the same between connections. It turns on when the consumer subscribes with that tag and no x-stream-offset, or when it sets x-stream-automatic-offset-tracking together with a starting offset. LavinMQ stores the position as each message is delivered, not when it’s acknowledged.

Skip the messages you don’t need

With stream filtering, the broker drops irrelevant messages before they ever cross the network. Producers tag each message with the x-stream-filter-value header, and consumers subscribe with x-stream-filter to get only the matching ones. LavinMQ can also filter on arbitrary header values and on geographic coordinates.

Our guide to server-side offset tracking and stream filtering covers both features in depth.

When to choose a stream

Several services need the same events: one stream replaces a fanout exchange and a queue per service.
Replay matters: a new service, a bug fix or a new model can reprocess history from any offset.
Audit and debugging: the log keeps what happened, in order, for as long as retention allows.

A standard queue is still the better fit when each message is a job that should run once, or when you need dead lettering, priorities, or per-message TTL, since streams don’t support them.

Compared with RabbitMQ streams and Kafka, the biggest difference is the protocol. LavinMQ streams work over plain AMQP 0.9.1, so existing AMQP clients read and write them without a separate stream protocol. RabbitMQ recommends its dedicated stream protocol for the best performance, and Kafka needs its own clients. Kafka still offers features LavinMQ streams don’t, such as log compaction and partitioning, so the full comparison of LavinMQ streams, Kafka and RabbitMQ streams is worth a read before switching.

Wrap up

Streams give every consumer its own bookmark in the same log. Pick a starting offset, set a retention target, and let LavinMQ handle the rest.

Want to go deeper?

The LavinMQ streams guide builds a user activity tracker on streams, step by step.
The streams Q&A answers common questions about mixing queues and streams.
LavinMQ streams vs Kafka vs RabbitMQ streams compares the three.
The stream queues documentation lists every argument and restriction.
Our interactive guide to AMQP routing shows how exchanges decide which queue or stream gets a message.