LavinMQ LavinMQ logo
LavinMQ logo
Home Documentation Blog Community
Get started
GitHub Repository

Part 1 - An overview of LavinMQ streams

Long before streams were introduced, LavinMQ only supported one queue type - for the purpose of this guide, let’s call it the traditional queue. Traditional queues are great and, in fact, work really well in most use cases. Regardless, they are limited in certain ways:

  1. They send the same message to multiple consumers by creating a separate queue for each one, which makes scaling more resource-intensive.
  2. Once a message is read from a queue and acknowledged, it is erased. Thus, you can’t re-read the same message multiple times or go back in time and start reading from a point in the queue.

Stream Queues, or simply Streams in LavinMQ, were introduced to mitigate these challenges.

What are streams?

LavinMQ Streams perform similar tasks as the traditional queues - they buffer messages from producers for consumers to read. However, Streams in LavinMQ differ from the traditional queues in one significant way:

  • Streams are immutable: Meaning, messages written to a Stream cannot be erased, they can only be read. Even though this behaviour is configurable to some extent(more on this later), Streams have been designed to persist messages permanently. As a result, consumers can subscribe to a Stream and read the same message as many times as needed.

It is important to note that the Stream queue in LavinMQ wasn’t created to replace the traditional queues but to complement them. Streams open up new possibilities for LavinMQ use cases. Let’s explore these use cases in more detail.

LavinMQ Streams use-cases

Streams are great for:

  • Fan-out architectures: Where many consumers need to read the same message.
  • Replay (time-travel): Where consumers need to re-process the same messages or start reading from any point in the Stream.
  • High throughput: Where producers need to stream thousands or even millions of messages per second.

Fan-out architectures

Implementing a fan-out architecture—where multiple services need to process the same event—can get messy with traditional queues in LavinMQ. Why? Each new consumer typically requires its own dedicated queue, duplicating messages and adding complexity. As the number of consumers grow, this approach becomes harder to scale.

With LavinMQ Streams, fan-outs are a breeze. Streams let consumers read messages non-destructively, meaning, the same event stays available for all consumers without duplication. Each service can process the same data at its own pace.

Let’s tie this to our user activity tracking app example:

  • The real-time analytics service reads the stream to keep dashboards up to date.
  • The recommendation engine analyses the same events to offer personalised suggestions.
  • The session analysis service uses the stream to piece together detailed user journeys.

You just need to declare a single stream for your user activity events and let all three services consume from it. No juggling extra queues. No unnecessary duplication. Just scalable, efficient fan-out. The accompanying illustration provides a visual depiction of what implementing a fan-out using a Stream would look like:

Fanouts with lavinmq streams

In contrast, achieving the same result with queues would involve a more complex setup, as shown in the image below:

Fanouts with lavinmq amqp queues

Replay( time travel )

Sometimes, you just need to hit rewind. Maybe a bug in one of your services caused it to miss some events. Or maybe you deployed a new feature and want to reprocess past events to see how it performs. Whatever the case, Streams offer the possibility to replay old messages or attach to a specific point in the stream.

With LavinMQ Streams, replaying events is as easy as picking a point in time or an index(formerly called offsets, we will get to that later) and consuming from there. Unlike traditional queues, where messages are gone once acknowledged, streams keep the event history intact, making it possible to revisit old data whenever you need it.

In our user activity tracking app, the recommendation engine could roll out a new algorithm and wants to apply it to historical data. With Streams, it can reprocess past clicks, searches, and page views to improve suggestions.

High throughput

When your application generates a lot of data—clicks, page views, searches, and more—you need a message broker that can keep up. LavinMQ Streams and LavinMQ in general were built for simplicity and speed, capable of comfortably processing millions of messages per second.

Working with Streams in LavinMQ

Consumers interact with Streams in LavinMQ using AMQP-based clients and the AMQP protocol, just as they do with queues. Currently, LavinMQ does not support any dedicated Stream protocol.

As with traditional queues, there are three steps to working with LavinMQ Streams via an AMQP client library:

  • Declare/Instantiate a Stream
  • Publish(write) messages to the Stream
  • Consume(read) messages from the Stream

In the next part in this series, we will look at creating a stream queue, publishing and consuming messages from the queue.

What’s next?

Previous Next