Avalanches start at the peak

It’s Saturday night, and the final of a TV talent show is live. Across the country, two million people are on the couch with their phones in hand, waiting to vote for their favourite.

At 21:43, the host turns to the camera: “The lines are now open.”

Two million thumbs hit “vote” at the same time.

For the viewers, this is the most exciting moment of the show. For the system behind the voting app, it’s the most intense one.

I’ve come to think of the moment the lines open as the clearest picture of what a message queue is for. A queue makes a simple promise: “I’ll hold this until you’re ready.” The voting app can receive votes much faster than the systems behind it can count them, so the queue sits in between and absorbs the difference.

When the lines are open, the message queue is what stands between millions of votes and a crashed system.

I’ve written before about the quiet, everyday work message queues do, and why nobody talks about it. This post is about the one moment they can’t hide: the peak.

What 14 years of peaks taught us

I work at 84codes, the company behind CloudAMQP and LavinMQ. CloudAMQP has been hosting message brokers since 2012, more than 1.6 million instances so far, and I’ve been there for 12 of those years. It gives you a unique view of the world. You don’t just see one system; you see thousands: a TV vote on a Friday night. A webshop at midnight on Black Friday. Ten thousand sensors coming back online at the same second after a power cut, all trying to send the data they saved up while they were offline. Different brokers, different industries, different teams. Same lesson every time: systems don’t fail on an average day. They fail at the peak.

The pattern goes like this. Traffic spikes, consumers fall behind, and the queue grows. That’s exactly what it should do; it’s the whole point. But the longer the queue gets, the more pressure it puts on the broker. Memory climbs until the broker has to start protecting itself, usually right when you need it most.

That’s not a flaw in any particular product. It’s a trade-off many systems make, and most of the time it’s a sensible one. But it leaves you with a paradox: the tool built for peaks is under the most strain during peaks.

When a peak goes wrong, it’s rarely just the peak. Usually something has been building quietly for a while, and the peak is the last straw.

A recent case from the CloudAMQP support team is a good example. A customer ran a large outbound campaign that produced around 425,000 messages in five hours, most of them into a single queue. The consumers couldn’t keep up, so the queue grew. But the server was already carrying weight from before: a few queues without consumers had been collecting messages for weeks, and nothing was alerting on memory or queue length. When the campaign hit, memory ran out, publishing went from a tenth of a second to more than two minutes, and the server restarted.

Our support team moved the instance to a bigger plan so it could recover. Once the backlog was drained, the customer set up alerts and cleaned up the forgotten queues. We’ve seen that story many times, in different shapes and on different brokers. The peak gets the blame, but the load had been piling up long before.

Why does a message broker use so much memory?

The short answer: during a traffic spike, consumers fall behind, and the queue grows. In many brokers, every message waiting in that queue costs memory, so the bigger the spike, the bigger the pressure.

We’ve answered different versions of that question for 14 years. In support tickets, in chat, in the middle of the night. Almost always during a peak, and almost always while someone’s production traffic was waiting.

Many brokers keep part of every waiting message in memory: the message itself (or at least a reference to where it is stored on disk). That’s fast when queues are short, but when a queue grows to hundreds of thousands of messages, memory grows with it. In the TV vote, that means two million waiting votes, each one taking up a little memory.

When I interviewed Carl Hörberg, our founder, about why we built LavinMQ, he went straight back to those support tickets:

We wanted to be able to give really good answers to our customers. Why does this use memory? Why can’t we scale here? We wanted to have full control of the whole experience.

So the team asked a different question. What if a growing queue wasn’t an emergency? What if a broker treated two million waiting votes as a perfectly normal Tuesday?

How LavinMQ handles long queues without running out of memory

That question led to LavinMQ: an open-source message broker that writes every message to an append-only log on disk, so the length of a queue has little effect on how much memory it uses.

Carl summed up the core idea in a couple of sentences. When a message arrives, LavinMQ reads it from the network, works out where it should go, and writes it straight to an append-only log on disk for that queue. As little as possible stays in memory, and the message gets copied as few times as possible on the way. Back to the TV vote: when the lines open, every vote goes straight into the log on disk, so memory doesn’t have to grow with the queue.

Memory bandwidth and system calls are often the expensive parts in message brokers.

Because LavinMQ doesn’t keep a message index in memory, a queue can grow, in Carl’s words, to be “basically infinitely long”. The queue grows on disk, not in memory. The same idea is behind LavinMQ’s stream queues, where consumers can read and replay messages from any point in the log.

Ahsan Nabi Dar, CTO at DAT Systems, saw memory use drop from 5 GB to under 40 MB in a development environment. I love that number, because it answers the question we’ve heard for 14 years: a long queue doesn’t have to mean high memory use.

Surviving the peak, whatever broker you run

Here’s what we see again and again in support:

Know your queue length before your users do. Alert on queue growth, not just on errors.
Plan for consumers falling behind. Make it easy to add more consumers when the queue starts growing, and tune prefetch so each one gets a fair share.
Decide what happens to old messages. A max length or a TTL keeps a queue from growing forever when something downstream is stuck.
Acknowledge your messages. Messages that are delivered but never acknowledged stay with the broker and quietly add to the load.
Test the peak, not the average. Your system is only as strong as its worst ten minutes.

Summary

Message queues are built to grow during traffic spikes. That’s their job: they absorb the difference between how fast messages arrive and how fast consumers can handle them. In many brokers, though, a longer queue also means more memory, which is why problems tend to show up exactly at the peak. LavinMQ writes messages to an append-only log on disk instead, so a growing queue doesn’t have to turn into a memory problem.

Built for the moment it matters

Peaks will come. The question is whether your queue is ready when they do.

LavinMQ is named after the Swedish word for avalanche, and strictly speaking, avalanches rarely start at the very top of a mountain. They start on the slope, when the load exceeds what the snow can hold. Message queues work the same way, and that moment is exactly what we built LavinMQ for.

LavinMQ is open source and speaks AMQP, so your existing clients work as they are. Spin up a free LavinMQ instance on CloudAMQP in under two minutes, or follow the getting started guide to run it yourself.