Developer insights: Switching to jump hash algorithm for LavinMQ consistent hash exchange

Our investigation, aiming to improve the current consistent hash exchange in LavinMQ, led us to Jump consistent hash. This new approach showed impressive results with more evenly distributed queues and reduced message shuffling, but also proved the high value of user feedback.

Implementing Jump consistent hash

Jump consistent hash is an algorithm that uses a function to instantly calculate the probability of whether messages should, or need to relocate (jump) when queues are added or removed. Since the function is based on fast math, it only needs the key and the number of queues. Simplified, the algorithm ensures that the least amount of messages needs to jump to a new queue from each existing one.

The previous solution for LavinMQ consistent hash exchange was based on the algorithm Ring consistent hash. While Ring has advantages, its binary search function can cause messages to be distributed unevenly between the bound queues, putting a heavy load on one consumer while others idle. Something that this issue made us aware of.

Becoming default in 3.0

When the decision was made, implementation started, and in a couple of weeks, with some help from Claude, testing resulted in an improvement of an issue we had wanted to improve for a long time. The new consistent hash exchange aligns with our aim for LavinMQ and also proves that user feedback is of high value.

Ring will remain the default algorithm. We’ll switch to Jump as the default in version 3.0, since it’s a breaking change. However, users can opt in to use Jump starting from version 2.7.