DAT Systems left Apache Kafka - experiences significant improvements with LavinMQ

Success isn’t determined by size alone. It’s a blend of skills, timing, coincidences, and a bit of luck. DAT Systems, based in Pakistan, is living proof of this. With some help from LavinMQ, they went from a small startup into a company working with Pakistan’s largest public sector employer in just eight months.

DAT Systems started as three friends looking for a solution to the chaos of Pakistan’s election day operations in February 2024. Initially, their idea was to track on-ground staff during elections, but an unexpected internet shutdown on election day derailed that plan. While it was a setback, it also presented an opportunity.

– A friend’s discussion revealed a need for a field staff management system. In Pakistan, where broadband access is limited and digital literacy is low, most people rely on mobile phones, said Ahsan Nabi Dar, CTO and co-founder of DAT Systems.

Many organizations in Pakistan were already using WhatsApp and other messaging systems to track staff because custom apps were challenging to implement, especially with poor connectivity. WhatsApp offered location sharing, video, pictures, and audio messages—features that suited the needs of field workers who struggle with English keyboards and often communicate through voice notes or “Roman Urdu” (Urdu written in English script).

– Custom apps were often impractical in remote areas, so we decided to build a system that integrated with these existing communication channels, addressing the unique challenges of our market.

Elixir Phoenix API receives messages from WhatsApp, Telegram, and WeChat. It then routes these messages to their corresponding queues in LavinMQ.

This small strategic shift grew into something bigger than DAT Systems team could have imagined. Fast forward to today, DAT Systems is working with Lahore Waste Management Company (LWMC), the largest public sector employer in Pakistan, with over 16,000 employees. LWMC’s field operations have been significantly enhanced by their new system. By integrating communication tools with backend management, the system enables more effective staff monitoring. This has been crucial in addressing a widespread problem: supervisors misusing janitorial staff for personal tasks instead of assigned fieldwork. The system ensures greater accountability and transparency, leading to improved efficiency and fairness in LWMC’s operations.

DAT Systems AI anomaly detection engine has helped LWMC to reduce the ghosting by its field staff from 14 percent to less than 1 percent. By adopting LavinMQ DAT Systems have been able to scale their AI pipelines more reliably to provide anomaly detection with higher throughput and lower latency.

But like many startups before them, they faced some technical challenges along the way.

The Apache Kafka problem and the search for a solution

– When we started building our system, we chose serverless Apache Kafka as our messaging backbone. It seemed like a great fit at first, with all the buzz around its performance. But using Apache Kafka proved to be more of a headache than we anticipated. We encountered persistent issues such as “broker not found,” “leader missing,” retry failures, or duplicate messages, said Ahsan Nabi Dar.

For a small startup trying to grow fast, these problems were unacceptable. They needed a more efficient, reliable solution and service provider. In the search for better options, Ahsan Nabi Dar came across CloudAMQP’s LavinMQ service. The promises of higher throughput and lower latency all on an industry wide-open source protocol were intriguing, so they decided to give it a try.

– I installed it locally to where I was running Apache Kafka to see if it could fit our needs. I was amazed to find that our memory usage dropped from 5 GB to under 40 MB in an instance in a development environment. It ran smoothly, provided a console with end-to-end observability, and everything just worked. That was my “wow” moment—I realized if it could perform this well with so much fewer resources, it was worth considering.

– Upon migrating to CloudAMQP’s LavinMQ service, we experienced significant performance improvements. Message-consuming times decreased threefold, while message publishing times improved 300-fold compared to our previous Kafka integration. These improvements were consistent across our globally distributed app, which is deployed in five regions.

Broadway consumers are bound to each queue—one for WhatsApp, one for Telegram, and so on. These consumers subscribe to messages from the queue, process them, and write them to a database. Messages are also added to a backup queue, which is read by consumers from vector.dev. These consumers back up the messages in cloud storage. An Oban worker publishes DB audit logs to LavinMQ, which are also consumed by vector.dev.

Switching to LavinMQ: a smooth transition

Impressed by these results, DAT Systems decided to fully commit to LavinMQ. Their system is written in Elixir, and they use Broadway for data consumption. Broadway support integrations to different sources, such as Amazon SQS, Apache Kafka, Google Cloud PubSub, RabbitMQ, LavinMQ and others.

– We made the switch over a weekend, without any downtime, thanks to how easy it was to adapt our code. I only needed to modify a small portion of code. We’re using Broadway to consume data from the queues, which actually offers better support for LavinMQ than Apache Kafka.

In this project, DAT Systems are building the application in Elixir.

– The main task was changing the Broadway connector code, as everything else Broadway does in Elixir is standardized. So, we simply updated the connection code for Broadway, and the consumer was automatically migrated.

LavinMQ becomes the backbone

Today, LavinMQ is the cornerstone of DAT Systems communication and operations. It handles not only field staff communications but also backups, inter-service communication, and even auto-registration for their system users. They have scaled from handling around 4,000 daily messages to processing over 50,000 messages daily—across multiple platforms like WhatsApp, Telegram, and WeChat.

– From message routing and backups to handling critical HR processes like payroll, LavinMQ has become the heartbeat of our system. We’re already managing thousands of staff members for LWMC and looking to scale even further. By early next year, we expect to handle millions of messages each month.

Looking ahead

The journey with LavinMQ has been transformative for DAT Systems.

– It’s not just a message queue for us. It’s a tool that’s helping us grow faster, operate smoother, and handle more complexity. Our success so far has opened doors to even more opportunities. We’re now talking to other public sector organizations in Pakistan and exploring possibilities to expand our field staff management system into new industries.

– As we continue to scale, LavinMQ will play an even bigger role in our operations. In just a few months, LavinMQ has proven to be more than just a tech solution; it’s become a key partner in our journey. And with LavinMQ, the future looks promising.