Building a live Bluesky viewer with streams and WebSockets

Try streaming! This guide walks you through building a Node.js application that consumes the Bluesky Jetstream firehose feed (a steady stream of all public activity) and uses a LavinMQ stream to power a web-based real-time viewer. This is a powerful way to see all public activity as it happens, without needing to poll the API.

The diagram below illustrates the data flow from the social network firehose, through our real-time streaming architecture, and to a real-time web viewer.

First, let’s break down the key technologies and their roles in this architecture: Bluesky, LavinMQ, WebSockets, and Streaming.

Bluesky

Bluesky is a decentralized social network protocol. In this demo project, it serves as the source of our data. The Node.js application connects to the Bluesky “firehose,” which is a real-time, unfiltered feed of every public post and action happening on the network. This is the raw data stream we’ll be working with.

Read more in the Bluesky Jetstream documentation.

LavinMQ and WebSockets

WebSockets is a communication protocol that enables a persistent, two-way connection between a client (like a web browser) and a server. It allows the server to push updates to the client in real time.

LavinMQ is a fast, lightweight message broker. Think of it as the central nervous system of our project. It sits in the middle, taking in the stream of messages from the Bluesky firehose and efficiently distributing them to the web-based viewer.

In our project, the web app stays connected to LavinMQ via WebSockets to receive new posts instantly, without needing to refresh the page. This is what powers the live viewer.

LavinMQ Streaming

In the context of LavinMQ, streaming refers to a specific type of communication where messages are stored in an append-only log. This is different from traditional queues, where messages are consumed and removed. A stream allows for messages to be stored and replayed, and a consumer’s reading position, also called offset, can be tracked. This feature ensures no posts are missed and lets you resume from where you left off. This is particularly useful for scenarios where you need to analyze your data multiple times.

Advanced filtering in the web UI

The project also includes a custom web UI that enhances the user experience. Instead of just displaying raw data, the UI allows users to filter posts based on various criteria like message type or language. This filtering is done on the client-side, making the interface highly responsive and personalized without needing to re-query the broker. This entire system is built exclusively with LavinMQ and WebSockets, highlighting the power and flexibility of our technology for real-time applications.

Coming soon: Track stream position

LavinMQ will soon have the unique ability to track the consumer’s reading position. This feature will be available soon. In the LavinMQ management interface, you will be able to see precisely how far your consumer has read into the stream. This provides a clear audit trail and ensures you can resume from where you left off if the connection is lost.

The whole project can be cloned from this GitHub repository.

Have fun!

We can’t wait to see what you build! If you want to dive deeper, get some hands-on help, or share your project with fellow developers, join the LavinMQ Slack community. It’s a place to ask questions, get advice from the core team, and connect with other innovators using LavinMQ for their projects.