Prometheus

Generally, monitoring is the process of tracking a system’s behavior through health checks and metrics over time. It helps detect anomalies, such as unavailability, unusual loads, resource exhaustion, or deviations from normal behavior. This ensures early identification of issues for prompt action to maintain optimal performance and stability.

While the LavinMQ’s management interface provides an easy to use built-in monitoring solution, it is not ideal for long term metric collection, storage and visualisation.

This is where support for Prometheus in LavinMQ comes in.

What is Prometheus?

In a nutshell, Prometheus is a monitoring tool that collects and stores metrics from different systems using a simple scraping method. It saves the data in a database and allows users to analyze it using a user-friendly query language called PromQL.

When can I use Prometheus with LavinMQ

You can integrate Prometheus with LavinMQ:

If you need a more long term metric collection and storage solution
If you need to adopt an external monitoring solution and by extension separate the system being monitored from the monitoring system.

How does LavinMQ support Prometheus?

Every LavinMQ installation/instance ships with two metrics endpoints ouf of the box. These endpoints expose data in a Prometheus-compatible format. To integrate Prometheus with LavinMQ, you simply point your Prometheus scraper to these endpoints:

/metrics: This endpoint returns aggregate metrics of all the metrics emitting objects
/metrics/detailed: This returns the metric for each metric emitting object depending on the query parameters it’s passed.

/metrics

Typically, Prometheus and other compatible solutions assume that metrics are accessible at the /metrics path. LavinMQ, by default, provides aggregated metrics on this endpoint.

As mentioned earlier, this endpoint returns a combined metrics for all the metrics emitting objects defined in your LavinMQ server. Metrics emitting objects here refers to connections, channels, queues, and consumers amongst others.

The table below lists all the relevant metrics returned from this endpoint.

Metric	Description
lavinmq_connections_opened_total	Total number of connections opened
lavinmq_connections_closed_total	Total number of connections closed or terminated
lavinmq_channels_opened_total	Total number of channels opened
lavinmq_channels_closed_total	Total number of channels closed
lavinmq_queues_declared_total	Total number of queues declared
lavinmq_queues_deleted_total	Total number of queues deleted
lavinmq_process_open_fds	Open file descriptors
lavinmq_process_open_tcp_sockets	Open TCP sockets
lavinmq_disk_space_available_bytes	Disk space available in bytes
lavinmq_process_max_fds	Open file descriptors limit
lavinmq_resident_memory_limit_bytes	Memory high watermark in bytes
lavinmq_connections	Connections currently open
lavinmq_channels	Channels currently open
lavinmq_consumers	Consumers currently connected
lavinmq_queues	Queues available
lavinmq_queue_messages_ready	Messages ready to be delivered to consumers
lavinmq_queue_messages_unacked	Messages delivered to consumers but not yet acknowledged
lavinmq_queue_messages	Sum of ready and unacknowledged messages - total queue depth
lavinmq_uptime	Server uptime in seconds
lavinmq_cpu_system_time_total	Total CPU system time
lavinmq_rss_bytes	Memory RSS in bytes

All the metrics returned from this endpoint are prefixed with lavinmq by default. However, this is customisable - just pass the the query param prefix with your custom prefix like so: /metrics?prefix=prometheus.

/metrics/detailed

As opposed to returning aggregate metrics, this endpoint allows filtering metrics per object via query params. This way, you only get the metrics you need.

By default, this endpoint returns nothing until you pass a filter/query param.

With this endpoint, you can either pass one ore family values or one or more vhost values. Let’s cover each of them one at a time.

family:

In the context of these metrics, every entity on a LavinMQ node belongs to a family. E.g all queues belong to the queue family, connections to the connection family and so on and so forth. The family query parameter essentially allows you retrieve metrics for a specific group of objects(queues, connections, channels etc).

For example, if you want to grab metrics on queues only, you will pass the following values to the family parameter:

/metrics/detailed?family=queue_coarse_metrics

LavinMQ supports the following family values:

`queue_coarse_metrics`

In this group, every metric points to a particular queue through its label. As a result, the size of the response here grows in direct proportion to the number of queues hosted on the node.

Thus, passing this value to the family query param will return the following metrics for each queue defined on your LavinMQ node:

Metric	Description
lavinmq_detailed_queue_messages_ready	Messages ready to be delivered to consumers
lavinmq_detailed_queue_messages_unacked	Messages delivered to consumers but not yet acknowledged
lavinmq_detailed_queue_messages	Sum of ready and unacknowledged messages - total queue depth

`connection_churn_metrics`

This retrieves generic metrics on connection. Passing this value to the family query param will return the following metrics:

Metric	Description
lavinmq_detailed_consumers_added_total	Total number of consumers added
lavinmq_detailed_consumers_removed_total	Total number of consumers removed
lavinmq_detailed_connections_opened_total	Total number of connections opened
lavinmq_detailed_connections_closed_total	Total number of connections closed or terminated
lavinmq_detailed_channels_opened_total	Total number of channels opened
lavinmq_detailed_channels_closed_total	Total number of channels closed
lavinmq_detailed_queues_declared_total	Total number of queues declared
lavinmq_detailed_queues_deleted_total	Total number of queues deleted

`queue_consumer_count`

This metric serves a valuable purpose by promptly identifying consumer-related issues, such as when no consumers are online. That’s why it’s made available separately for easy access and monitoring.

Passing this value to the family query param will return just one metric:

Metric	Description
lavinmq_detailed_queue_consumers	Consumers on a queue

`connection_coarse_metrics`

Passing this value to the family query param will return the following metrics:

Metric	Description
lavinmq_detailed_connection_incoming_bytes_total	Total number of bytes received on a connection
lavinmq_detailed_connection_outgoing_bytes_total	Total number of bytes sent on a connection
lavinmq_detailed_connection_process_reductions_total	Total number of connection process reductions

`channel_metrics`

Passing this value to the family query param will return the following metrics:

Metric	Description
lavinmq_detailed_channel_consumers	Consumers on a channel
lavinmq_detailed_channel_messages_unacked	Delivered but not yet acknowledged messages
lavinmq_detailed_channel_prefetch	Total limit of unacknowledged messages for all consumers on a channel

vhost:

Family values would allow you retrieve metrics for a group of objects on a node. For example, the queue_coarse_metrics will return metrics for all the queues defined on a node. But what if you just want metrics for all the queues defined in a specific vhost?

You can do that with the vhost query param like so:

/metrics/detailed?vhost=vhost-name&family=queue_coarse_metrics

The query above will return metrics for the queues in the specified vhost only.

Note: All the metrics returned from the /metrics/detailed endpoint are prefixed with lavinmq_detailed. However, this is customisable - again, just pass the the query param prefix with your custom prefix like so: /metrics/detailed?prefix=prometheus.

Introduction

Configuration

Language Support

AMQP 0-9-1 Overview

More Exchange Types

More Consumer Features

Queue Deep-dive

Other Features

Reliable Message Delivery

High Availability and Backup

Monitoring

Management HTTP API

Tutorials

Networking

LavinMQ CLI