Introduction
Configuration
Language Support
AMQP 0-9-1 Overview
More Exchange Types
More Consumer Features
Queue Deep-dive
Other Features
Reliable Message Delivery
High Availability
Monitoring
Management HTTP API
Tutorials
Networking
LavinMQ CLI
Prometheus
Generally, monitoring is the process of tracking a system’s behavior through health checks and metrics over time. It helps detect anomalies, such as unavailability, unusual loads, resource exhaustion, or deviations from normal behavior. This ensures early identification of issues for prompt action to maintain optimal performance and stability.
While the LavinMQ’s management interface provides an easy to use built-in monitoring solution, it is not ideal for long term metric collection, storage and visualisation.
This is where support for Prometheus in LavinMQ comes in.
What is Prometheus?
In a nutshell, Prometheus is a monitoring tool that collects and stores metrics from different systems using a simple scraping method. It saves the data in a database and allows users to analyze it using a user-friendly query language called PromQL.
When can I use Prometheus with LavinMQ
You can integrate Prometheus with LavinMQ:
- If you need a more long term metric collection and storage solution
- If you need to adopt an external monitoring solution and by extension separate the system being monitored from the monitoring system.
How does LavinMQ support Prometheus?
Every LavinMQ installation/instance ships with two metrics endpoints ouf of the box. These endpoints expose data in a Prometheus-compatible format. To integrate Prometheus with LavinMQ, you simply point your Prometheus scraper to these endpoints:
/metrics: This endpoint returns aggregate metrics of all the metrics emitting objects/metrics/detailed: This returns the metric for each metric emitting object depending on the query parameters it’s passed.
/metrics
Typically, Prometheus and other compatible solutions assume that metrics are accessible
at the /metrics path. LavinMQ, by default, provides aggregated metrics on this endpoint.
As mentioned earlier, this endpoint returns a combined metrics for all the metrics emitting objects defined in your LavinMQ server. Metrics emitting objects here refers to connections, channels, queues, and consumers amongst others.
The table below lists all the relevant metrics returned from this endpoint.
| Metric | Description |
|---|---|
| lavinmq_connections_opened_total | Total number of connections opened |
| lavinmq_connections_closed_total | Total number of connections closed or terminated |
| lavinmq_channels_opened_total | Total number of channels opened |
| lavinmq_channels_closed_total | Total number of channels closed |
| lavinmq_queues_declared_total | Total number of queues declared |
| lavinmq_queues_deleted_total | Total number of queues deleted |
| lavinmq_process_open_fds | Open file descriptors |
| lavinmq_process_open_tcp_sockets | Open TCP sockets |
| lavinmq_disk_space_available_bytes | Disk space available in bytes |
| lavinmq_process_max_fds | Open file descriptors limit |
| lavinmq_resident_memory_limit_bytes | Memory high watermark in bytes |
| lavinmq_connections | Connections currently open |
| lavinmq_channels | Channels currently open |
| lavinmq_consumers | Consumers currently connected |
| lavinmq_queues | Queues available |
| lavinmq_queue_messages_ready | Messages ready to be delivered to consumers |
| lavinmq_queue_messages_unacked | Messages delivered to consumers but not yet acknowledged |
| lavinmq_queue_messages | Sum of ready and unacknowledged messages - total queue depth |
| lavinmq_uptime | Server uptime in seconds |
| lavinmq_cpu_system_time_total | Total CPU system time |
| lavinmq_rss_bytes | Memory RSS in bytes |
All the metrics returned from this endpoint are prefixed with lavinmq by default.
However, this is customisable - just pass the the query param prefix
with your custom prefix like so: /metrics?prefix=prometheus.
/metrics/detailed
As opposed to returning aggregate metrics, this endpoint allows filtering metrics per object via query params. This way, you only get the metrics you need.
By default, this endpoint returns nothing until you pass a filter/query param.
With this endpoint, you can either pass one ore family values or one or more vhost values. Let’s cover each of them one at a time.
family:
In the context of these metrics, every entity on a LavinMQ node belongs to a family. E.g all queues belong to the queue family, connections to the connection family and so on and so forth. The family query parameter essentially allows you retrieve metrics for a specific group of objects(queues, connections, channels etc).
For example, if you want to grab metrics on queues only, you will pass the following values to the family parameter:
/metrics/detailed?family=queue_coarse_metrics
LavinMQ supports the following family values:
queue_coarse_metrics
In this group, every metric points to a particular queue through its label. As a result, the size of the response here grows in direct proportion to the number of queues hosted on the node.
Thus, passing this value to the family query param will return the following metrics for each queue defined on your LavinMQ node:
| Metric | Description |
|---|---|
| lavinmq_detailed_queue_messages_ready | Messages ready to be delivered to consumers |
| lavinmq_detailed_queue_messages_unacked | Messages delivered to consumers but not yet acknowledged |
| lavinmq_detailed_queue_messages | Sum of ready and unacknowledged messages - total queue depth |
connection_churn_metrics
This retrieves generic metrics on connection. Passing this value to the family query param will return the following metrics:
| Metric | Description |
|---|---|
| lavinmq_detailed_consumers_added_total | Total number of consumers added |
| lavinmq_detailed_consumers_removed_total | Total number of consumers removed |
| lavinmq_detailed_connections_opened_total | Total number of connections opened |
| lavinmq_detailed_connections_closed_total | Total number of connections closed or terminated |
| lavinmq_detailed_channels_opened_total | Total number of channels opened |
| lavinmq_detailed_channels_closed_total | Total number of channels closed |
| lavinmq_detailed_queues_declared_total | Total number of queues declared |
| lavinmq_detailed_queues_deleted_total | Total number of queues deleted |
queue_consumer_count
This metric serves a valuable purpose by promptly identifying consumer-related issues, such as when no consumers are online. That’s why it’s made available separately for easy access and monitoring.
Passing this value to the family query param will return just one metric:
| Metric | Description |
|---|---|
| lavinmq_detailed_queue_consumers | Consumers on a queue |
connection_coarse_metrics
Passing this value to the family query param will return the following metrics:
| Metric | Description |
|---|---|
| lavinmq_detailed_connection_incoming_bytes_total | Total number of bytes received on a connection |
| lavinmq_detailed_connection_outgoing_bytes_total | Total number of bytes sent on a connection |
| lavinmq_detailed_connection_process_reductions_total | Total number of connection process reductions |
channel_metrics
Passing this value to the family query param will return the following metrics:
| Metric | Description |
|---|---|
| lavinmq_detailed_channel_consumers | Consumers on a channel |
| lavinmq_detailed_channel_messages_unacked | Delivered but not yet acknowledged messages |
| lavinmq_detailed_channel_prefetch | Total limit of unacknowledged messages for all consumers on a channel |
vhost:
Family values would allow you retrieve metrics for a group of objects on a node.
For example, the queue_coarse_metrics will return metrics for all the queues
defined on a node. But what if you just want metrics for all the queues defined
in a specific vhost?
You can do that with the vhost query param like so:
/metrics/detailed?vhost=vhost-name&family=queue_coarse_metrics
The query above will return metrics for the queues in the specified vhost only.
Note: All the metrics returned from the /metrics/detailed endpoint are prefixed with lavinmq_detailed.
However, this is customisable - again, just pass the the query param prefix
with your custom prefix like so: /metrics/detailed?prefix=prometheus.
Ready to take the next steps? Here are some things you should keep in mind:
Managed LavinMQ instance on CloudAMQP
LavinMQ has been built with performance and ease of use in mind - we've benchmarked a throughput of about 1,000,000 messages/sec. You can try LavinMQ without any installation hassle by creating a free instance on CloudAMQP. Signing up is a breeze.
Help and feedback
We welcome your feedback and are eager to address any questions you may have about this piece or using LavinMQ. Join our Slack channel to connect with us directly. You can also find LavinMQ on GitHub.