Please review the metrics overview if you haven’t yet.
Once you have Connexion, Prometheus, and Grafana configured, you can leverage this infrastructure to view built-in and custom metrics. Built-in metrics are compiled with each version of Connexion. Custom metrics can be written in compiled custom devices as well as custom code-based devices.
Metrics Configuration
The metrics configuration screen can be used to enable and disable metrics, as well as configure which channel(s), plugin(s), and repositories are associated with a specific metric.
Metrics calculations can consume considerable resources. Do not arbitrarily enable metrics on production systems without ensuring:
The metrics will be used / monitored
The additional load will not degrade message processing performance
Always verify proper system operation and performance after changing the metrics configuration.
This screen follows the same pattern as the Alerting, Authorization, and Monitoring screens. You create a metric group to configure the enabled state for one or more metrics (which may be tied to a specific set of channels, plugins, repositories, etc.). By creating multiple metric groups, you can create a fine-grained set of enabled metrics. The metrics system will search all metric groups to determine if a metric should be written for a specific connexion item (channel, plugin, repo, etc.).
By default, all channels, plugins, and repositories are enabled. If you wish to enable a metric for a specific object, you must explicitly deselect objects to be ignored.
Metric Association
Most metrics are associated with Connexion objects. For example, the connexion_channel_message_count metric is only written for selected channels. The connexion_repository_disk_space metric is only written for selected repositories.
In some cases, the same metric exists in multiple locations. For example, connexion_channel_message_count (calculated at the channel-level) is also provided at the system-level by connexion_system_message_counts. The typical (Grafana) method for calculating the system-level message counts would be to sum all the connexion_channel_message_count metrics, however, this would require Connexion to publish this metric for every channel. In a system with thousands of channels, this may an be unnecessary overhead when the goal is to display a single system-level count.
Some metrics may simply be on or off, such as the process-level metrics.
Enabling a Metric
To enable a metric, click the checkbox beside its name. If you wish to write this metric only for a specific set of connexion objects, navigate to the appropriate tab and set the corresponding checked state. Save the metric.
Metrics are typically created on first use. This means you may need to start a channel or process a single message before the metric is published to the http endpoint.
You can verify that a metric is being properly published by visiting the metrics endpoint of your Connexion server (https://your-server:8092/metrics) and searching for the metric name.
Understanding Prometheus Metrics
Prometheus metrics are comprised of 2 parts: a name, and zero or more labels. The name identifies the metric (connexion_channel_message_count in the image above), and the labels identify any association(s). The labels in the above image Group=X , Tab=Y , Channel=Z etc. denote the group/tab/channel for which we are writing message count values. In Prometheus (and Grafana) we can query on these labels, allowing us to create a graph or table for a specific channel. We can also use functions to create other metrics, such as a sum , average , or delta (among many others) of all queued messages with a tab label equal to ‘XYZ’.
If you create your own metrics, you will need to think about the types of visualizations you want to support, and ensure you supply labels which support this.
Metric Types
Connexion supports 3 types of metric: Counters, Gauges, and Histograms.
Counters are for data that only ever increases (such as request count) and reset to zero when the service restarts. When you want to show the rate/speed of an operation, you typically want a Counter.
Gauges are like counters, but allow values to increase and decrease. For example, cpu and memory metrics would be gauges.
Histograms allow you to measure the distribution, count, and rate/time. For example, if you wanted to measure the size of a payload, a histogram would show you how many of your payloads were between X and Y in size, and how many were between Y and Z in size etc. It can also calculate the rate payload operations as well as other more advanced metrics (such as how many payloads were above/below a certain threshold).