Content Comparison

...

The vast majority of custom metrics will be part of custom compiled devices. As an example, let’s assume you’re a device author and your device sends messages to a web service. You’d like to track the duration of the web service call, the upload volume (bytes uploaded) and rate (bytes per second) as well as the same metrics on the download side. You also want to track the message size sent to the service.

For tracking operation duration, you will need a histogram. You will also need a histogram to track the message size, transfer rate, and total transfer volume (one histogram for upload, and one for download).

Connexion’s BaseDevice class, from which all custom devices derive, contains scaffolding for declaring metrics. There are three methods exposed for creating metrics: GetOrCreateCounter, GetOrCreateGauge, and GetOrCreateHistogram. The MetricsProvider is also exposed should you need more advanced functionality.

Your metrics should be created in the InitializeMetrics() override. By using this method, your metrics will automatically be updated as users enable and disable metrics via the UI. Let’s look at an example device class:

  public class MetricSample : BaseDevice<MetricSampleConfiguration>
  {
    private IHistogram m_SendDurationHistogram;   // measure the time taken to send the payload
    
    public override void InitializeMetrics()
    {
      // construct your metrics here
      m_SendDurationHistogram = GetOrCreateHistogram("MyDevice.send_duration_in_s",                           // name of your metric. See notes about X.y syntax
                                                     "The duration (in seconds) to send to the foo service",  // description
                                                     new HistogramConfiguration                               // configuration
                                                     {
                                                       LabelNames = new[] { "Group", "Tab", "Channel", "ChannelKey" },          // associate this metric with the owner channel
                                                       Buckets = new double[] { 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10 }  // buckets (in seconds)
                                                     }).WithLabels(MessageChannel.GroupName, MessageChannel.TabName, MessageChannel.ChannelName, MessageChannel.ChannelKeyString);
    }

    public override Task ProcessMessageAsync(IMessageContext context, CancellationToken cancellationToken)
    {
      // ...your code           
      using (m_SendDurationHistogram.NewTimer())    // the Histogram.NewTimer() method is used to measure the wrapped methods duration.
      { 
        await SendToFooServiceAsync(...).ConfigureAwait(false);
      }
      // ... your code
    }
  }

• The histogram is defined on line 3 (using the interface type)

• The InitializeMetrics() method is called (automatically) after the device is loaded (after the Load method). Any labels used within the InitializeMetrics() method must be initialized in your constructor or Load methods.

• The histogram is created in the InitializeMetrics method.

• The operation to be measured is wrapped with the .NewTimer() method. The NewTimer method is a specialized function of the histogram type for measuring duration (in seconds). Other operations (like size) are measured using the .Observe(X) method.

The GetOrCreateX methods all take a name, description, and configuration parameter. Names must be globally unique, so you should namespace your metric using the notation Namespace.metric_name. The portion in front of the . is the namespace and is displayed in the UI as a separate block. This is typically your device name. The portion after the . is the name as presented to Prometheus. connexion_ is automatically added to your name for ease of use within Grafana. Note that convention is to use all lower-case with underscore separators for the name (the namespace can be upper case and is removed from the Prometheus name).

...

The configuration object defines the labels and buckets for the histogram:

new HistogramConfiguration
{
  LabelNames = new[] { "Group", "Tab", "Channel", "ChannelKey" },          
  Buckets = new double[] { 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10 }
}).WithLabels(MessageChannel.GroupName,
              MessageChannel.TabName, 
              MessageChannel.ChannelName, 
              MessageChannel.ChannelKeyString);

The LabelNames collection defines the labels which will be supplied for this metric. It is convention to include Group, Tab, Channel, and ChannelKey labels. Add additional labels as needed. For example, a “TargetUri” label would enable you to break down metrics on a per-target basis.

The Buckets collection defines the buckets into which each metric will be placed. Duration is measured in seconds in Grafana, so our buckets are [up to 1ms], [1ms to 5ms], [5ms to 10ms]…[10 seconds]. Values greater than 10 seconds go into a special infinite bucket.

The .WithLabels(...) method creates a child metric with the given label values. In this case, the label values never change, so we can define our class-level histogram using the .WithLabels(...) method. In some cases, the label values will change dynamically. In this cases, you should define the class-level histogram without the .WithLabels(...) method, and then use the .WithLabels(...) each time you write a value.

m_SendDurationHistogram = new HistogramConfiguration
{
  LabelNames = new[] { "Group", "Tab", "Channel", "ChannelKey", "DynamicValue" },          
  Buckets = new double[] { 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10 }
});

public override Task ProcessMessageAsync(IMessageContext context, CancellationToken cancellationToken)
{
  // ...your code
  var dynamicValue = context.Message.SomeValueThatChangesWithEachMessage;
  using (m_SendDurationHistogram.WithLabels(MessageChannel.GroupName, MessageChannel.TabName, MessageChannel.ChannelName, MessageChannel.ChannelKeyString, dynamicValue).NewTimer())
  { 
    await SendToFooServiceAsync(...).ConfigureAwait(false);
  }
  // ... your code
}

Once you have deployed your device and processed at least one message, your metric will be published on the Connexion http metric endpoint. At this point you can open Grafana in a browser and start creating a dashboard.

Custom Metrics in the UI

In order for custom metrics to appear in the Connexion UI metric tab, they must be created/accessed once. The process of calling the GetOrCreateX method will register a metric within Connexion. Currently, the Connexion UI must be restarted (or navigate away and back to the metric tab), for newly registered metrics to be displayed.