Metrics Overview

BETA

We are actively developing and refining this feature. Usage and APIs are subject to change. Please send your feedback.

Version 16.1 introduces a new metrics platform based on Prometheus and Grafana. This feature allows you to build advanced dashboards, reports, and alerts based on both built-in and custom developed metrics. Connexion already provides several built-in dashboards which provide a general overview of current state and speed, however, these UIs are not customizable nor are they historical (except for the legacy processing history graphs).

The new metrics API lets you define your own metrics (typically within your custom devices) which can surface performance and operational state. By default, this data is exposed to Prometheus, where it is stored and queried to produce tables and visuals within Grafana.

As Prometheus and Grafana are external to Connexion, you can create your own queries, charts, tables, and alerts independent of Connexion version. You can also aggregate data across many different Connexion instances, or other platforms such as AWS, operating systems, database hosts, etc.

Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit. It is a standalone open source project and maintained independently of any company. To emphasize this, and to clarify the project's governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.

Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.

Grafana

Grafana allows you to query, visualize, alert on and understand your metrics no matter where they are stored. Create, explore, and share beautiful dashboards with your team and foster a data driven culture. Read more about Grafana.

Architecture Overview

Connexion metrics are http-hosted in the Prometheus text format. Connexion aggregates metrics across all child processes and then surfaces them over an http(s) connection. Prometheus is configured to ingest these metrics and store them in its time-series database. The frequency of ingestion and the data retention period are configurable within Prometheus.

Metrics calculation requires extra cpu and memory resources on your Connexion server(s). You must ensure there is enough headroom to minimize any impact on message processing. If you choose to host Prometheus and/or Granafa on your Connexion servers, you must also account for extra disk I/O.

Performance Implications

In order to minimize any performance degradation, each metric can be individually enabled (often on a per-channel basis). We strongly recommend you only enable metrics being actively monitored, or, only enable metrics required for debugging/diagnostic purposes temporarily.

Using a Grafana dashboard to monitor the performance of Connexion servers can be a good way to gauge the impact of enabled metrics.

Enable Connexion Metrics

The Connexion metrics platform is disabled by default. To enable the metrics endpoint (which will be scraped by Prometheus), run the Configuration Wizard and navigate to the Services Options page.

Enable the endpoint and select a port (if the default is already in use). Complete the configuration wizard and then navigate to the metrics endpoint https://your-server-name:8092/metrics. The metrics endpoint uses the default Connexion self-signed certificate, so you will need to trust this certificate in your browser.

Your browser output should appear similar to this:

Once you have confirmed the metrics endpoint is working, you will need to point your installed copy of Prometheus at it.

Prometheus Scraping

Once you have installed Prometheus (https://prometheus.io/docs/prometheus/latest/installation/, no additional packages are required).

Navigate to the prometheus.yml configuration file.

Update the configuration file with the following:

job_name: You can optionally set this to ‘Connexion’. This can be useful if you choose to include other metrics sources (database servers, AWS, etc.)
scheme: This must be changed to https
targets: For each Connexion instance you wish to scrape, enter a - targets: MyCxnServerDnsName:Port entry.
labels / instance: If your server DNS name is not what you want displayed in your dashboards, enter a descriptive name in the instance block.
insecure_skip_verify: Set this to true to tell Prometheus to accept the Connexion self-signed certificate.

Example:

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "Connexion"
    scheme: https

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:8092"]
        labels:
          instance: 'JR-Work'
    tls_config:
       insecure_skip_verify: true

Prometheus exposes a web console on port 9090. Navigate to this page on your Prometheus host and then select the Status menu and Targets item. If Prometheus is able to scrape the Connexion endpoint, your screen should display an UP state in green.

Once you have confirmed the UP state, you should install and configure Grafana.

Prometheus doesn’t support installation as a service (on Windows) out of the box. If you want to install Prometheus as a service on Windows, you will need to download nssm, or, use the copy of nssm distributed with Grafana. You can use the following commands to install prometheus as a service:

nssm install prometheus <path to prometheus.exe here>
nssm set prometheus AppParameters --web.listen-address :9092 <--only if you want to change the default port
nssm set prometheus AppParameters --storage.tsdb.retention.time 365d <--only if you want to change the default retention

Grafana Configuration

Once installed, Grafana exposes a web console on port 3000. Navigate to your Grafana host and select the Add your first data source link.

Select the Prometheus option and fill in the Host address. If your Grafana instance will be accessible outside a secured network, you must secure it (there are many options for this).

Now that you have the metrics infrastructure in place, learn how to create and configure metrics within Connexion.