Alerting

Owned by Nick Judson
Last updated: Mar 04, 2016
4 min read

The alerting updates in 14.1 allow you specify multiple alert 'targets', such as SMTP, SMS, SNMP, as well as custom targets. Additionally, alerts can be routed and filtered based on day/time and alert source (group, tab, channel). These new features make it possible for different stakeholders to receive a specific alert sets. By default, 14.1 systems have one 'Default' alert group receives all events. By creating new alert groups, you can route your alerts as required.

Alerts are configured via the 'System Configuration' icon (gear) and alerting tab. Alert groups are listed on the left side and configuration tabs and options are listed on the right.

By creating multiple alert groups, you can specify where, when, and from which channel(s) alerts will be sent. You may wish to send alerts from certain groups/tabs/channels to a specific person/group, or perhaps you wish to create 'office hours' and 'after hours' support groups and have alerts route depending on the day and time of day.

The first tab ('Options') groups all the settings from 14.0. These settings let you specify the general system-level alert settings such as the minimum alert interval.

The filtering pane at the bottom of this tab let's you filter out specific alerts ("Include all events except"), or only include specific alerts ("Exclude all events except"). Find the event ID you with to include/exclude by viewing events within the events dialog (the icon to the right of the gear icon on the main window) and enter the event ID into the include/exclude pane. Note that text after a '#' is treated as a comment.

The Email, SNMP, and SMS (text message) tabs let you configure these optional endpoints. The fields on these tabs are self-explanatory, although please note that SMS requires the use of a Twilio account.

   

 

The 'Custom' tab contains a custom code device which let's apply custom logic to alerts, as well as to apply custom filtering. This custom code device acts in the same fashion as any other, but uses an event callback "OnAlertReceived" to let you add custom logic. The "OnAlertReceived" method includes an AlertReceivedEventArgs parameter which contains information about the event being generated. The object contains the following items:

  • Event: The full event object of the generated event. This object contains the event ID, severity, description, created datestamp, source channel key, and source device key.
  • IsInExcludeList (True/False): Is the current event id within the include/exclude pane for this alert group.
  • IsWithinSchedule (True/False): Are we currently within a scheduled day/time for this alert group.
  • IsHandled (True/False): Set this to true if you don't want any other targets (SMTP/SNMP/SMS) to process this alert.

There are many potential uses for the custom alert target. If you wish to apply some complex filtering and/or routing logic, you can use the IsHandled property to mark alerts for inclusion/exclusion. You may also wish to generate a log file of a specific format for your external log ingestion software, or perhaps would like to send alerts to an edge-logging service. As this is a standard custom-code device, you can include standard and web references, resources etc., making it easy to connect to and distribute alerts to other platforms. For example, to connect to Nagios, you could implement the following:

 

The Scheduling tab let's you specify when your alert group will receive events. It has three options:

  • No Scheduling: Your alert group will receive all events regardless of day or time.
  • Days & Hours: Choose when your alert group will receive events based on the day and hour (minimum resolution 1 hour)
  • CRON: Specify a CRON string (which must contain an internal marker, ie ' - ')

The most common usage would be 'Days & Hours'. You can toggle a day/hour by clicking with the mouse. Dragging the mouse let's you select multiple adjacent blocks. 'Office hours' and 'After hours' would appear as follows:

The final tab, 'Tabs & Channels' lets you specify which group(s)/tab(s)/channel(s) will be included. By default, the "All" option in each header is checked, including all items. To start filtering, uncheck the "All" checkbox and check the items you wish to include: