User Guide - Statseeker Version 3.x

Alerting

Purpose of this Guide

Statseeker can be configured and used as a comprehensive proactive and reactive alerting solution. Alerts can trigger messages for specific events or can be configured to integrate with Service Desk applications, Event Correlation systems or other Network Management platforms.

This Guide provides:

  • An overview of the Alerting concepts;
  • Examples of how to configure Alerting.

Index

Set up your SMTP Gateway

Your SMTP gateway must be configured before Statseeker can send email alerts.

  1. Telnet or ssh to the Statseeker server;

  2. At the login prompt, enter the username: statseeker

  3. Enter your password (Either your root password or Admin password);

  4. Enter the command: ssadmin

  5. Enter the Statseeker server root password;

  6. You should now see the following menu:

  7. Choose option 2. Email and configure the SMTP gateway.

Statseeker Alerting Concepts

Statseeker’s Alerting is based on a simple concept of Actions & Filters.

  1. Actions

    2. Actions are created to:

    • Send an email;
    • Forward a Syslog message;
    • Send an SMS;
    • Send a pager message;
    • Send an alert to an external application (Network Management System etc).

    Please note that depending on the SMS/Paging/NMS system, Users may need to write their own custom action scripts.

  2. Filters

    3. Filters configure the Events that trigger Actions. Basically, Filters configure the alerts Users will receive from any of the 5 Event Databases.

  3. Event Databases

    4. Before Actions and Filters are created, it is important to understand what Events can trigger Alerts.

    In Version 3.5 there are 5 Event Databases:

    1. Syslog

      This database is only populated if network devices have been configured to send Syslog messages to Statseeker;

    2. SNMP Traps

      This database is only populated if network devices have been configured to send SNMP Trap messages to Statseeker;

    3. Device Events

      This database is populated by any device event that Statseeker polls for, such as pingstate changes, temperature state changes on network devices etc…

    4. Interface Events

      This database is populated by any interface event that Statseeker polls for, such as OperStatus changes;

    5. Thresholds

      This database is populated every time a Statseeker threshold is exceeded.

What Events Can Statseeker Alert on?

Statseeker can trigger an alert for any event from any of the 5 Statseeker Event Databases.

To view a sample of the events that Statseeker can alert on:

  1. Log into the Statseeker Network Infrastructure Monitor;
  2. Select a Time Filter;
  3. Run any of the Event or Threshold reports within the Report List.

Out of the box Actions

Before configuring Filters, an appropriate Action for the event must be configured. An Action is simply a command that executes a User created script written in shell, C, PERL, etc... These scripts can be as simple as piping a NIM Event, SNMP Trap or Syslog message to an email or as complex as raising a trouble ticket.

Statseeker provides a number of Action scripts out of the box:

These include:

  • nim-alert-email-generic - used to send email for a generic NIM Event such as device PING state change;
  • nim-alert-email-ifOperStatus - used to send an email alert on Operstatus change of an interface;
  • nim-alert-syslog-generic - used to send a syslog message for a generic NIM event;
  • nim-alert-syslog-ifOperStatus - used to send a syslog message for an Operstatus change on an interface.

In addition to the above out of the box supported scripts, we have created an email script with the following additional options:

  • The ability to customize the subject line of the message;
  • The ability to customize the message body of the email;
  • Forward the message to multiple groups, multiple Statseeker Users, and multiple email addresses;
  • Set a wait period so that devices must be unreachable for x minutes before an alert email is sent;
  • Email a daily report on the number of devices that have gone down overnight and stayed down;
  • Use a roster system so that only on call staff members are notified.

Contact our Customer Service Team at csm@statseeker.com for information about this script.

Alert Configuration Examples

  1. Example - Creating an Alert for When Devices go Up or Down.

    1. Go to the Administration Tool and select ‘Actions / Filters’ under the Device Events section on the left side menu bar;

      Note: The 5 Event Databases are listed as “black headings” on the left side menu bar.

    2. Select ‘Actions / Filters’ under Device Events.

      Note: This will list all previously created actions.

    3. To Configure an Action:
      1. Select ‘Add’, which will launch the ‘Device Events Action Configuration’ options;
      2. Enter a Name for the Action.

    4. Enter a command. For a simple device state change email alert the following command could be used:

      5. Where: 

      -b    Is an integer describing the number of minutes to bundle the alerts
      before sending the email. In the above example, all messages received
      within a 2 minute window will be bundled together into one email.

-e    Sends the alert to the specified email address.

Other options available: 

-g    Sends the alert to any Users associated to the specified Statseeker group. 

-s    Sends the alert with a custom subject line. The subject should be
      enclosed in single quote (') or double quote (") characters. This option
      will not work in conjunction with the -b option. 

-u    Sends the alert to the email address associated with the specified User.

    5. Once the Action has been saved Users will then need to configure a Filter to inform Statseeker what events they wish this action to be applied to.

      To do this:

      1. Select the Go To Filters button;
      2. Select Add;
      3. Give the filter a name.

    6. Then enter the regular expression that will match the events that are to be alerted on.

      For example, when a device becomes unreachable via a ping from the Statseeker server, a ping_state down event is created. When it becomes reachable again, a ping_state up event is created.

      To alert only on devices becomming unreachable type "ping_state down" into the regex field. To alert only on devices becomming reachable again type "ping_state up" into the regex field.

      To be alerted on devices going up OR down, then the ‘Regex’ would read:

    7. The status of the filter is ON by default but can be turned OFF.

      Turning this filter to OFF will stop these Alerts. For example if there is a scheduled outage and Users do not want to be alerted during this time.

      The next step is to choose a previously created Action. Appropriate Actions should be chosen for the devices, geographical locations or business units this Alert will apply to. For example Users could apply this filter to a Group of Servers and choose an Action that only alerts 'Server Administrator's'.

      The example below uses a previously created "Email Alert".

    8. Use the Group selection to apply this filter to all devices in a specific Group, or the use the Entity selection to apply this filter to one device.

      In this example we are leaving these filters in their default state of applying to 100% of the network.

      Complete the Time Filter section shown below to apply this filter only during certain hours. For example Monday to Friday from 8:00 am to 6:00 pm.

      If no options are selected then the filter is applied 24x7 as shown below.

      When Save Filter is selected, an email will be sent every time a device on the network changes state.

  2. Example - Creating an Alert for WAN Interfaces Changing Their Operational State.

    1. Statseeker strongly recommends Users limit ifOperStatus polling and alerting to critical interfaces to reduce the number of potential events.

      Statseeker suggests that Users create Groups of these interfaces for reporting and alerting. Before configuring Actions and Filters for ifOperStatus alerting, Users need to check to see if Statseeker is polling the relevant interfaces. In the below example we are checking the contents of the '3: Primary WAN Links' Group for ifOperStatus polling.

      To do this:

      1. Go to the ‘Network Infrastructure Monitor – Advanced Console;
      2. Select the Primary WAN Links from the Group Filter and then run the ‘Interfaces – Details’ report.

        The ‘Oper’ column defines if the interfaces are set ‘poll’ or ‘nopoll’ for the ifOperStatus OID.

    2. To turn polling on for any interface that is currently set to ‘nopoll’, click on the relevant cell inm the report and a new window will appear to inform Users that the config change has been successful. The report will refresh and display ‘poll’.

      This config change can be automated by using one of Statseeker's Custom Scripts. Please email csm@statseeker.com for further information.

    3. Once ifOperStatus polling has been configured for the interfaces to be alerted on:
      1. Go To the Administration Tool;
      2. Select the ‘Actions / Filters’ option under the appropriate event database heading, in this case under Interface Events.

        This will list any previously created actions.

    4. To configure an Action:
      1. Select ‘Add’, which will launch the ‘Interface Events Action Configuration’;
      2. Enter a Name for the Action.

    5. Then enter a command.

      Statseeker has provided an out of the box command for ifOperStatus changes: 

      6. Where:

-b    Is an integer describing the number of minutes to bundle the alerts before sending
      the email. 

-e    Sends the alert to the specified email address.

Other options available: 

-g    Sends the alert to any Users associated to the specified Statseeker group. 

-s    Sends the alert with a custom subject line. The subject should be enclosed in
      single quote (') or double quote (") characters. This option will not work in
      conjunction with the -b option. 

-u    Sends the alert to the email address associated with the specified Statseeker User.
    6. Once the Action has been saved, then a Filter needs to be configured to inform Statseeker what events this Action will be applied to.

      To do this:

      1. Select the Go To Filters button;
      2. Select Add;
      3. Give the filter a name.

    7. Then enter the regular expression that will match the events to be alerted on.

      In this example, we are looking for OperStatus changes which are displayed in the below format: 

      IF-MIB.ifOperStatus.5 down
OR
IF-MIB.ifOperStatus.5 up

      Where 5 is the index number of the interface.

      The Regex entry would be as shown below.

      8. The status of the filter is ON by default but can be turned OFF. Turning this filter to OFF will stop these Alerts, for example if there is a scheduled outage and Users do not want to be alerted during this time.

      The next step is to choose a previously created Action. Appropriate Actions should be chosen for the interfaces, geographical locations or business units this Alert will apply to. For example Users could apply this filter to a Group of Camera interfaces and choose an Action that only aletrs the 'Security Team'.

      For the example below we use the previously created 'OperStatus Email Alert' Action.

    8. Use the Group selection to apply this filter to all interfaces in a specific Group, or the use the Entity selection to apply this filter to a specific device. The default is to apply the filter to 100% of the network.

      In this example, the 3: Primary WAN Links group will be used.

    9. Now specify when to apply this filter.

      For example Monday to Friday from 8:00 am to 6:00 pm.

      If no options are selected the filter is applied 24x7 as shown below.

      When Save Filter is selected, an email will be sent every time one of the Primary WAN Links changes state.

  3. Example - Creating an alert for Broadcast Storms Occurring During Business Hours.

    1. Go to the Administration Tool.

      Select ‘Actions / Filters’ under the Syslog heading.

      This will list all previously created actions.

    2. To configure an Action:
      1. Select ‘Add’, which will launch the ‘Syslog Action Configuration’ window;
      2. Enter a Name for the Action.

    3. Enter a command.

      To forward a syslog message to another host, the following command could be used where: 

      4. -h     is the hostname of the NMS that will receive the message.

Other options available: 

-s     Sends the alert with a custom subject line. The subject should be enclosed
       in single quote (') or double quote (") characters.

    4. Once the Action has been saved, configure a filter to inform Statseeker what events this action applies to.

      To do this:

      1. Select the Go To Filters button;
      2. Select Add;
      3. Give the filter a name.

    5. Then enter the regular expression that will match the events to be alerted on. When alerting on syslog messages, any text from the syslog message can be entered into the regular expression field of the filter.

      For example, for the message: 

      6. STORM_CONTROL-3-FILTERED: A Broadcast storm detected on Gi1/0/28. A packet filter action
has been applied on the interface.

      …the following text could go into the Regex field:

    6. The status of the filter is ON by default but can be turned OFF. Turning this filter to OFF will stop these Alerts, for example if there is a scheduled outage and Users do not want to be alerted during this time.

      The next step is to choose a previously created Action. Appropriate Actions should be chosen for the devices, geographical locations or business units this Alert will apply to. For example Users could apply this filter to a Group of Routers and choose an Action that only alerts 'WAN Administrator's'.

      For the example below we will use the previously created 'Forward Syslog to NMS' Action.

    7. Use the Group selection to apply this filter to all devices in a specific Group, or the use the Entity selection to apply this filter to one device.

      In this example we are leaving these filters in their default state of applying to 100% of the network.

      Complete the Time Filter section shown below to apply this filter only during certain hours, for example Monday to Friday from 8:00 am to 6:00 pm.

      If no options are selected then the filter is applied as 24x7.

      When Save Filter is selected, a syslog message will be forwarded anytime a Storm Control message is received by the Statseeker server during business hours.

© 1998-2012 Statseeker Pty Ltd. All rights reserved.