Grafana. Configuring notifications

You can configure notifications to users about the status of virtual machines (VMs) and cluster nodes. Grafana allows you to send notifications by email and/or to various messengers (e.g. Slack or Telegram).

Notification channels

Telegram

To configure the notifications channel in Telegram:

1. Create a Telegram bot according to the instructions in the Sending notifications to Telegram article. Save the token value for the bot.
2. Enter Alerting → Notification channels.
   
   
3. Press Add channel (New Channel).
4. Specify the notification Name. For example, Telegram notify.
5. In the Type field, select the notification type — Telegram.
6. Enable the option Default (send on all alerts) to use this method for all notifications.

7. Enable the option Include image to add the dashboard image to the notification.

   We do not recommend using this option, as creating an image places high load on the Grafana docker container.
8. Enable the option Disable Resolve Message so as not to receive a resolve message for an issue regarding which notifications have been sent.

9. Enable the option Send Reminders to receive reminders of the notification created and specify the frequency of sending in the Send reminder every parameter. For example, 15m — for sending the reminder every 15 minutes.
   

   Reminders cannot be sent more frequently than the notifications themselves.
10. Specify the Telegram settings:
    
    1. BOT API Token — Telegram bot token.

    2. Chat ID — Telegram chat id. 

       To get the chat id, add the bot @getmyid_bot to the chat.

       
       

11. Press Send Test to send a test message.
12. Press Save to save the changes.

Example of settings

Slack

To configure the notifications channel in Slack:

1. Enter Alerting → Notification channels.
   
2. Press Add channel (New Channel).
3. Specify the notification Name. For example, Slack notify.
4. In the Type field, select the notification type — Slack.
5. Enable the option Default (send on all alerts) to use this method for all notifications.

6. Enable the option Include image to add the dashboard image to the notification.

   We do not recommend using this option, as creating an image places high load on the Grafana docker container.
7. Enable the option Disable Resolve Message so as not to receive a resolve message for an issue regarding which notifications have been sent.

8. Enable the option Send Reminders to receive reminders of the notification created and specify the frequency of sending in the Send reminder every parameter. For example, 15m — for sending the reminder every 15 minutes.
   

   Reminders cannot be sent more frequently than the notifications themselves.
9. Specify the Slack settings:
   
   1. Url — URL address of incoming Slack webhooks. Read more in the official Slack documentation.
   2. Recipient — message recipient. Enter the channel name in the format #channel_name or user name in the format @username.
   3. Username — notification sender user name.
      
   4. Icon emoji — emoji for the notification sender icon. For example, :exclamation:.
      
   5. Icon URL — URL with the notification sender icon image.
      
   6. Mention Users — ids of the Slack users to be mentioned in the message. You can list the names separated by commas.
   7. Mention Groups — Slack groups to be mentioned in the message.
   8. Mention Channel — to which members of the channel to send a notification:
      • Disabled — don't send;
      • Every active channel member;
      • Every channel member.
   9. Token — bot token for using Slack file.upload API function. To use this feature, specify the recipient of the message in the Recepient field.
      
10. Press Send Test to send a test message.
11. Press Save to save the changes.

Example of settings

Notification parameters

To configure the notification sending rules:

1. Press the graph header → Edit.
   
   
2. Press  → Create alert.
   
3. Specify the notification parameters:
   
   1. Name — discretionary notification name.
   2. Evaluate every — notification interval. For example, 5m for notification every five minutes or 1h for notification every hour.
   3. For — the time that Grafana waits after the threshold value of the parameter has been reached. For example, 15m for 15 minutes waiting time.

   4. Conditions — conditions for notification:

      Condition format

      WHEN function OF query(id, start_time, end_time) condition value

      Comments to the format

      function — function for parameter calculation. Possible values:

      • avg() — average value;
      • min() — minimum value;
      • max() — maximum value;
      • sum() — sum of all values;
      • count() — number of values;
      • last() — last value;
      • median() — median value;
      • diff() — difference of values;
      • percent_diff() — percentage difference of values;
      • count_non_null() — number of values other than null;

      id — queue id. Default value for the first graph — A, for the second — B etc. You can view the id in the   (Queries) section;

      start_time — calculation start time. The start_time parameter is set relative to the current time. For example, 15m — 15 minutes before the current time;

      end_time — calculation end time. The end_time parameter is set relative to the current time in the now-<time> format. For example, now-5m — 5 minutes before the current time;

      condition — condition. Possible values:

      • IS ABOVE — higher;
      • IS BELOW — lower;
      • IS OUTSIDE RANGE — out of range;
      • IS WITHIN RANGE — within range;
      • HAS NO VALUE — no value is available;

      value — value or range.

      Example of condition

      WHEN avg() OF query(A, 15m, now) IS ABOVE 90

      Comments to the example

      The condition will be met if the average value of data from queue A has exceeded 90 in the last 15 minutes.

      To set an additional condition, press  . To add a condition through the logical "AND" specify the operand AND at the beginning of the condition. To add a condition through the logical "OR" specify the operand OR at the beginning of the condition. Additional conditions are checked sequentially in the order they are entered.

      Example of condition

      WHEN avg() OF query(A, 15m, now) IS ABOVE 90
      AND max() OF query(A, 15m, now) IS ABOVE 95

      Comments to the example

      The condition will be met if for the last 15 minutes, the average value of data from queue A has exceeded 90 and the maximum has exceeded 95.
      Read more about the condition format in the official Grafana documentation.
   5. If no data or all values are null — notification status if data are missing or have the null value. Possible values: Alerting, No Data, Keep Last State, OK;
   6. If execution error or timeout — сnotification status in case of an error during its execution. Possible values: Alerting, Keep Last State.
4. Select the notification channels in the Send to field.
5. Enter the notification text in the Message field.
6. Save the settings: on the upper right menu, click the icon → enter the settings comment → Save button.