Create a metrics threshold alert

edit

Create a metrics threshold alert

edit

Based on the metrics that are listed on the Metrics Explorer page within the Metrics app, you can create a threshold alert to notify you when a metric has reached or exceeded a value for a specific time period.

Additionally, each alert can be defined using multiple conditions that combine metrics and thresholds to create precise notifications.

  1. To access this page, go to Observability > Metrics, and then click Metrics Explorer.
  2. On the Metrics Explorer page, click Alerts > Create alert.

When you select Alerts > Create alert, the alert will automatically be populated with the same parameters you’ve configured on the Metrics Explorer page. If you’ve chosen a graph per value, your alert will be pre-configured to separately monitor and notify about each individual graph displayed on the page.

You can also create an alert based on a single graph on the Metrics Explorer page. Click Actions and then select Create alert. The condition and filter sections of the threshold alert are automatically populated.

Metric conditions

edit

Conditions for each alert can be applied to specific metrics that you select. You can select the aggregation type, the metric, and specify the threshold value to be alerted on. To help you determine which thresholds are meaningful to you, the preview charts provide a visualization.

When you select Alert me if there’s no data, the alert is triggered if the metrics don’t report any data over the expected time period, or if the alert fails to query Elasticsearch.

The Filters control the scope of the alert, and the Create alert per creates an instance of the alert for every unique value of the field added. For example, create an alert per host, or per every mount point of each host. You can also add multiple fields.

Inventory alert

Before creating an alert, you can preview whether the alert would have been triggered in the last hour, day, week, or month.

Preview metric alert

Action types

edit

You can extend your alerts by connecting them to actions that use the following supported built-in integrations.

Action types

Action variables

edit

This section details the variables that metrics threshold alerts will send to your actions.

Basic variables
edit
The default metrics threshold alert message detailing basic variables

The default message for a metrics threshold alert displays the basic variables you can use.

  • context.group: This variable resolves to the group that the alert’s condition detected. For Inventory alerts, this is the name of a monitored host, pod, container, etc. For metric threshold alerts, this is the value of the field specified in the Create alert per field or * if the alert is configured to aggregate your entire infrastructure.
  • context.alertState: Depending on why the action is triggered, this variable resolves to ALERT, NO DATA, or ERROR. ALERT means the alert’s condition is detected, NO DATA means that no data was returned for the time period that the alert queried, and ERROR indicates an error when querying the data.
  • context.reason: This variable describes why the alert is in its current state. For each of the alert’s conditions, it includes the detected value of the monitored metric, and a description of the threshold.
  • context.timestamp: This variable resolves to the timestamp of when the alert was evaluated.
Advanced variables
edit
The default metrics threshold alert message detailing advanced variables

Instead of using context.reason to provide all the information you need, there may be cases when you’d like to customize your action message. Metrics threshold alerts provide advanced context variables that have a tree structure.

These variables must use the structure of {{context.[Variable Name].condition[Number]}}. For example, {{context.value.condition0}}, {{context.value.condition1}}, and so on. This is required even if your alert has only one condition (accessible with .condition0). Using just {{context.[Variable Name]}} evaluates to a blank line or [object Object] depending on the action type.

  • context.value.condition[X]: This variable resolves to the detected value of conditions 0, 1, 2, and so on.
  • context.value.threshold[X]: This variable resolves to the threshold values of conditions 0, 1, 2, and so on.
  • context.value.metric[X]: This variable resolves to the monitored metric of conditions 0, 1, 2, and so on.

Settings

edit

With metrics threshold alerts, it’s not possible to set an explicit index pattern as part of the configuration. The index pattern is instead inferred from Metrics indices on the Settings page of the Metrics app.

With each execution of the alert check, the Metrics indices setting is checked, but it is not stored when the alert is created.

The Timestamp field that is set under Settings determines which field is used for timestamps in queries.