New

The executive guide to generative AI

Read more
About usPartnersSupport|Login
IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Common Issues Test connectors »
Elastic Docs ›Kibana Guide [7.14] ›Alerting ›Troubleshooting

Event log index

edit

Event log index

edit

Use the event log index to determine:

  • Whether a rule successfully ran but its associated actions did not
  • Whether a rule was ever activated
  • Additional information about rule execution errors
  • Duration times for rule and action executions

Example Event Log Queries

edit

Event log query to look at all event related to a specific rule id:

GET /.kibana-event-log*/_search
{
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ],
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "event.provider": {
              "value": "alerting"
            }
          }
        },
        // optionally filter by specific action event
        {
          "term": {
            "event.action": "active-instance"
              | "execute-action"
              | "new-instance"
              | "recovered-instance"
              | "execute"
          }
        },
        // filter by specific rule id
        {
          "nested": {
            "path": "kibana.saved_objects",
            "query": {
              "bool": {
                "filter": [
                  {
                    "term": {
                      "kibana.saved_objects.id": {
                        "value": "b541b690-bfc4-11eb-bf08-05a30cefd1fc"
                      }
                    }
                  },
                  {
                    "term": {
                      "kibana.saved_objects.type": "alert"
                    }
                  }

                ]
              }
            }
          }
        }
      ]
    }
  }
}

Event log query to look at all events related to executing a rule or action. These events include duration.

GET /.kibana-event-log*/_search
{
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ],
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "event.action": {
              "value": "execute"
            }
          }
        },
        // optionally filter by specific rule or action id
        {
          "nested": {
            "path": "kibana.saved_objects",
            "query": {
              "bool": {
                "filter": [
                  {
                    "term": {
                      "kibana.saved_objects.id": {
                        "value": "b541b690-bfc4-11eb-bf08-05a30cefd1fc"
                      }
                    }
                  }
                ]
              }
            }
          }
        }
      ]
    }
  }
}

Event log query to look at the errors. You should see an error.message property in that event, with a message from the action executor that might provide more detail on why the action encountered an error:

{
  "event": {
    "provider": "actions",
    "action": "execute",
    "start": "2020-03-31T04:27:30.392Z",
    "end": "2020-03-31T04:27:30.393Z",
    "duration": 1000000
  },
  "kibana": {
    "namespace": "default",
    "saved_objects": [
      {
        "type": "action",
        "id": "7a6fd3c6-72b9-44a0-8767-0432b3c70910"
      }
    ],
  },
  "message": "action executed: .server-log:7a6fd3c6-72b9-44a0-8767-0432b3c70910: server-log",
  "@timestamp": "2020-03-31T04:27:30.393Z",
}

And see the errors for the rules you might provide the next search query:

{
  "event": {
    "provider": "alerting",
    "start": "2020-03-31T04:27:30.392Z",
    "end": "2020-03-31T04:27:30.393Z",
    "duration": 1000000
  },
  "kibana": {
    "namespace": "default",
    "saved_objects": [
      {
        "rel" : "primary",
        "type" : "alert",
      	  "id" : "30d856c0-b14b-11eb-9a7c-9df284da9f99"
      }
    ],
  },
  "message": "alert executed: .index-threshold:30d856c0-b14b-11eb-9a7c-9df284da9f99: 'test'",
  "error" : {
    "message" : "Saved object [action/ef0e2530-b14a-11eb-9a7c-9df284da9f99] not found"
  },
}

You can also query the event log for failures, which should return more specific details about rules which failed by targeting the event.outcome:

GET .kibana-event-log-*/_search
{
  "query": {
	"bool": {
  		"must": [
    		{ "match": { "event.outcome": "failure" }}
  	  ]
	  }
  }
}

Here’s an example of what failed credentials from Google SMTP might look like from the response:

"error" : {
  "message" : """error sending email: Invalid login: 535-5.7.8 Username and Password not accepted. Learn more at
535 5.7.8  https://support.google.com/mail/?p=BadCredentials e207sm3359731pfh.171 - gsmtp"""
},
« Common Issues Test connectors »

On this page

Was this helpful?
Feedback