Machine learning

As data sets increase in size and complexity, the human effort required to inspect dashboards or maintain rules for spotting infrastructure problems, cyber attacks, or business issues becomes impractical. Elastic machine learning features such as anomaly detection and outlier detection make it easier to notice suspicious activities with minimal human interference.

Kibana includes a free Data Visualizer to learn more about your data. In particular, if your data is stored in Elasticsearch and contains a time field, you can use the Data Visualizer to identify possible fields for anomaly detection:

You can also upload a CSV, NDJSON, or log file. The Data Visualizer identifies the file format and field mappings. You can then optionally import that data into an Elasticsearch index. To change the default file size limit, see fileUpload:maxFileSize advanced settings.

If Elastic Stack security features are enabled, users must have the necessary privileges to use machine learning features. Refer to Set up machine learning features.

You can find the data comparison view in Machine Learning > Data Visualizer in Kibana. The data comparison view shows you the differences in each field for two different time ranges in a given data view. The view helps you to visualize the changes in your data over time and enables you to understand its behavior better.

Select a data view that you want to analyze, then select a time range for the reference and the comparison data in the appearing histogram chart. You can adjust the time range for both the reference and the comparison data by moving the respective brushes. When you finished setting the time ranges, click Run analysis.

You can decide whether you want to see all the fields in the data view or only the ones that contains drifted data. The analysis results table displays the fields, their types, if drift is detected, the p-value that indicates how significant the detected change is, the reference and comparison distribution, and the comparison chart. You can expand the results for a particular field by clicking the arrow icon at the beginning of the field’s row.