Predicting delayed flights with classification analysis

To predict whether a specific flight is delayed:

Now you have a new index that contains a copy of your source data with predictions for your dependent variable.

When you view the classification results in Kibana, it shows contents of the destination index in a tabular format:

In this example, the table shows a column for the dependent variable (FlightDelay), which contains the ground truth values that we are trying to predict with the classification analysis. It also shows a column for the prediction values (ml.FlightDelay_prediction) and a column that indicates whether the document was used in the training set (ml.is_training). You can filter the table to show only testing or training data and you can change which fields are shown in the table.

If you examine this destination index more closely in the Discover app in Kibana or use the standard Elasticsearch search command, you can see that the analysis predicts the probability of all possible classes for the dependent variable (in a top_classes object). In this case, there are two classes: true and false. The most probable class is the prediction, which is what’s shown in the classification results table. If you want to understand how sure the model is about the prediction, however, you might want to examine the class probability values. A higher number means that the model is more confident.

GET df-flight-delayed/_search

          ...
          "FlightDelay" : false,
          ...
          "ml" : {
            "top_classes" : [ 
              {
                "class_probability" : 0.939335365058496, 
                "class_score" : 0.6757432490367542, 
                "class_name" : "false"
              },
              {
                "class_probability" : 0.06066463494150393,
                "class_score" : 0.06835090015710144,
                "class_name" : "true"
              }
            ],
            "FlightDelay_prediction" : "false",
            "is_training" : false
          }

Though you can look at individual results and compare the predicted value (ml.FlightDelay_prediction) to the actual value (FlightDelay), you typically need to evaluate the success of your classification model as a whole.

Kibana provides a normalized confusion matrix that contains the percentage of occurrences where the analysis classified data points correctly with their actual class and the percentage of occurrences where it misclassified them.

If you want to see the exact number of occurrences, select a quadrant in the matrix. In this example, we’ve filtered the table to contain only testing data so we can see how well the model performs on previously unseen data. There are 2945 documents in the testing data that have the true class. 847 of them are predicted as false; this is called a false negative. 2098 are predicted correctly as true; this is called a true positive. The confusion matrix therefore shows us that 71% of the actual true values were correctly predicted and 29% were incorrectly predicted in the test data set.

Likewise if you select other quadrants in the matrix, it shows you that there are 8775 documents that have the false class as their actual value in the testing data set. The model labeled 7093 documents (out of 8775) correctly as false; this is called a true negative. 1682 documents are predicted incorrectly as true; this is called a false positive. Thus 81% of the actual false values were correctly predicted and 19% were incorrectly predicted in the test data set.

For more information about interpreting the evaluation metrics, see Classification evaluation.

You can also generate these metrics with the data frame analytics evaluate API.

POST _ml/data_frame/_evaluate
{
 "index": "df-flight-delayed",
   "query": {
    "term": {
      "ml.is_training": {
        "value": true  
      }
    }
  },
 "evaluation": {
   "classification": {
     "actual_field": "FlightDelay",
     "predicted_field": "ml.FlightDelay_prediction",
     "metrics": {
       "multiclass_confusion_matrix" : {}
     }
   }
 }
}

POST _ml/data_frame/_evaluate
{
 "index": "df-flight-delayed",
   "query": {
    "term": {
      "ml.is_training": {
        "value": false  
      }
    }
  },
 "evaluation": {
   "classification": {
     "actual_field": "FlightDelay",
     "predicted_field": "ml.FlightDelay_prediction",
     "metrics": {
       "multiclass_confusion_matrix" : {}
     }
   }
 }
}

{
  "classification" : {
    "multiclass_confusion_matrix" : {
      "confusion_matrix" : [
        {
          "actual_class" : "false", 
          "actual_class_doc_count" : 8775, 
          "predicted_classes" : [
            {
              "predicted_class" : "false", 
              "count" : 7093 
            },
            {
              "predicted_class" : "true",
              "count" : 1682
            }
          ],
          "other_predicted_class_doc_count" : 0
        },
        {
          "actual_class" : "true",
          "actual_class_doc_count" : 2945,
          "predicted_classes" : [
            {
              "predicted_class" : "false",
              "count" : 847
            },
            {
              "predicted_class" : "true",
              "count" : 2098
            }
          ],
          "other_predicted_class_doc_count" : 0
        }
      ],
      "other_actual_class_count" : 0
    }
  }
}

If you don’t want to keep the data frame analytics job, you can delete it by using the delete data frame analytics job API. When you delete data frame analytics jobs, the destination indices remain intact.