Evaluate the performance of a churn model (part 3)

Learn how to estimate the risk of churn in a customer database by using the power of Alteryx (part 3)

In our previous article, we built a random forest model in order to predict the risk of churn among customers. We are now going to evaluate the performance of this model on an independent validation set to verify that our model can safely be applied on unknown data. This analysis can be handled in Alteryx Designer using the Model Comparison tool.

Prepare a validation set

In the previous article (July ’19), we defined a training set to build a random forest model by randomly selecting 50% of our entries with the Random % Sample tool.

We now use the remaining data as a validation set. These data can be isolated by connecting the Left (L) anchor of the Join tool with an Output Data tool. The File Format is set by default as “Alteryx Database (*.yxdb)”. We will keep this format in our case and specify a name for the validation set.

Save model

Simultaneously, we also save our model by connecting an Output Data tool on the Output (O) anchor of the Random Forest tool. We keep the same file format.

The plus of saving the model at this stage is that we will be able to apply it on new datasets as often as we wish, without spending time at retraining the model again and again. This is a valuable advantage if you need to industrialise your model.

Configure the Model Comparison tool

When our random forest model and our validation set are saved separately, we introduce them in a new Alteryx workflow thanks to two Input Data tools. The performance of the model is now analysed by connecting a Model Comparison tool, which is a tool of the Predictive Toolset.

The validation set and the random forest model will be connected to the Data (D) anchor and Model (M) anchor respectively. The tool will automatically detect that we tackle a classification problem, so no further configuration is required. The output can be visualised by connecting the Report (R) anchor to a Browse tool.

As the problem at hand is a classification, the report provided by the tool will display error measures and a set of diagnostic plots suited for this type of models. Should we deal with a regression model, the report would deliver similar performance indicators in regression frameworks. Additionally, the class labels predicted by the model on the validation set are available at the Prediction (P) anchor of the tool.

Validate model on independent data

The first part of the report provides fits and error measures indicating how well the model behaves on new data. Please note that this is different from the analysis in the previous article, which was performed on the training data and possibly overoptimistic from performance estimation point of view.

As we can see in the two last columns of the table above, accuracy on class 0 (loyal) and class 1 (churner) are around 90% and 49% respectively. This means that we can expect capturing about half of the churners with our model. Even if the fit is not perfect, it is a clear improvement compared to the initial ratio of churners in the validation set (26%). Therefore, actions directed towards churners predicted by the model would reach much more actual churners than actions performed on randomly selected customers. A confusion matrix is also presented to dig further in the details if necessary.

Performance Diagnostic Plots are also displayed in the report. For instance, the ROC curve indicates the proportion of churners captured by the model in line with the entries wrongly classified as churners. Our model allows us to target more than 60% of churners while targeting only 20% of loyal customers.

The ROC curve can also be put in line with the AUC value displayed in the table, which is measuring the area below this curve. An ideal model would exhibit a ROC curve passing through the three points(0,0), (0,1) and (1,1), which implies that the AUC would reach 100%. In contrast, a ROC curve following the dashed line in the figure would indicate a model with no discriminating power. The corresponding AUC would be 50%. In our case, the ROC curve is well separated from the dashed line and the AUC is above 80% which indicates a good ability to identify churners based on our random forest model.

Conclusion

Alteryx provides us with a Model Comparison tool which allows us to easily assess the behaviour of a given model on new data and estimate what to expect from an industrialised model. It is however crucial to perform this evaluation on an independent validation set in order to get rid of the bias induced by using the learning data.

During these three articles you have learned to build a predictive model in order to track churners in a customer base. Important steps were the standardisation of your data, building and training the model and finally validating it. In this process we only used the preconfigured tools of Alteryx Designer and obtained a rather good result. Now it is your turn to start developing your first predictive model, and don’t forget we are there to help you out!