Monitoring Performance with Accuracy and Data Drift
In the majority of cases, your models will degrade over time. The composition or type of data may change, or the way you collect and store it may change.
On the Accuracy page, we see the difference between the predictions made and the actual values (and in this case shown here, we can see in the image below that the model is fairly consistently under-predicting the actuals). Most likely, the degraded predictions are a result of a change in the composition of data.
Figure 14. Data Drift
The Data Drift page shows you how the prediction data changes over time from the data you originally used to train a model. In the plot on the left, each green, yellow or red dot represents a feature. The degree of feature importance is on the X-axis, and a calculation of the severity of data drift on the Y-axis. In the plot on the right, we see the range of values of each selected feature, with the original training data in dark blue and more recent prediction data in light blue. Looking at a few examples, we can see how the composition of the data has changed.
Figure 15. Accuracy
So inevitably you’ll want to retrain your model on the latest data and replace the model currently in the deployment with the new model. DataRobot MLOps makes this easy by providing a simple interface to swap out your model, all the while maintaining the lineage of models and all collected drift and accuracy data. And this occurs seamlessly, without any service disruption.
If you’re a licensed DataRobot customer, search the in-app Platform Documentation for Data Drift tab and Accuracytab.