All machine learning models tend to degrade over time. While DataRobot does monitor your deployment in real-time, you can always check on it to review the model health. To further assist you, DataRobot provides automated monitoring with a notification system. You can configure notifications to alert you when the service health, incoming prediction data, or model accuracy exceed your defined acceptable levels.
To configure the conditions for notifications, navigate to the Deployment Settings menu, and click Notifications.
You have three options for controlling notification delivery via email:
Receive all event notifications—includes critical, at risk, and scheduled delivery emails,
Receive only the critical event emails, or
Disable notifications for the deployment
Figure 1. Types of notifications settings
The Monitor tab is where you set exactly what values trigger the notifications. Users that have the role of “Owner” will be able to modify these settings; however, any user with whom the deployment has been shared can configure the level for the notifications that they want to receive, as shown on the Notifications tab. A user that isn’t an owner of the deployment can still view the same settings information.
Monitoring is available for Service Health, Data Drift, and Accuracy. The check box enables notification delivery at regularly scheduled intervals, ranging from minimally on the hour for service health, all the way to as long as once a quarter, which is available for all three performance monitors.
Figure 2. Monitoring settings for deployment notifications
For Data Drift, you are setting thresholds for feature Drift and Importance. Drift refers to a measure of how new incoming prediction data differs from the original data used to train the model. This is calculated using a metric that assesses how the distribution of data changes across all features, for a range defined with the Comparison period dropdown menu. The range compares training data to the selected period of recent prediction data.
Figure 3. Configuring data drift notification thresholds
The threshold you set refers to the amount of drift change you will allow for before a notification is triggered. The Importance threshold setting allows you to separate the important features you care most about from those that are less important. For both Drift and Importance, you can visualize the thresholds and how they separate the features on the Data Drift page. A low importance feature that exceeds the drift threshold is considered At-Risk, while for a high importance feature it is considered Failing.
Figure 4. Configuring importance threshold for data drift notifications
At the far right, you can also set specific rules to trigger At-Risk and Failing notifications as a function of the number of low or high importance features that cross the drift threshold.
Figure 5. Configure At-Risk and Failing notifications
For Accuracy, the notification conditions are related to a performance optimization metric for the underlying model in the deployment. The metric dropdown allows you to select from the same set of metrics that are available on the DataRobot model Leaderboard. Broadly, there is one set of metrics for classification and another for regression. Measurement defines the unit of measurement for the accuracy and its thresholds, Value will measure the metric by a specific value threshold, and Percent measures by percent changed. Enter the values for At-Risk and Failing at the far right.
Figure 6. Set metric for notifications about deployment accuracy
When you’re done configuring the notifications, simply click Save new settings to enable them.
If you’re a licensed DataRobot customer, search the in-app Platform Documentation for Settingstab, then locate the section "Using notifications" for more information.