This article showcases the Model Monitoring capabilities that come with DataRobot, which include service health, data drift, and accuracy monitoring.
Deploying a model as a REST API through DataRobot’s deploy method, means you have access to the wide range of monitoring capabilities available in the Deployments page.
Figure 1. Landing page of DataRobot
Figure 2. Deployments page
You’ll find a summary of all deployments that you have created (using DataRobot) or that have been shared with you. At the top of the page, you’ll see a summary for these deployments: how many there are, how many predictions have happened recently, and (most importantly) a synopsis of service health, data drift, and accuracy.
Below this information. You’ll find a list of all of the available deployments. To do a deep dive into one of them, you can just click on it.
When you select a model from the Deployments inventory page, DataRobot opens to that model’s Overview page. The overview provides a model-specific summary that describes the deployment, including the information you provided when creating the deployment and any model replacement activity.
You will see information similar to Figure 3.
Figure 3. Overview tab
The Service Health tab tracks metrics about a deployment’s ability to respond to prediction requests quickly and reliably. You can use this information to understand bottlenecks and assess capacity, which is critical to proper provisioning.
Figure 4. Service Health tab
When you deploy a model through DataRobot, training data is automatically uploaded with the model. The Data Drift dashboard leverages that data to help you to analyze a model’s performance after it has been deployed. The dashboard provides three interactive visualizations to identify the health of a deployed model over a specified time interval.
Figure 5. Data Drift tab
The Accuracy tab enables you to analyze the performance of model deployments over time, using standard statistical measures and visualizations. Use this tool to determine whether a model’s quality is decaying and if you should consider replacing it. The Accuracy tab renders insights based on the problem type and its associated metrics—metrics that vary depending on regression or binary classification projects.
Figure 6. Accuracy tab
Monitoring and Notification Settings
DataRobot provides a wide range of options to customize your deployments (as shown in Figures 7 and 8). For example, you can set a specific threshold for data drift based on your tolerance. When data drift is larger than that defined limit, DataRobot will flag the deployment’s data drift as either yellow or red, depending on the defined threshold.
Figure 7. Monitoring settings
Lastly, you can set the notifications you receive as well as the frequency for the notifications. Do you want to be warned only when something important happens, or do you want to receive an email notification every 2 hours even when your deployment is working as intended?
Figure 8. Notification settings
Sharing a Deployment
Every deployment in DataRobot can be shared with your colleagues. Figure 9 shows the procedure for sharing a deployment: simply insert the email of the person you want to share your project with.
Figure 9. Sharing a deployment
There are also different roles you can assign based on the governance protocols of your company. An individual can be a consumer, a user, or an owner of a deployment. (More information on the exact permissions for each role can be found in the in-app documentation.)
If you’re a licensed DataRobot customer, search the in-app documentation for Manage deployments or Optimization metrics. Also search Authentication, roles, and permissions, then locate more information for "Deployment roles and permissions."