MLops Accuracy Alerts Calculation

[IraWatt]

I am using the Accuracy Alerts in MLOps to trigger model replacement however, I'm unsure quantitatively at what point this is triggered. The documentation describes the red alert as "Accuracy has severely declined since the model was deployed", what does this mean numerically?

Could I also use the API to set my thresholds to trigger model replacement?

Many thanks

[RonBuckley · Accepted solution]

Accuracy monitoring is defined by a single accuracy rule. Every 30 seconds, the rule evaluates a deployment's accuracy. Notifications trigger when this rule is violated. Deployment owners must define the metric, measurement, and threshold components of the accuracy rule.

The full description of how to configure the rules is at:

https://docs.datarobot.com/en/docs/mlops/governance/deploy-notifications.html#accuracy-notifications

The API allows you to replace a model and note that it was replace for accuracy reasons.

https://datarobot-public-api-client.readthedocs-hosted.com/en/v2.26.0/entities/deployment.html?highl...

import datarobot as dr
from datarobot.enums import MODEL_REPLACEMENT_REASON

project = dr.Project.get('5cc899abc191a20104ff446a')
model = project.get_models()[0]

deployment = Deployment.get(deployment_id='5c939e08962d741e34f609f0')
deployment.model['id'], deployment.model['type']
>>> ('5c0a979859b00004ba52e431', 'Decision Tree Classifier (Gini)')

deployment.replace_model('5c0a969859b00004ba52e41b', MODEL_REPLACEMENT_REASON.ACCURACY)
deployment.model['id'], deployment.model['type']
>>> ('5c0a969859b00004ba52e41b', 'Support Vector Classifier (Linear Kernel)')