Custom loss functions for accuracy tracking

[Tom B]

I have a prediction task where the average level of outcome is likely to vary significantly (and- for the sake of argument- unpredictably) in future, but the relativities between outcomes may not (and it's this that is of interest).    Is there any way to select an accuracy tracking metric - like Gini coefficient or RMSE adjusted for mean daily drift - that would be appropriate for this task? at the moment any accuracy degradation figures I get for my model are just dominated by the drift in the mean value.

[desmond_lim]

My instinct would be re-examine the objective of the modeling task if you already know the outcomes are likely to vary significantly. Perhaps a 2-stage or multi-stage modeling process would be appropriate for this?

[IraWatt]

Hey @Tom B, 

I think your question is quite similar to this one 'Custom Metrics for Monitoring'. The default metrics available through the UI are listed in the Accuracy tab. To create your customer metric what you would do is use the DataRobot API to calculate your metric and feed that back to the platform to trigger what ever action your looking for. 

Python API example:😞

import datarobot as dr

project = dr.Project.get('5cc899abc191a20104ff446a')
model = project.get_models()[0]

deployment = Deployment.get(deployment_id='5c939e08962d741e34f609f0')
deployment.model['id'], deployment.model['type']
>>> ('5c0a979859b00004ba52e431', 'Decision Tree Classifier (Gini)')

#Calculate custom metric