Showing results for 
Search instead for 
Did you mean: 

Triggering crossvalidation for a custom metric

Triggering crossvalidation for a custom metric



I have defined a custom metric (adjusted MAPE) and I have added it to each of my top 20 models like this:




for model in top20:
    # Compute adjusted MAPE for each partition
    validation_score = adjustedMAPE(
        y_true=model.__y_true[model.__partition_id == "0.0"],
        y_pred=model.__y_pred[model.__partition_id == "0.0"])
    holdout_score =adjustedMAPE(
        y_true=model.__y_true[model.__partition_id == "Holdout"],
        y_pred=model.__y_pred[model.__partition_id == "Holdout"])

    # Create new metrics entry 
    model.metrics["AdjustedMAPE"] = {
        "validation": validation_score,
        "crossValidation": None,
        "holdout": holdout_score,
        "training": None,
        "backtestingScores": None,
        "backtesting": None,



Please observe that, as this custom metric is defined after the models have been built, there is no crossvalidation value available for it.


How can I trigger the computation of the crossvalidation score once I have defined a custom metric as in this case?



Labels (1)
0 Kudos
1 Reply
DataRobot Alumni

I'm not sure if you're using a different definition of adjusted MAPE but SMAPE is available from the leaderboard.


Otherwise, you can call the partitions used in cross-validation just like you've done for the validation set. Simply swap out partition_id == "0.0" for partition_id == "1.0" and so on to access the other partitions. You can compute the cross-validation score by averaging the validation score across all these partitions.

0 Kudos