Triggering crossvalidation for a custom metric

[LittleNeutrino]

Hi

I have defined a custom metric (adjusted MAPE) and I have added it to each of my top 20 models like this:

for model in top20:
    
    # Compute adjusted MAPE for each partition
    validation_score = adjustedMAPE(
        y_true=model.__y_true[model.__partition_id == "0.0"],
        y_pred=model.__y_pred[model.__partition_id == "0.0"])
    
    holdout_score =adjustedMAPE(
        y_true=model.__y_true[model.__partition_id == "Holdout"],
        y_pred=model.__y_pred[model.__partition_id == "Holdout"])

    # Create new metrics entry 
    model.metrics["AdjustedMAPE"] = {
        "validation": validation_score,
        "crossValidation": None,
        "holdout": holdout_score,
        "training": None,
        "backtestingScores": None,
        "backtesting": None,
    }

Please observe that, as this custom metric is defined after the models have been built, there is no crossvalidation value available for it.

How can I trigger the computation of the crossvalidation score once I have defined a custom metric as in this case?

Thanks

[mel]

I'm not sure if you're using a different definition of adjusted MAPE but SMAPE is available from the leaderboard.

Otherwise, you can call the partitions used in cross-validation just like you've done for the validation set. Simply swap out partition_id == "0.0" for partition_id == "1.0" and so on to access the other partitions. You can compute the cross-validation score by averaging the validation score across all these partitions.