Can't run Prediction Explanation with Recommend model

[khoa.hv]

Hello,

Thank you for a great product!

I have one question about Datarobot platform.

Did Datarobot change how Prediction Explanations work?

Background: Today 2023/01/05, after created a regression model with Date/time partitioning using Autopilot mode, I try to run Prediction Explanation with the recommended model, by using this code:

# Compute predictions
predict_job = model.request_predictions(dataset.id)
predict_job.wait_for_completion()
# Initialize prediction explanations
pei_job = dr.PredictionExplanationsInitialization.create(project.id, model.id)
pei_job.wait_for_completion()
# Compute prediction explanations with default parameters
pe_job = dr.PredictionExplanations.create(project.id, model.id, dataset.id)
pe = pe_job.get_result_when_complete()
# Iterate through predictions with prediction explanations
for row in pe.get_rows():
    print(row.prediction)
    print(row.prediction_explanations)
# download to a CSV file
pe.download_to_csv('prediction_explanations.csv')

but this error occurred:

ClientError: 422 client error: {'message': 'Prediction explanations are not supported for models that use validation data for training.'}

The command cause error is:

pei_job = dr.PredictionExplanationsInitialization.create(project.id, model.id)

From the error message, I think the problem is validation data is included on recommend model, but this error has never happened before, and the last time is used without any problem is 2022/12/26. So I think there must be some change inside Datarobot.

I even worst, when I try to deploy that recommend model and run prediction with this Batch prediction code:

job = dr.BatchPredictionJob.score(
    deployment=DEPLOYMENT_ID,
    intake_settings={
        'type': 'localFile',
        'file': input_file,
    },
    output_settings={
        'type': 'localFile',
        'path': output_file,
    },
    max_explanations=3
)

The new error occurred and say that I can't do that because the prediction explanations is not initialized.

Is this intentional or a bug? And how can I fix it?

Some more information about my environment:

- Python client version: 2.24.0

- Here is my new model when I check deploy tab in GUI

[khoa.hv]

Hi, Bogdan,

Thanks for your fast reply!

I agreed with "don't have their metrics calculated" point too, but I don't think "don't have prediction explanation calculated" is standard behavior. Here are some logs from models I created previously, which all show that I successfully used Prediction Explanations on 100% dataset model. The newest one is in 2022/12/26

Here is a model created at 2022/10/03

Or a model created on 2021/12/01 and is re-run Prediction Explanations on 2022/05/13.

On the other hand, here is the log of today's model. As you can see, no Prediction Explanations after the normal prediction on 01-05-2023 07:31:53, which is the time the error occured.

Any ideas on how things changed like that?

[Bogdan Tsal-Tsalko]

Hi, Khoa!

I guess this is standard behavior of DataRobot, Time Series projects don't have their metrics calculated and prediction explanation calculated when they are retrained on 100% of dataset. You can use model with validation available instead to produce prediction explanations.

In case if you've experienced different behavior, may you show example of 100% data trained TS model with prediction explanations (screenshot for example from different project)?