Visual AI with DataRobot Python API Getting Started

Showing results for 
Search instead for 
Did you mean: 

Visual AI with DataRobot Python API Getting Started

DataRobot Visual AI enables you to include images in your supervised machine learning pipeline when solving classification and regression use cases. The DataRobot Python Client API lets you use Visual AI programmatically, making it easier to reproduce experiments and set up a series of different tasks. (You can find documentation for the latest DataRobot Python package here.)

In this article we will use the popular Oxford-IIIT Pet dataset to build a classification model for different dog and cat breeds: 37 categories in total! Although this sounds like a challenging task, using DataRobot Visual AI (and our Jupyter notebook), we can build efficient and accurate models.

The full notebook can be found here in the Community GitHub.

Let's get started!

Before you can use the DataRobot Python client, you need to get your API key and determine your DataRobot API endpoint (which is dependent on your DataRobot installation).

In your notebook, declare all the libraries you need for your DataRobot Python Client and any additional utility modules that will help you interpret the results.

import pandas as pd
import datarobot as dr
import numpy as np
import requests
import io
import PIL.Image

import matplotlib.pyplot as plt
from matplotlib.offsetbox import OffsetImage, AnnotationBbox
from matplotlib import pyplot as plt
from matplotlib.image import BboxImage
from matplotlib.transforms import Bbox, TransformedBbox
import seaborn as sns

from IPython.core.display import display, HTML

from datarobot import AUTOPILOT_MODE
from datarobot.models import Project, Dataset
from datarobot.models.visualai import SampleImage, ImageActivationMap, ImageEmbedding
from graphviz import Source

Once you’ve successfully loaded the library toolkit, you are ready to define the project settings in another cell.

We’ve already prepared the data from into an S3 data bucket: DataRobot S3 Link


dr.Client(token=API_TOKEN, endpoint=ENDPOINT)


PROJECT_NAME = "oxford_pets_dataset_demo_notebook" # name of the project
IMAGE_COLUMN = "image" # image column
TARGET_CLASS = "class" # target class
PROJECT_METRIC = "Accuracy" # project metric
SAMPLE_SIZE = 5 # number of sample size for image sampling and activation maps
EMBEDDINGS_SAMPLE = 50 # number of image samples for Image Embeddings

The next logical step is a few lines of code to kick off Autopilot for Visual AI. Using project.wait_for_autopilot lets you determine when Autopilot is finished; here, we’re checking every 5 minutes (check_interval=300).

project = dr.Project.start(
        S3_DATA_PATH, #Pandas Dataframe with data. 
                      #Could also pass the folder link itself
        project_name = PROJECT_NAME, #Name of the project
        target = TARGET_CLASS, #Target of the project
        worker_count = -1, #Number of workers to use. 
                           #-1 means all available workers
        autopilot_on = True #Run on Autopilot (Default value)

project.wait_for_autopilot(check_interval=300) #Wait for Autopilot to complete, check every 5 minutes
project_id =

So far, so good! In just a few minutes and with few clicks, you have generated image classification results. Now, let’s interpret the results.

def get_model_score(mod, metric):
    res = {}
    res['model_number'] = mod.model_number
    res['model_type'] = mod.model_type
    res['sample_pct'] = mod.sample_pct
    res['metric_v'] = mod.metrics.get(metric, {}).get('validation')
    res['metric_cv'] = mod.metrics.get(metric, {}).get('crossValidation')
    return res

def get_model_scores(proj, metric=None, ascending=True):
    iterate trough the project models and get their performance metric
    if metric is None:
        metric = proj.metric        
    df = pd.DataFrame([get_model_score(m, metric) for m in proj.get_models(with_metric=metric)])
    return df.sort_values(['metric_cv', 'metric_v'], ascending=ascending, na_position='last')

def get_model_number_to_network_mapping(models_endpoint, api_token):
    # retrieve pretrained Neural Network model for each model_number on the Leaderboard
    response = requests.get(models_endpoint, headers={'Authorization': 'Token {}'.format(api_token)})
    if response.status_code != 200:
        msg = 'API returned a non-200 status code (is your API token valid for this project?)'
        raise RuntimeError(msg)
    df = pd.DataFrame(columns=['model_number','pretrained_network'])
    network_name_in_bp = "No pretrained model"
    result = response.json()
    for idx in range(len(result)):
        model_number = result[idx]["modelNumber"]
        model_graph = result[idx]["processes"]
        network_name_in_bp_graph = network_name_in_bp
        for graph_node in model_graph:
            if "Pretrained" in graph_node:
                network_name_in_bp_graph = graph_node
        df = df.append({'model_number' : np.int64(model_number), 'pretrained_network' : network_name_in_bp_graph}, ignore_index=True)
    df["model_number"] = df["model_number"].astype(np.int64)
    return df

# get model_id -> neural network name mapping
models_leaderboard_endpoint = "{}/projects/{}/models".format(ENDPOINT,project_id)
blueprint_to_pretrained_network = get_model_number_to_network_mapping(models_leaderboard_endpoint, API_TOKEN)
model_scores = get_model_scores(project, metric=PROJECT_METRIC, ascending=False)
model_scores = model_scores.merge(blueprint_to_pretrained_network, on=['model_number'])

You can use the output from the helper function(s) to check the total number of trained models and identify the most accurate model.


Whoa! With only a few clicks, DataRobot Visual AI achieved 93% cross-validation accuracy!

If you add just a few more lines of code, you can create the DAG (directed acyclic graph) of the best Visual AI model from the Leaderboard.

best_model_on_lb = project.get_models()[0] # top model from the Leaderboard
filtered_chart = best_model_on_lb.get_model_blueprint_chart()

Displaying a detailed confusion chart shows information for each class (i.e., named directory containing images), including f1 score, precision score, recall score, confusion matrix, etc.

best_validation_model = project.get_models(order_by = '-metric')[1]

confusion_chart = best_validation_model.get_confusion_chart(source='validation')

And, you can visualize the confusion matrix for the best model from the Leaderboard.

chart = best_validation_model.get_confusion_chart(dr.enums.CHART_DATA_SOURCE.VALIDATION)


Accessing image insights for Visual AI data is straightforward in the DataRobot GUI; likewise, from your API client you can access and interpret them in your notebook.

datarobot_insights = "{}/projects/{}/insights".format(DATAROBOT_URL, project_id)
display(HTML("""<a href="{}">DataRobot Insights Project Page</a>""".format(datarobot_insights)))

Let's take a peek at some of the images in our image dataset and their respective labels.

for sample in SampleImage.list(, IMAGE_COLUMN)[:SAMPLE_SIZE]:
    print("target value = {}".format(sample.target_value))
    bio = io.BytesIO(sample.image.image_bytes)
    img =
atanas_atanasov_2-1606778428135.png atanas_atanasov_3-1606778427940.png

We can easily get the activation maps.

for model_id, feature_name in ImageActivationMap.models(
    for amap in ImageActivationMap.list(, model_id, IMAGE_COLUMN)[:SAMPLE_SIZE]:
        bio = io.BytesIO(amap.overlay_image.image_bytes)
        img =

Visual AI provides unsupervised image embeddings for clustering your data in an unsupervised manner. It allows you to visualize a sample of images projected from their original N-dimensional feature space to a new two-dimensional feature space. This makes it easy to see which images are considered similar.

# helper function for visualizing image embeddings, by mapping coordinates to a point on the image

def imscatter(x, y, image, ax=None, zoom=1):
    if ax is None:
        ax = plt.gca()
        bio = io.BytesIO(image.image.image_bytes)
        image = plt.imread(bio, format='JPG')
    except TypeError:
    im = OffsetImage(image, zoom=zoom)
    x, y = np.atleast_1d(x, y)
    artists = []
    for x0, y0 in zip(x, y):
        ab = AnnotationBbox(im, (x0, y0), xycoords='data', frameon=False)
    ax.update_datalim(np.column_stack([x, y]))
    return artists

# style/visualization settings
fig, ax = plt.subplots()
ax.scatter(1000, 1000)
scaling_factor = 1000.0

project = Project.list(search_params={"project_name": PROJECT_NAME})[0]

for model_id, feature_name in ImageEmbedding.models( 
    for embedding in ImageEmbedding.list(, model_id, IMAGE_COLUMN)[:EMBEDDINGS_SAMPLE]:
                  zoom=0.3, # control image size on the grid

DataRobot UI provides powerful visualization tools for viewing and interpreting your Visual AI projects. But if you want to create and access Visual AI projects and models automatically and in a repeatable way, you need an API. As shown here, the DataRobot API makes it easy to build and interpret Visual AI models.

Want to learn more?

The community has some more resources for learning about Visual AI:

Visual AI: Quick Overview

Visual AI: Classify Bell Pepper Leaves

Visual AI (learning session)

If you’re a licensed DataRobot customer, search the in-app Platform Documentation for Visual AI.

What do you think?

You can reply to this article with your feedback or PM me (@atanas_atanasov). 

Labels (3)
Version history
Last update:
‎01-25-2021 03:32 PM
Updated by: