This article provides an end-to-end walkthrough of using DataRobot Visual AI capabilities to incorporate images in Supervised Machine Learning projects.
Visual AI enables you to train new models on images alone or use images in combination with other data types such as numeric, categorical, date, location, and text (Figure 1). Models built using images have new visualization tools to help you evaluate them. You can deploy Visual AI models in a few clicks, then monitor and manage them just as you do with projects that don’t contain images.
Figure 1. Sample dataset with a mixture of data types
Despite the underlying architecture of Visual AI being a complex deep-learning architecture, when you use this platform in your projects you get to focus on your business problem. You don’t need any GPUs because Visual AI is performant on commodity CPU hardware. You also don’t need millions of labeled images to train a new model from scratch. DataRobot Visual AI uses pre-trained deep learning models to speed up this process. Most importantly, you don’t need to have any deep learning expertise to quickly create accurate models.
All you need to do is follow the same end-to-end workflow used for DataRobot projects that don’t have images (Figure 2).
Figure 2. Typical workflow for projects in DataRobot
In this demo we will use a collection of images of healthy and diseased bell pepper leaves (Figure 3). In Figure 4, the label inside each image represents the prediction target for this demo. We’ll use DataRobot to train a variety of deep learning models to classify these leaves by looking at the pixels in their respective images.
Figure 3. Sample images of bell pepper leaves
Figure 4. Sample images of diseased (left) and healthy (right) bell pepper leaves
The data needs to have at least two columns: the target column with the class for each image and the image column with the relative file path to each image (Figure 5). If this dataset had more features, they could have been added to this spreadsheet by attaching more columns to it.
Figure 5. A sample from the bell pepper leaves image dataset with two columns, one of which contains relative file path to the images
Before importing this data into DataRobot, make sure that it is packaged into a ZIP archive that contains all the images of a given class in a separate folder and a CSV file with at least two columns: the target column with the class for each image and the image column with the relative file path to the respective image (Figure 6).
Figure 6. A sample zipped folder with required content for importing into DataRobot for use with Visual AI
Configure modeling options in DataRobot relative to your project needs
This dataset is a standard binary classification project. After setting the target to be the class variable, we will use the default modeling options that DataRobot selected for this project and press Start to kick off the modeling process.
In order to convert images into numbers, Visual AI runs each image through a featurizer, which converts images into numeric vectors (Figure 7). These vectors can be combined with other data types such as categoricals, text, and numerics, and then fed into a model that is typically used with tabular data (Figure 8).
Figure 7. A featurizer that uses the SqueezeNet model for transfer learning
Figure 8. Feeding the results of the featurizer into a model that is typically used for tabular data
If you want to see the whole architecture used for the featurizer, you can do so in the Describe tab, under Neural Network Visualizer (Figure 9).
Figure 9. A segment of the neural network architecture from the Neural Network Visualizer
After DataRobot is finished building the models, select one to evaluate. Since this is a binary classification project, all the tools for evaluating binary classification projects are also available for this project. Additionally, since this project contains images, it will also have two additional tools: Image Embeddings (Figure 10) and Activation Maps (Figure 11).
Image Embeddings allow you to visualize a sample of the input images. In Figure 10, for example, the model is able to discriminate between the healthy and diseased leaves. The Image Embedding placed most of the healthy leaves to the left of the orange line, while most of the diseased leaves are to the right of that line.
Figure 10. Image Embeddings showing what images were clustered together due to similar image features
DataRobot uses a state-of-the-art, non-linear dimensionality reduction technique called Trimap to project the “featurized” images from their original N-dimensional feature space to a new 2-dimensional feature space that can be displayed as shown in Figure 10. This embedding is what allows you to see what “similar images” mean to the model. You can use this visualization to ascertain whether the model’s concept of “similar” matches human intuition. Is the model using logical parts of the images to classify them, or is it missing something obvious? Is there anything considered similar that is clearly not?
Activation Maps allow you to see areas in the image that the model is using to make decisions. This display highlights the regions that the model considers most important by giving them a lighter color, e.g., yellow in Figure 11 and white in Figure 12. Activation Maps are computed similarly to how DataRobot computes Feature Effect for numeric/categorical variables. DataRobot relies on permutation method and/or Shap techniques to capture how the prediction changes when data is modified.
Figure 11. Activation Maps showing areas of high and low activation on a purple-yellow scale
Figure 12. Activation Maps showing areas of high and low activation on a grayscale
In both Figure 11 and Figure 12, the model correctly ignores pixels in the background and concentrates on those covering the leaf areas.
At the same time, it seems to look at “healthy” areas of the leaves when there is a disease (Figure 13) ...
Figure 13. Activation maps of diseased bell pepper leaves showing the model paying attention to the healthy parts of some of these leaves while ignoring the diseased parts
… and not take a look at the whole leaf when there is no disease (Figure 14).
Figure 14. Activation maps of healthy bell pepper leaves showing the model ignoring considerable amount of areas along these leaves
Both issues render the performance of this model suboptimal.
Improving the model
DataRobot offers a number of options to experiment with model improvement:
Swap the current featurizer with another one. Each featurizer is a pre-trained deep learning model. Under the Advanced Tuning tab, you can swap the current featurizer with any of the following: resnet50, xception, squeezenet, efficientnet-b0, efficientnet-b4, preresnet10, and darknet (Figure 15). By default DataRobot runs squeezenet because it provides the best speed vs accuracy tradeoff. Although Resnet50 takes longer to run it can deliver significant increases in accuracy.
Figure 15. Available networks for converting images into numerics
Control the “eyes” of the model so it pays attention to the right image constructs/objects. Each featurizer shown in Figure 16 comprises sequential layers as depicted in Figure 17. Each layer aggregates information from previous layers. The initial layers capture low level patterns made of a small set of pixels: points, edges, and corners. The next layers capture shapes and textures. The final layers capture objects. DataRobot does not use the last layer with high-level features, but combines features out of the preceding layers (Figure 17).
Figure 16. A simplified version of the deep learning architecture used in the Visual AI platform
Figure 17. Advanced Tuning tab with options to output image features at different levels
Don’t use high level features if you feel Visual AI should focus on points, edges, corners to make its decisions. Likewise, don’t use low level features if you feel Visual AI should focus on detecting common objects. Note also that the more layers you enable, the more time it will take DataRobot to build the model.
After thoroughly scrutinizing your preferred model and making relevant adjustments to its hyperparameters, you can deploy, monitor, and manage Visual AI models in the same way you do with models without image data types.
If you’re a licensed DataRobot customer, search the in-app Platform Documentation for Visual AI Overview and Visual AI reference.