Path-based routing to Portable Prediction Servers on AWS

cancel
Showing results for 
Search instead for 
Did you mean: 

Path-based routing to Portable Prediction Servers on AWS

  • Introduction
  • Prerequisites
  • Main steps
    • Step 1. Create Amazon EKS cluster
    • Step 2. Deploy NGINX Ingress controller
    • Step 3. Create and deploy services to Kubernetes
    • Step 4. Create and deploy Ingress resource for path-based routing
  • Clean up
  • Conclusion

Introduction

DataRobot MLOps (Machine Learning Operations) facilitates the routing of machine learning models to production and provides all the needed deployment, governance, and monitoring functionalities. 

Using DataRobot MLOps, you can deploy DataRobot models into their own Kubernetes clusters (cloud/on-premise) using Portable Prediction Servers (PPSs). A PPS is a Docker container that contains a DataRobot model with a monitoring agent, and can be deployed by container orchestration tools such as Kubernetes. In doing so, DataRobot customers still have the advantages of all the model monitoring provided by DataRobot’s model monitoring platform, such as service health, data drift, etc. 

When deploying multiple PPSs in the same Kubernetes cluster, you often want to have a single IP address as the entry point to all of the PPSs. A typical approach to this is path-based routing, which can be achieved using different Kubernetes Ingress Controllers. Some of the existing approaches to this include Traefik, HAProxy, Ambassador, and NGINX.

This tutorial describes how to use the NGINX Ingress controller for path-based routing to a few PPSs deployed on Amazon EKS. 

Prerequisites

There are some prerequisites to interacting with AWS and the underlying services. If any (or all) of these tools are already installed and configured for you, you can skip the corresponding steps. The detailed instructions for each step can be found here.

  1. Install the AWS CLI, version 2:
    aws --version

  2. Configure your AWS CLI credentials.

  3. Install eksctl:
    eksctl version

  4. Install and configure kubectl (CLI for Kubernetes clusters):
    kubectl version --short --client

  5. Check that you successfully installed all tools.

Output

pustinov_0-1603475286964.png

Main steps

We assume that you have already created and locally tested some PPS containers for different DataRobot AutoML models, and pushed them to Amazon Elastic Container Registry (ECR). The detailed step-by-step procedure can be found in this other community tutorial.

We created two PPSs with the models of linear regression and image classification use cases, using Kaggle housing prices dataset and Food 101 dataset, respectively.

The first PPS (housing prices) contains an eXtreme Gradient Boosted Trees Regressor (Gamma Loss) model. The second PPS (image binary classification - hot dog not hot dog), contains a SqueezeNet Image Pretrained Featurizer + Keras Slim Residual Neural Network Classifier using Training Schedule model.

The latter model has been trained using DataRobot Visual AI functionality.

Step 1. Create Amazon EKS cluster

With the Docker images stored in ECR, you can spin up an Amazon EKS cluster. The EKS cluster needs a VPC with either:

  • two public subnets and two private subnets, or 
  • three public subnets. 

Amazon EKS requires subnets in at least two Availability Zones. A VPC with public and private subnets is recommended so that Kubernetes can create public load balancers in the public subnets that load-balance traffic to pods running on nodes that are in private subnets.

  1. (Optional) Create or choose two public and two private subnets in your VPC. (Make sure that “Auto-assign public IPv4 address” is enabled for the public subnets.)

    Note: eksctl will create all necessary subnets behind the scenes if you don’t provide the corresponding --vpc-private-subnets and --vpc-public-subnets parameters.

  2. Create the cluster:

    eksctl create cluster \
    --name multi-app \
    --vpc-private-subnets=subnet-XXXXXXX,subnet-XXXXXXX \
    --vpc-public-subnets=subnet-XXXXXXX,subnet-XXXXXXX \
    --nodegroup-name standard-workers \
    --node-type t3.medium \
    --nodes 2 \
    --nodes-min 1 \
    --nodes-max 3 \
    --ssh-access \
    --ssh-public-key my-public-key.pub \
    --managed


    Note: Using the --managed parameter enables Amazon EKS-managed nodegroups. This feature automates the provisioning and lifecycle management of nodes (EC2 instances) for Amazon EKS clusters. You can provision optimized groups of nodes for the clusters; EKS will keep their nodes up-to-date with the latest Kubernetes and host OS versions. The eksctl tool makes it possible to choose the specific size and instance type family via command line flags or config files.

    Note: Although --ssh-public-key is optional, it is highly recommended that you specify it when you create your node group with a cluster. This option enables SSH access to the nodes in your managed node group. Enabling SSH access lets you connect to your instances and gather diagnostic information if there are issues; you cannot enable remote access after the node group is created.

    This command will finish as follows:

    pustinov_1-1603475287000.png

  3. Cluster provisioning usually takes between 10 and 15 minutes. When your cluster is ready, test that your kubectl configuration is correct:
    kubectl get svc

    pustinov_2-1603475286984.png

Step 2. Deploy NGINX Ingress controller

We’d like to recall here that AWS Elastic Load Balancing supports three types of load balancers: Application Load Balancers (ALB), Network Load Balancers (NLB), and Classic Load Balancers (CLB). The details can be found here.

The NGINX Ingress controller uses NLB on AWS. NLB is best suited for load balancing of TCP, UDP, and TLS traffic where extreme performance is required. Operating at the connection level (Layer 4 of the OSI model), NLB routes traffic to targets within Amazon VPC and is capable of handling millions of requests per second while maintaining ultra-low latencies. NLB is also optimized to handle sudden and volatile traffic patterns.

Deploy the NGINX Ingress controller (this manifest file also launches the NLB):

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/aws/deploy.yaml 

Step 3. Create and deploy services to Kubernetes

  1. Create a Kubernetes namespace:
    kubectl create namespace aws-tlb-namespace

  2. Save the following contents to a file named house-regression-deployment.yaml on your local machine:

    Note: You should provide the value of the image.

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: house-regression-deployment
      namespace: aws-tlb-namespace
      labels:
        app: house-regression-app
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: house-regression-app
      template:
        metadata:
          labels:
            app: house-regression-app
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                    - amd64
          containers:
          - name: house-regression-model
            image: <your_image_in_ECR>
            ports:
            - containerPort: 80

  3. Save the following contents to a file named house-regression-service.yaml on your local machine:
    apiVersion: v1
    kind: Service
    metadata:
      name: house-regression-service
      namespace: aws-tlb-namespace
      labels:
        app: house-regression-app
    spec:
      selector:
        app: house-regression-app
      ports:
        - protocol: TCP
          port: 8080
          targetPort: 8080
      type: NodePort
  4. Create a Kubernetes service and deployment:

    kubectl apply -f house-regression-deployment.yaml
    kubectl apply -f house-regression-service.yaml

  5. Save the following contents to a file named hot-dog-deployment.yaml on your local machine:

    Note: You should provide the value of the image.

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: hot-dog-deployment
      namespace: aws-tlb-namespace
      labels:
        app: hot-dog-app
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: hot-dog-app
      template:
        metadata:
          labels:
            app: hot-dog-app
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                    - amd64
          containers:
          - name: hot-dog-model
            image: <your_image_in_ECR>
            ports:
            - containerPort: 80
  6. Save the following contents to a file named hot-dog-service.yaml on your local machine:
    apiVersion: v1
    kind: Service
    metadata:
      name: hot-dog-service
      namespace: aws-tlb-namespace
      labels:
        app: hot-dog-app
    spec:
      selector:
        app: hot-dog-app
      ports:
        - protocol: TCP
          port: 8080
          targetPort: 8080
      type: NodePort
  7. Create a Kubernetes service and deployment:

    kubectl apply -f hot-dog-deployment.yaml
    kubectl apply -f hot-dog-service.yaml

  8. View all resources that exist in the aws-tlb-namespace:

    kubectl get all -n aws-tlb-namespace

Output

pustinov_3-1603475286992.png

Step 4. Create and deploy Ingress resource for path-based routing

  1. Save the following contents to a file named nginx-redirect-ingress.yaml on your local machine:

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      name: nginx-redirect-ingress
      namespace: aws-tlb-namespace
      annotations:
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/rewrite-target: /$2
      labels:
        app: nginx-redirect-ingress
    spec:
      rules:
        - http:
            paths:
              - path: /house-regression(/|$)(.*)
                backend:
                  serviceName: house-regression-service
                  servicePort: 8080
              - path: /hot-dog(/|$)(.*)
                backend:
                  serviceName: hot-dog-service
                  servicePort: 8080


    Note:
    The "nginx.ingress.kubernetes.io/rewrite-target" annotation rewrites the URL before forwarding the request to the backend pods. As a result, the paths /house-regression/some-house-path and /hot-dog/some-dog-path transform to /some-house-path and /some-dog-path, respectively.

  2. Create Ingress for path-based routing:
    kubectl apply -f nginx-redirect-ingress.yaml

  3. Verify that Ingress has been successfully created:
    kubectl get ingress/nginx-redirect-ingress -n aws-tlb-namespace

    Output
    pustinov_4-1603475286966.png

  4. (Optional) Use the following if you want to access the detailed output about this ingress:
    kubectl describe ingress/nginx-redirect-ingress -n aws-tlb-namespace

    Output
    pustinov_5-1603475286988.png

  5. Note ADDRESS for the next two scoring requests.

  6. Score the house-regression model:
    curl -X POST http://<ADDRESS>/house-regression/predictions -H "Content-Type: text/csv" --data-binary @kaggle_house_test_dataset_10.csv

    Output
    pustinov_6-1603475286995.png

  7. Score the hot-dog model:

    Note:  for_pred.csv is a CSV file containing one column with the header: the content for that column is a Base64 encoded image.

    curl -X POST http://<ADDRESS>/hot-dog/predictions -H "Content-Type: text/csv; charset=UTF-8" --data-binary @for_pred.csv

    Original picture for prediction (downloaded from here).

    pustinov_7-1603475286994.jpeg

    Output (yes, it’s predicted as hot_dog)
    pustinov_8-1603475286998.png

    Original picture for prediction (downloaded from here).
    pustinov_9-1603475287014.jpeg

    Output (right, it would be strange to name it hot_dog)
    pustinov_10-1603475286999.png

Clean up

  1. Remove the sample services, deployments, pods, and namespaces:

    kubectl delete namespace aws-tlb-namespace
    kubectl delete namespace ingress-nginx

  2. Delete the cluster:

    eksctl delete cluster --name multi-app

    Output
    pustinov_11-1603475287246.png

Conclusion

The deployment of a few Kubernetes services behind the same IP address allows you to minimize the number of load balancers needed and facilitate the maintenance of the applications. Applying Kubernetes Ingress Controllers makes it possible.

This tutorial explained how to develop the path-based routing to a few Portable Prediction Servers (PPSs) deployed on the Amazon EKS platform. This solution has been implemented via NGINX Ingress Controller.

Labels (2)
Version history
Revision #:
9 of 9
Last update:
‎10-23-2020 02:38 PM
Updated by:
 
Contributors