A-B testing models?

A-B testing models?

Hi Is there a way to deploy 2 models to do A/B testing? hoping yes and that you can give an example?

2 Replies
DataRobot Alumni

Hi @krista-kelly 

A/B testing of models is possible with our MLOps product. MLOps has a feature called Challenger Models. The idea is that you deploy multiple models at the same time - one Champion model, and the others as Challenger models. While only the Champion model is used to make predictions, all models' performance is constantly evaluated and if a Challenger model performs better than the Champion you can easily replace it.

The feature is described in this blog post:
https://www.datarobot.com/blog/introducing-mlops-champion-challenger-models/ and we will share a tutorial once one is available.

Hey there, Krista-Kelly! Absolutely, A/B testing with machine learning models is a fascinating topic! It's akin to orchestrating a symphony of algorithms to enhance decision-making. I've delved into this realm before and have a nifty solution to share.

Firstly, you'll need a solid framework to deploy your models. I'd recommend leveraging a containerization platform like Docker for this purpose. It's like having little magic boxes that encapsulate your models and dependencies. Once you've got your models neatly packed, you can deploy them to separate containers. So, If you are as passionate about the topic of testing as I am, I suggest researching this topic - Automated Testing Benefits: Why It's Importance for Your Business?

Now, here comes the twist: orchestration tools like Kubernetes can be your maestro. They allow you to manage and scale these containers effortlessly. Deploy Model A in one container and Model B in another, and let Kubernetes handle the rest.

For A/B testing, use a load balancer or a proxy server to route incoming requests to the respective model containers. This ensures that a portion of your traffic interacts with Model A while another experiences Model B.

Remember to carefully monitor and analyze the results using metrics like conversion rates or user engagement to gauge the model performance. Adjust the traffic split dynamically based on these metrics, and voila, you've got a dynamic A/B testing setup for your models!

0 Kudos