Set up A/B testing using Nginx

A/B testing, also known as split testing, is a powerful method used by businesses to compare two versions of a webpage or app to determine which one performs better. It’s an essential tool for optimizing user experience, increasing conversions, and maximizing results. There are several A/B testing tools available, such as Google Optimize, Optimizely, VWO, or even custom solutions. However, all of these may cost you.

With Nginx, a rudimentary A/B testing system can be set up. The basic concept is very simple

• Divide traffic into buckets
• Set cookies for these buckets
• Pass these cookies with every request to and from your application
• Basis on these cookies modify your application logic

Nginx has a module named ngx_http_split_clients_module. As per the documentation this module

creates variables suitable for A/B testing, also known as split testing

This module does the main work, divide traffic into buckets. Here is an example:

  split_clients "${remote_addr}" $cookie_group {
    50% "A";
    50% "B";
  }

In this example, the incoming requests or traffic is divided into two groups, A and B. Once this is done, next steps are easy

set $ab_cookie_value '';  

if ($cookie_group = "A") {
 set $ab_cookie_value 'AB_cookie=A; Path=/';
}
if ($cookie_group = "B") {
 set $ab_cookie_value 'AB_cookie=B; Path=/';
}

# Send cookie to application
proxy_set_header Cookie "$ab_cookie_value";

# Send cookie to client
add_header Set-Cookie "$ab_cookie_value";

Set all this in your http or server context. Or in plain English, this goes inside your site configuration file.

This is a simple setup and may not scale well. In case you have Cloudflare set up checkout this article which talks about using Cloudflare workers for A/B testing.