Analyze A/B Tests in TeamSQL

In these days A/B Tests are a common tool used by owners of e-commerce web sites. In few words we can say that A/B tests are like the speedometer of a race car, they serve to measure the most important metric of an e-commerce site: The conversion rate. In this article we will explain how to create a chart in TeamSQL to see, compare and evaluate how an e-commerce web site is performing.

 

First step: To have a metric

One of the most important metrics in these days to measure the efficiency of a E-commerce web site is the conversion rate, which is a number between 0 and 1 representing the percentage of people who closed a transaction in a e-commerce site, related to all the people who visited the site during a period of time.

We don’t need an A/B test to know the conversion rate, we can obtain it, by simply count all the visitors to the site during a period of time and the quantity of sells we made, then we can do the next math operation:

 

Conversion Rate = Number of Sells done / Number of visitor received

Let’s suppose during a day we had 1000 visitors to our site, and we closed 120 transactions. Then our conversion rate is 120/1000 = 0.12 (0.12 means 12%)

 

Can I trust in Conversion Rate?

Hummm, yes you can. But we need to know what a conversion rate is, in order to understand the real significance of this metric. Conversion rate is the result of an estimation (or a guess), if we try to estimate it again, we will likely get other different value (perhaps very close to the previous one). This is the reason why all estimators should came along with an error level measure. And based on that error level we define a range instead of a single value. A range where the estimator (the conversion rate in our case) is with a high level of confidence. For example we can say:

The conversion rate of the site is IN the range [11% , 13%] with a 95% of confidence.

However in order to simplify, the statement, many people would say: the conversion rate is 12% (which is the medium value in the range). This simplification can be the root of a wrong site evaluation and wrong decisions.

 

What are A/B Tests for ?

A/B Tests are a compare tool or technique to determine if a change to the site improves the conversion rate or not, however you don’t fall on the simple compare of conversion rate single values. You must extract conclusion by comparing both conversion rate ranges.

 

In the previous image we can conclude that Site Variation B is better because the range for conversion rate is higher ever. Even in the worst case (23%) of Variation B we have a value of conversion rate higher than any value in Variation A range. In this example we say we don’t have overlapping of ranges.

 

Let’s see a case with overlapping:

 

Is the Variation B better or not ? Well, seems to be better, however we cannot be sure, What about if “Variation A site” has a conversion rate of 21% and “Variation B site” has a conversion rate of 20%?. In this case A is better than B. Anyway, we can conclude that we need to continue testing.

 

Sample table: e-commerce A/B Test data

The following table includes date of 6 A/B tests done on a e-commerce site. There are 7 sets of test data: one set of data corresponds to the test of the original site and the other 6 data sets were obtained by testing the site after each of 6 improvements done. Each data set is composed by 5 tests (or records) done with 100 users each. The column “test_code” represents the name of the improvement, while the column “test_rate” represents the conversion rate obtained on this test.

 

 

How to obtain the range for conversion rate

In order to obtain the range for the conversion rate, we need to use two aggregation functions. First we will use the average (avg) function to obtain the media or average value of all the conversion rates. The media will be the central point of the range. Then, the second function we need to use is the standard deviation. With the result of the standard deviation function we will define the start and end points of the range. Let’s see how to calculate both points

 

In order to have a confident of 95%, the start of the range will be the point:

average — 2 * standard deviation

and the end of the range will be the point:

average + 2 * standard deviation

 

Let’s go now to TeamSQL, in the charts window, we can use the aggregation functions. In this case we will use the average function to show the central point of the conversion rate range as we can see in the following print screen.

 

 

The chart is the following:

 

 

However, the previous chart only shows the average value or central point in the conversion rate range. We would like to see the range of each test in the chart. Then we will add two more lines: one line (black) at the bottom representing the start of the range, and one line on top (green) showing the end of the range. Let’s see the new chart

 

In order to understand the previous chart we need to isolate and identify each conversion range. For example in the following image we can see the conversion range for the test NEW UX. The range for this test is 14% to 15.7%, this means that conversion rate is in the range [14% , 15.7%] with a confidence of 95%

 

 

 

The next step is to put all the pieces together and understand how to compare the range of Variation A with the range of variation B, Let’s see an example:

 

In the previous image we can see that Variation A range is 8%-11%, while variation B range is 14%-15.7%. In this case, it is clear that Variation B is better than A, because the worst case on Variation B (14%) is better than the best case on variation A (11.8%). We leave to the reader the analysis of range when there are range overlapping.

 

Conclusion

In this article we covered an important metric called conversion rate metric, the A/B Tests technique and how to create a chart in TeamSQL to evaluate and analyze results of A/B Tests. Charts in TeamSQL are very flexible, you can use aggregation functions (like sum, average, standard deviation) to calculate new metrics on which base the charts.. You can apply this technique to any change to your e-commerce web site. Let’s try to adapt to your database!