Analyze A/B Tests in TeamSQL

In these days A/B Tests are a standard tool used by owners of e-commerce websites. In a few words we can say that A/B tests are like the speedometer of a race car, they serve to measure the most critical metric of an e-commerce site: The conversion rate. In this article, we will explain how to create a chart in TeamSQL to see, compare and evaluate how an e-commerce website is performing.

 

First step: To have a metric

One of the most critical metrics in these days to measure the efficiency of an E-commerce website is the conversion rate, which is a number between 0 and 1 representing the percentage of people who closed a transaction in an e-commerce site, related to all the people who visited the site during a period.

We don’t need an A/B test to know the conversion rate, we can obtain it, by counting all the visitors to the site during a period and the number of sells we made, then we can do the next math operation:

 

Conversion Rate = Number of Sells done / Number of visitor received

Let’s suppose during a day we had 1000 visitors to our site, and we closed 120 transactions. Then our conversion rate is 120/1000 = 0.12 (0.12 means 12%)

 

Can I trust in Conversion Rate?

Hmmm, yes you can. But we need to know what a conversion rate is, to understand the real significance of this metric. The conversion rate is the result of an estimation (or a guess), if we try to estimate it again, we will likely get other different value (perhaps very close to the previous one). This is the reason why all estimators should come along with an error level measure. And based on that error level we define a range instead of a single value. A range where the estimator (the conversion rate in our case) is with a high level of confidence. For example, we can say:

The conversion rate of the site is IN the range [11 %, 13%] with a 95% of confidence.

However to simplify, the statement, many people would say: the conversion rate is 12% (which is the medium value in the range). This simplification can be the root of wrong site evaluation and wrong decisions.

 

What are A/B Tests for?

A/B Tests are a compare tool or technique to determine if a change to the site improves the conversion rate or not. However, you don’t fall on the simple comparison of conversion rate single values. You must extract the conclusion by comparing both conversion rate ranges.

 

In the previous image, we can conclude that Site Variation B is better than the Site Variation A because the range for the conversion rate is higher ever. Even in the worst case (23%) of Variation B, we have a value of conversion rate higher than any value in Variation A range. In this example, we say we don’t have an overlapping of ranges.

 

Let’s see a case with overlapping:

 

Is the Variation B better or not? Well, seems to be better. However, we cannot be sure. What about if “Variation A site” has a conversion rate of 21% and “Variation B site” has a conversion rate of 20%?. In this case, A is better than B. Anyway; we can conclude that we need to continue testing.

 

Sample table: e-commerce A/B Test data

The following table includes the date of 6 A/B tests done on an e-commerce site. There are seven sets of test data: one set of data corresponds to the analysis of the original site, and the other six datasets were obtained by testing the site after each of 6 improvements done. Each data set is composed of five tests (or records) done with 100 users each. The column “test_code” represents the name of the improvement, while the column “test_rate” represents the conversion rate obtained on this test.

 

 

How to obtain the range for conversion rate

To get the range for the conversion rate, we need to use two aggregation functions. First, we will use the average (avg) function to obtain the media or average value of all the conversion rates. The media will be the central point of the range. Then, the second function we need to use is the standard deviation. With the result of the standard deviation function, we will define the start and end points of the range. Let’s see how to calculate both points

 

To have a confident of 95%, the start of the range will be the point:

average — 2 * standard deviation

and the end of the range will be the point:

average + 2 * standard deviation

 

Let’s go now to TeamSQL, in the charts window; we can use the aggregation functions. In this case, we will use the average function to show the central point of the conversion rate range as we can see in the following print screen.

 

 

The chart is the following:

 

 

However, the previous chart only shows the average value or central point in the conversion rate range. We would like to see the variety of each test in the chart. Then we will add two more lines: one line (black) at the bottom representing the start of the range, and one line on top (green) showing the end of the spectrum. Let’s see the new chart

 

To understand the previous chart we need to isolate and identify each conversion range. For example, in the following image, we can see the conversion range for the test NEW UX. The field for this test is 14% to 15.7%; this means that the conversion rate is in the range [14 %, 15.7%] with a confidence of 95%

 

 

 

The next step is to put all the pieces together and understand how to compare the range of Variation A with the range of variation B, Let’s see an example:

 

In the previous image, we can see that Variation A range is 8%-11%, while variation B range is 14%-15.7%. In this case, it is clear that Variation B is better than A because the worst case on Variation B (14%) is better than the best case on variation A (11.8%). We leave the reader the analysis of range when there is range overlapping.

 

Conclusion

In this article, we covered a vital metric called conversion rate metric, the A/B Tests technique and how to create a chart in TeamSQL to evaluate and analyze the results of A/B Tests. Charts in TeamSQL are very flexible; you can use aggregation functions (like sum, average, standard deviation) to calculate new metrics on which base the charts. You can apply this technique to any change to your e-commerce website. Let’s try to adapt to your database!