In this exercise, you will learn how to analyze the results of an A/B test to determine which variant performed better and why. This involves understanding key metrics, statistical significance, and interpreting data to make informed decisions.

• Understand key metrics used in A/B testing.
• Learn how to calculate and interpret statistical significance.
• Practice analyzing A/B test results using real-world data.

• Conversion Rate (CR): The percentage of users who complete a desired action (e.g., making a purchase, signing up for a newsletter).
• Click-Through Rate (CTR): The percentage of users who click on a specific link or call-to-action.
• Bounce Rate: The percentage of visitors who navigate away from the site after viewing only one page.
• Average Order Value (AOV): The average amount spent by customers per order.

• P-Value: A measure that helps determine the significance of your results. A p-value less than 0.05 is typically considered statistically significant.
• Confidence Interval: A range of values that is likely to contain the true effect size. A 95% confidence interval is commonly used.
• Sample Size: The number of observations in each variant. Larger sample sizes generally provide more reliable results.

• Lift: The percentage increase or decrease in a metric between the control and the variant.
• Significance Level: The probability of rejecting the null hypothesis when it is true. Commonly set at 5% (0.05).

You conducted an A/B test on your e-commerce website to determine if changing the color of the "Buy Now" button from blue (Control) to green (Variant) affects the conversion rate.

1. Calculate the Conversion Rate (CR):

   • Control: \( \text{CR}_{\text{Control}} = \frac{500}{10,000} \times 100 = 5% \)
   • Variant: \( \text{CR}_{\text{Variant}} = \frac{600}{10,000} \times 100 = 6% \)

2. Calculate the Lift: \[ \text{Lift} = \frac{\text{CR}{\text{Variant}} - \text{CR}{\text{Control}}}{\text{CR}_{\text{Control}}} \times 100 = \frac{6% - 5%}{5%} \times 100 = 20% \]

3. Determine Statistical Significance:

   • Use an online A/B test significance calculator or statistical software to input the conversion rates and sample sizes.
   • For this example, let's assume the p-value calculated is 0.03.

4. Interpret the Results:

   • Since the p-value (0.03) is less than 0.05, the result is statistically significant.
   • The green button variant resulted in a 20% lift in conversion rate compared to the blue button control.

Analyze the following A/B test data to determine which variant performed better and if the results are statistically significant.

1. Calculate the conversion rates for both the control and variant.
2. Calculate the lift in conversion rate.
3. Determine if the results are statistically significant using an A/B test significance calculator.
4. Interpret the results and provide a conclusion.

1. Calculate the Conversion Rate (CR):

   • Control: \( \text{CR}_{\text{Control}} = \frac{400}{8,000} \times 100 = 5% \)
   • Variant: \( \text{CR}_{\text{Variant}} = \frac{520}{8,000} \times 100 = 6.5% \)

2. Calculate the Lift: \[ \text{Lift} = \frac{\text{CR}{\text{Variant}} - \text{CR}{\text{Control}}}{\text{CR}_{\text{Control}}} \times 100 = \frac{6.5% - 5%}{5%} \times 100 = 30% \]

3. Determine Statistical Significance:

   • Using an A/B test significance calculator, input the conversion rates and sample sizes.
   • Assume the p-value calculated is 0.02.

4. Interpret the Results:

   • Since the p-value (0.02) is less than 0.05, the result is statistically significant.
   • The new design variant resulted in a 30% lift in conversion rate compared to the old design control.

The new design variant significantly outperformed the old design control, resulting in a 30% increase in conversion rate. Given the statistical significance, it is recommended to implement the new design.

In this exercise, you learned how to analyze A/B test results by calculating key metrics, determining statistical significance, and interpreting the data. These skills are crucial for making data-driven decisions in digital marketing.