Funnel experiments use Bayesian statistics with a beta model to evaluate the win probabilities and credible intervals for an experiment. Read the statistics primer for an overview if you haven't already.
What is a Beta model?
Imagine you run a pizza shop and want to know if customers say "yes" to adding pineapple. Some customers will say yes, others will say no. Knowing what percentage of customers want pineapple on their pizza helps you decide how much to order and what options to offer.
The beta distribution is a statistical model that's great for analyzing proportions or probabilities. It helps us understand:
- The true probability of customers saying yes to adding pineapple.
- How certain we are about that probability.
For example, if:
- Only 2 out of 4 customers (50%) say yes, the Beta distribution will be wide, indicating high uncertainty.
- 150 out of 300 customers (50%) say yes, the Beta distribution will be narrow, showing we're more confident about that 50% rate.
So when we say we're using a Beta model for funnel experiments, we're:
- Using the Beta distribution to model conversion rates between 0% and 100%.
- Getting more confident in our estimates as we collect more data.
One more thing worth noting: Bayesian inference with an initial guess that then gets updated as more data comes in. Our model uses a "minimally informative prior" of ALPHA_PRIOR = 1 and BETA_PRIOR = 1, which is like starting with a blank slate instead of making an upfront assumption about the results.
Win probabilities
The win probability tells you how likely it is that a given variant has the highest conversion rate compared to all other variants in the experiment. It helps you determine whether the experiment shows a statistically significant real effect vs. simply random chance.
Let's say you're testing a new way of presenting pineapple on the website and have these results:
- Control (current design): 100 pineapple orders from 1000 customers (10% acceptance)
- Test (suggesting pineapple with a photo): 150 pineapple orders from 1000 customers (15% acceptance)
To calculate the win probabilities for the experiment, our methodology will:
Model each variant's conversion rate using a Beta distribution:
- Control: Beta(100 + ALPHA_PRIOR, 900 + BETA_PRIOR)
- Test: Beta(150 + ALPHA_PRIOR, 850 + BETA_PRIOR)
Take 10,000 random samples from each distribution.
Check which variant had the higher conversion rate for each sample.
Calculate the final win probabilities:
- Control wins in 40 out of 10,000 samples = 0.4% probability
- Test wins in 9,960 out of 10,000 samples = 99.6% probability
These results tell us we can be 99.6% confident that showing photos of pineapple pizza performs better than the current design.
Credible intervals
A credible interval tells you the range where the true conversion rate lies with 95% probability. This is different than a confidence interval, which describes how often such intervals would contain the true rate if you repeated the experiment many times (not a direct probability statement about where the rate lies).
For example, if you have these results:
- Control (current design): 100 pineapple orders from 1000 customers (10% acceptance)
- Test (suggesting pineapple with a photo): 150 pineapple orders from 1000 customers (15% acceptance)
To calculate the credible intervals for the experiment, our methodology will:
Create a Beta distribution for each variant:
- Control: Beta(100 + ALPHA_PRIOR, 900 + BETA_PRIOR)
- Test: Beta(150 + ALPHA_PRIOR, 850 + BETA_PRIOR)
Find the 2.5th and 97.5% percentiles of each distribution:
- Control: [8.3%, 12%] = "You can be 95% confident the true conversion rate is between 8.3% and 12.0%"
- Test: [12.9%, 17.3%] = "You can be 95% confident the true conversion rate is between 12.9% and 17.3%"
Since these intervals don't overlap, you can be quite confident that the test variant performs better than the control. The intervals will become narrower as you collect more data, reflecting your increasing certainty about the true conversion rates.