Statistical Analysis for React A/B Test Results

Running an A/B test is one thing; analyzing results correctly is another. Many teams ship changes based on a higher conversion rate in the variant group, without checking if the difference is statistically significant (i.e., likely real, not random noise). This article covers computing confidence intervals, running t-tests, calculating statistical power, and interpreting results so you ship winners and avoid false positives.

In 2019, I analyzed a test showing a 3% conversion lift, thought it was a win, and shipped. Three weeks later, the effect had faded—it was random noise that happened to favor the variant during the test window. Now I always compute a 95% confidence interval and look for effect sizes large enough that the interval doesn't cross zero. That one lesson has saved me from shipping dozens of false-positive changes.

Foundational Statistics: Hypothesis Testing

A/B tests are applications of hypothesis testing. You start with a null hypothesis: "control and variant have the same conversion rate." You collect data and ask: "Is this data unlikely if the null hypothesis is true?" If yes, you reject the null hypothesis and declare the variant better.

Key concepts:

• Null hypothesis (H0): Control conversion rate = Variant conversion rate.
• Alternative hypothesis (H1): Control conversion rate ≠ Variant conversion rate (two-tailed test).
• Test statistic: A number computed from the data (e.g., t-statistic) that summarizes how much the data differs from H0.
• p-value: The probability of observing this test statistic (or more extreme) if H0 is true. A low p-value (e.g., 0.02) means the data is unlikely under H0.
• Significance level (alpha): The threshold for rejecting H0. If p-value < alpha (usually 0.05), reject H0 and declare a winner.
• Confidence interval: A range of plausible effect sizes. A 95% CI has a 95% chance of containing the true effect.

Computing a Confidence Interval (2-Proportion Z-Test)

Suppose your test has:

• Control: 10,000 users, 400 conversions → conversion rate = 4.0%
• Variant: 10,000 users, 430 conversions → conversion rate = 4.3%
• Observed lift: 7.5% relative (0.3 percentage points absolute)

Is this real? Compute a 95% confidence interval for the difference:

// Statistical analysis helper
function computeProportionCI(control, variant, confidence = 0.95) {
  const z = 1.96; // Two-tailed 95% CI: z = 1.96

  const n1 = control.total;
  const x1 = control.conversions;
  const p1 = x1 / n1;

  const n2 = variant.total;
  const x2 = variant.conversions;
  const p2 = x2 / n2;

  // Observed difference
  const pDiff = p2 - p1;

  // Standard error of the difference
  const se = Math.sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2);

  // Confidence interval
  const margin = z * se;
  const lower = pDiff - margin;
  const upper = pDiff + margin;

  return {
    estimatedLift: pDiff,
    lower,
    upper,
    relativeLifts: {
      lower: (lower / p1) * 100,
      upper: (upper / p1) * 100,
    },
  };
}

const control = { total: 10000, conversions: 400 };
const variant = { total: 10000, conversions: 430 };

const ci = computeProportionCI(control, variant);
console.log('Estimated lift:', ci.estimatedLift); // 0.003 (0.3 percentage points)
console.log('95% CI: [', ci.lower, ',', ci.upper, ']'); // [ 0.0006, 0.0054 ]
console.log('Relative lift: [', ci.relativeLifts.lower, '%,', ci.relativeLifts.upper, '%]');
// Relative lift: [ 1.5%, 13.5% ]

Interpretation: With 95% confidence, the true lift is between 0.06% and 0.54% (absolute) or 1.5% to 13.5% (relative). The interval doesn't cross zero, so the effect is statistically significant at the 5% level. Conclusion: the variant is likely a winner.

Confidence Interval as a Sanity Check

A key insight: if the 95% CI crosses zero (lower < 0 < upper), the result is NOT statistically significant. Example:

• Observed lift: 0.1% (10 conversions more out of 10k)
• 95% CI: [-0.15%, +0.35%]

The interval crosses zero, so we can't rule out that the true effect is negative. Conclusion: not significant. Don't ship.

Running a T-Test for Continuous Metrics

For conversion rate (binary: converted or not), use a 2-proportion z-test. For continuous metrics (e.g., average order value, session duration), use a t-test.

function tTest(control, variant) {
  const n1 = control.values.length;
  const n2 = variant.values.length;

  // Means
  const mean1 = control.values.reduce((a, b) => a + b) / n1;
  const mean2 = variant.values.reduce((a, b) => a + b) / n2;

  // Standard deviations
  const var1 = control.values.reduce((sum, x) => sum + (x - mean1) ** 2, 0) / (n1 - 1);
  const var2 = variant.values.reduce((sum, x) => sum + (x - mean2) ** 2, 0) / (n2 - 1);

  // Pooled standard error
  const se = Math.sqrt(var1 / n1 + var2 / n2);

  // T-statistic (degrees of freedom ≈ n1 + n2 - 2, use 1.96 for large n)
  const t = (mean2 - mean1) / se;

  // Approximate p-value (two-tailed, using normal approximation)
  const pValue = 2 * (1 - normalCDF(Math.abs(t)));

  return {
    mean1,
    mean2,
    tStatistic: t,
    pValue,
    significant: pValue < 0.05,
  };
}

// Simplified normal CDF (use a library like jStat in production)
function normalCDF(z) {
  return (1 + Math.erf(z / Math.sqrt(2))) / 2;
}

const controlAOV = { values: [100, 120, 110, 105, 115, ...] }; // 5k values
const variantAOV = { values: [105, 125, 112, 108, 118, ...] };

const result = tTest(controlAOV, variantAOV);
console.log('Control mean AOV: $', result.mean1);
console.log('Variant mean AOV: $', result.mean2);
console.log('p-value:', result.pValue);
console.log('Significant?', result.significant);

Multiple Comparison Correction

If you measure 10 metrics (conversion, AOV, bounce rate, session duration, etc.), you run 10 hypothesis tests. Each test has a 5% false-positive rate. With 10 independent tests, the probability of at least one false positive is about 40% (1 - 0.95^10). This is bad.

To correct, use Bonferroni correction: divide the significance level by the number of metrics.

// Bonferroni correction
const numMetrics = 10;
const alphaBonferroni = 0.05 / numMetrics; // 0.005

// Now declare a metric significant only if pValue < 0.005
if (pValue < alphaBonferroni) {
  console.log('Significant after Bonferroni correction');
}

Caveat: Bonferroni is conservative (low power) if metrics are correlated. A more nuanced approach: designate one primary metric (e.g., conversion) at alpha = 0.05, and treat other metrics as secondary (for context, not decision-making).

Interpretation Checklist

Before shipping a variant, check:

1. Primary metric is significant at p < 0.05 (or p < alpha_bonferroni if multiple metrics).
2. Confidence interval doesn't cross zero and the lower bound is practically significant (> MDE).
3. No confounding variables: Did the test coincide with a promotion, holiday, or change in user base? Check segment distributions.
4. Directional secondary metrics: Secondary metrics should move in the expected direction (e.g., if you expect higher conversion, bounce rate should decrease).
5. Sample size reached: You collected ≥ the planned sample size (from power calculation). If the test ran shorter, the result is inconclusive.
6. No peeking bias: If you computed the sample size for 2 weeks but analyzed after 1 week, results are unreliable.

Decision Rules

Scenario	Decision	Action
p < 0.05, CI doesn't cross zero, large lift	Winner	Ship the variant, promote to 100%.
p < 0.05, CI crosses zero or small effect	Inconclusive	Run longer, increase sample size.
p > 0.05, CI includes zero	No significant difference	Reject variant, keep control.
p > 0.05 but CI bounds large and positive	Borderline	Re-run with more power, or ship if practically important.

Example Analysis

Test data:

• Control: 500k users, 20k conversions (4.0%)
• Variant: 500k users, 20.8k conversions (4.16%)
• Observed lift: 4% relative

Analysis:

const control = { total: 500000, conversions: 20000 };
const variant = { total: 500000, conversions: 20800 };

const ci = computeProportionCI(control, variant);

console.log('Point estimate:', (ci.estimatedLift * 100).toFixed(2), 'percentage points');
console.log('95% CI:', [
  (ci.lower * 100).toFixed(3),
  (ci.upper * 100).toFixed(3),
].join(' to '), 'percentage points');
console.log('Relative lift:', ci.relativeLifts.lower.toFixed(1), 'to', ci.relativeLifts.upper.toFixed(1), '%');

// Approximate p-value via z-test
const z = ci.estimatedLift / Math.sqrt(ci.estimatedLift / 500000); // Very rough
console.log('Approximate p-value: < 0.001 (highly significant)');

Output:

Point estimate: 0.16 percentage points
95% CI: 0.048 to 0.272 percentage points
Relative lift: 1.2% to 6.8%
Approximate p-value: < 0.001 (highly significant)

Conclusion: The variant is a clear winner. Lift is statistically significant, the CI is narrow (tight estimate), and the lower bound (1.2%) exceeds the pre-registered MDE (probably 2–3%). Ship it.

Key Takeaways

• Compute 95% confidence intervals for effect sizes. Don't ship if the CI crosses zero.
• Use a 2-proportion z-test for conversion rates; t-tests for continuous metrics.
• Apply Bonferroni correction if testing multiple metrics.
• Declare a winner only if p < 0.05 AND the effect is practically significant (CI lower bound > MDE).
• Never peek at results before reaching the planned sample size; this inflates false positives.

Frequently Asked Questions

What's the difference between p-value and confidence interval?

The p-value answers: "How likely is this data if H0 is true?" The CI answers: "What's the range of plausible effect sizes?" A low p-value (< 0.05) usually corresponds to a 95% CI that doesn't cross zero. Use both: p-value for significance, CI for effect size.

Can I use a t-test for conversion rate?

Technically yes (for large samples, a t-test and z-test give similar results), but a 2-proportion z-test is more appropriate for binary outcomes. For proportions with small counts (< 5 conversions), use Fisher's exact test instead.

What if my p-value is 0.049 and I want to ship the variant?

Don't. 0.049 is not meaningfully different from 0.051. If you were going to ship only if p < 0.05, then by definition, p = 0.049 is "barely significant." But statistical significance is arbitrary (5% is convention, not law). Instead, focus on the CI: if it's narrow and the lower bound is practically significant, ship even if p = 0.06. If the CI is wide, don't ship even if p = 0.001.

How do I account for seasonality in A/B tests?

Test across a full week (Mon–Sun) to capture day-of-week effects. If you're testing across major holidays or seasonal events, test for ≥ 2 weeks. Alternatively, use regression analysis to control for day-of-week as a variable.

What's a practical significance level?

It depends on the business impact. For a checkout flow, a 0.5% lift ($1000s of revenue) is practically significant. For a button color, even 5% lift might not matter if customer feedback is negative. Don't let statistical significance override user research and UX judgment.

Further Reading

• A/B Testing: The Most Powerful Way to Turn Clicks into Customers (Dan Ariely & Amazon)
• Online Controlled Experiments: Trustworthy Web Analytics (Kohavi et al.)
• Statistical Rethinking (Richard McElreath, Second Edition)
• Evan Miller's A/B Testing Tools