Skip to main content

A/B Testing in React: Setup and Instrumentation

A/B testing is the scientific method for product decisions: you hypothesize that change X improves outcome Y, run an experiment where 50% of users see X and 50% don't, measure Y, and use statistics to decide if X is real progress or random noise. This article covers designing rigorous A/B tests in React, assigning users consistently to variants, and instrumenting tracking so your results are valid.

I ran my first A/B test in 2018 on a checkout flow, thinking higher CTA buttons would increase conversions. We didn't randomize properly (we assigned by browser type, not uniformly), and statistical noise drowned out the signal. The test was inconclusive, but we thought we had results. A year later, I learned proper experiment design. Now, every test we run includes pre-registered hypotheses, power calculations, and valid statistical analysis. That rigor has saved us from shipping dozens of wrong changes.

A/B Test Fundamentals: Hypothesis and Power​

Before you code, you need a hypothesis. "Taller buttons are better" is vague. A rigorous hypothesis is: "Adding 8 px of padding to the checkout CTA will increase conversion rate by at least 2%, from baseline 4% to 4.08%, with 80% statistical power."

Key terms:

  • Control: the baseline experience (existing CTA size).
  • Variant: the treatment (new CTA size).
  • Conversion rate: the outcome metric (% of visitors who complete checkout).
  • Baseline: the current conversion rate for the metric (4%).
  • Minimum detectable effect (MDE): the smallest real improvement you care about (2 percentage points = 50% relative lift).
  • Statistical power: the probability of detecting a real effect if it exists. 80% power means if the true effect is 2 points, you'll detect it 80% of the time (20% chance of false negative).
  • Significance level (alpha): the false-positive rate. 5% alpha means if there is no true effect, there's a 5% chance you'll incorrectly declare a winner (false positive).

To determine sample size, use a power calculator:

n = (z_alpha + z_beta)^2 * (p1(1-p1) + p2(1-p2)) / (p1 - p2)^2

where:
p1 = baseline conversion rate (0.04)
p2 = variant conversion rate (0.0408)
z_alpha = 1.96 (two-tailed, 5% significance)
z_beta = 0.84 (80% power)

n ≈ 391,000 users per variant

So you need roughly 391k users in control and 391k in variant to detect a 2-point lift. If your site gets 100k users/week, the test takes 8 weeks.

Use an online calculator like evan.miller.org rather than doing math by hand.

Designing a Rigorous A/B Test​

A rigorous A/B test has these properties:

  1. Pre-registered hypothesis: Before the test, document what you expect to find (baseline, MDE, power).
  2. Consistent variant assignment: Each user is deterministically assigned to control or variant (based on user ID hash, not browser state). The same user always sees the same variant.
  3. Valid randomization: Users are randomly assigned (50% control, 50% variant), not by browser type, geography, or any factor that correlates with the outcome.
  4. Blinded evaluation: The person analyzing results doesn't know which variant is which (you label them "A" and "B" during analysis, label them "control"/"treatment" after).
  5. Pre-registered success metrics: You declare upfront which metrics matter (primary: conversion rate; secondary: AOV, bounce rate).
  6. Intention-to-treat analysis: You analyze all users assigned to control vs variant, regardless of whether they were exposed (e.g., if a user in variant wasn't actually shown the variant due to a bug, they're still analyzed as variant).
  7. Multiple comparison correction: If you analyze 10 metrics, you need to account for false positives by adjusting the significance level (e.g., Bonferroni: alpha = 0.05 / 10 = 0.005 per metric).

Variant Assignment in React​

Assign users to variants in the backend when they log in or on the flag/config fetch. If you assign in the frontend, you risk inconsistency (user reloads page, gets different variant). If you assign on the backend, it's deterministic and auditable.

Backend approach (Recommended):

// Backend pseudocode
app.get('/api/flags', (req, res) => {
  const userId = req.user.id;
  const experimentId = 'exp_checkout_cta';
  
  // Hash the user ID to assign to variant
  const hash = hashFunction(userId + experimentId);
  const variantIndex = hash % 2; // 0 or 1
  const variants = ['control', 'variant'];
  const assignedVariant = variants[variantIndex];

  const flags = {
    exp_checkout_cta: assignedVariant,
    // ... other flags
  };

  res.json(flags);
});

The hash ensures the same user always gets the same variant (deterministic), and hashing spreads users evenly (50/50 split).

Frontend: Use the assigned variant from the backend.

// React component
export function Checkout() {
  const checkoutCTA = useFeatureFlag('exp_checkout_cta', 'control');

  return (
    <div>
      {checkoutCTA === 'control' && <CTAControl />}
      {checkoutCTA === 'variant' && <CTAVariant />}
    </div>
  );
}

For multi-variant tests (e.g., 3-way test with A, B, C), hash to 0, 1, 2 instead:

const variantIndex = hash % 3; // 0, 1, or 2
const variants = ['control', 'variant_a', 'variant_b'];

Instrumentation: Tracking Events for Analysis​

To measure the effect of your experiment, you need to track events: which variant the user is in, whether they completed the goal, and any relevant context.

Track two types of events:

  1. Exposure events: "User was assigned to variant X." Fired once per session per experiment.
  2. Conversion events: "User completed the goal (checkout)." Fired when the goal is achieved.
// React: track exposure on component mount
import { useEffect } from 'react';
import { useFeatureFlag } from './FlagContext';
import { analytics } from './analytics';

export function Checkout() {
  const checkoutCTA = useFeatureFlag('exp_checkout_cta', 'control');

  useEffect(() => {
    // Fire exposure event
    analytics.track('experiment_exposure', {
      experimentId: 'exp_checkout_cta',
      variant: checkoutCTA,
      timestamp: new Date().toISOString(),
    });
  }, [checkoutCTA]);

  const handleCheckoutComplete = () => {
    // Fire conversion event
    analytics.track('checkout_completed', {
      experimentId: 'exp_checkout_cta',
      variant: checkoutCTA,
      orderValue: 100,
      timestamp: new Date().toISOString(),
    });
    // ... complete checkout
  };

  return (
    <div>
      {checkoutCTA === 'control' && (
        <CTAControl onClick={handleCheckoutComplete} />
      )}
      {checkoutCTA === 'variant' && (
        <CTAVariant onClick={handleCheckoutComplete} />
      )}
    </div>
  );
}

Good exposure/conversion data:

userIdexperimentIdvariantevent_typeorder_valuetimestamp
u123exp_checkout_ctacontrolexperiment_exposurenull2026-06-02T10:00:00Z
u123exp_checkout_ctacontrolcheckout_completed1502026-06-02T10:05:00Z
u456exp_checkout_ctavariantexperiment_exposurenull2026-06-02T10:01:00Z
u456exp_checkout_ctavariantcheckout_completed1802026-06-02T10:06:00Z
u789exp_checkout_ctacontrolexperiment_exposurenull2026-06-02T10:02:00Z

From this data, you calculate: control conversion = 1/2 = 50%, variant conversion = 1/1 = 100%. But this is a toy example; real data has thousands of rows.

Preventing Leaks and Bias​

Common pitfalls:

  1. Assignment leaks: A user is assigned to variant but the variant code isn't actually shown to them (due to a bug, cache, or conditional logic). Always compare the assigned variant to the exposure event to catch this.

  2. Multiple-variant bias: If you run two overlapping experiments (checkout CTA and checkout design), they may interact. Document which experiments can overlap and which are mutually exclusive.

  3. Data collection bias: If variant users drop off or don't trigger tracking events, your data is biased. Always use intention-to-treat analysis: count all users assigned to control/variant, regardless of whether they were exposed.

  4. Novelty effects: Variant users might convert more just because the change is new and surprising. Run tests for at least 1 full week (user behavior changes day-of-week) and 2 weeks if possible.

To catch bias, compare segment distributions (age, geography, device type) between control and variant. If they differ, randomization failed. Good randomization means the segments are identical.

Key Takeaways​

  • A rigorous A/B test requires a pre-registered hypothesis with sample size calculation.
  • Assign users to variants deterministically in the backend (via hash), not in the frontend.
  • Track two event types: exposure (user assigned to variant) and conversion (user completed goal).
  • Use intention-to-treat analysis: analyze all users assigned, regardless of whether they saw the variant.
  • Run tests for ≥ 2 weeks to account for day-of-week effects and novelty wear-off.
  • Validate randomization by comparing segment distributions between control and variant.

Frequently Asked Questions​

How do I handle repeated experiments on the same user?​

Tag each experiment with a unique ID (exp_checkout_cta, exp_homepage_layout). Users can participate in multiple experiments simultaneously, but each exposure/conversion event should include both the experiment ID and variant. During analysis, filter by experiment ID.

What if my test doesn't reach the required sample size?​

Run the test longer, or increase the MDE (accept a smaller improvement as "real"). Going live without sufficient power is risky: you may have a false negative and miss a winning variant, or a false positive and ship a loser.

Should I analyze results while the test is running?​

No. Peeking at results before the test is finished inflates the false-positive rate. If you "see" a winner at 50% of the calculated sample size, there's a higher than 5% chance it's random noise. Declare the test schedule upfront (e.g., "test runs for 2 weeks"), and analyze only after that period.

How do I account for multiple metrics?​

Use the Bonferroni correction: if you analyze N metrics, divide the significance level by N. So for 5 metrics, use alpha = 0.05 / 5 = 0.01 per metric. Or use a more conservative method like Holm-Bonferroni. Always declare your metrics pre-test.

Can I stop a test early if the winner is clear?​

Yes, but use sequential testing (e.g., Wald's SPRT) to maintain validity. Sequential testing adjusts the significance level so you can peek early without inflating false positives. It's beyond the scope of this article, but tools like Statsig and Optimizely handle sequential testing automatically.

Further Reading​