How Many Visitors Do I Need for an A/B Test?

There is no fixed visitor count that makes every A/B test reliable. The traffic you need depends on your baseline conversion rate, the smallest improvement worth detecting, your statistical method and your tolerance for uncertainty. Work out the sample per variant before launching, then estimate how long eligible traffic will take to reach it.

For a conventional two-group conversion test, a small improvement at a low baseline can require tens of thousands of eligible visitors. "Run it until each version has 100 conversions" is not a universal sample-size rule.

If you are evaluating Niwa's product assistance on WordPress, define the completed purchase outcome first. Our CRO introduction explains the distinction between a useful business outcome and an intermediate action. A count of chat openings cannot establish the traffic needed to detect a purchase-rate change.

Start with four planning inputs

Baseline conversion rate. Use a relevant recent estimate from the eligible population. A whole-site rate is unsuitable if only visitors to selected product pages enter the test.

Minimum detectable effect. This is the improvement you design the test to be able to detect. It should be commercially meaningful. At a 3% baseline, moving to 3.6% is a 0.6 percentage point increase and a 20% relative lift.

Significance level. In a conventional frequentist test, a 5% level limits the probability of rejecting the null hypothesis when it is true, under the test's assumptions. It does not mean there is a 95% probability that the winning version is better.

Statistical power. Power is the chance of detecting an effect of the planned size when it exists, under the design assumptions. An 80% power target still permits some real effects of that size to go undetected.

Optimizely's sample planning guidance explains why the baseline and detectable effect matter. Its calculator uses its own Stats Engine method. A calculator using a different statistical method can legitimately give a different result.

A worked traffic example

Consider an illustrative, fixed-horizon test with two equally sized groups:

• Baseline purchase conversion rate: 3%.
• Target effect for planning: 3.6%, a 20% relative lift.
• Two-sided significance level: 5%.
• Power: 80%.
• One primary binary conversion outcome per independently assigned user.

A standard normal approximation for two independent proportions gives approximately 13,914 users per variant. Rounding the plan up to 14,000 per variant gives 28,000 total eligible users.

This is a reproducible planning approximation, not an Optimizely calculator output or a guarantee that a result will be conclusive. It assumes independent groups, equal allocation, stable rates and no additional variants or multiple-comparison adjustment. Other methods may require a different sample.

The displayed planning formula is:

n ≈ [z(1 − α/2)√(2p̄(1 − p̄)) + z(power)√(p₁(1 − p₁) + p₂(1 − p₂))]² ÷ (p₂ − p₁)²

Here, n is the sample per group, p₁ and p₂ are the planned conversion proportions, p̄ is their average, and z is the standard normal quantile. The calculation assumes two groups of equal size. Use a calculator or analyst whose method matches your experiment system rather than mixing different stopping rules.

Turn the sample into a duration estimate

Divide the total required sample by the number of eligible newly assigned users entering daily.

At 500 per day, 28,000 users take about 56 days to accumulate. At 2,000 per day, the same sample takes about 14 days. These estimates exclude setup and any wait for delayed conversions.

Use test-entry traffic, not total site visits. If only a small share reaches the tested page, the experiment will accumulate participants more slowly. Repeat sessions from the same assigned person do not automatically create new independent participants.

Also consider business cycles. Reaching the count during a short burst of promotional traffic may not represent ordinary demand.

Do not stop because one version briefly leads

With a fixed-horizon design, follow the sample and analysis plan. Repeatedly checking ordinary significance tests and stopping at a favourable moment changes the error properties.

A sequential experiment method can permit ongoing decisions, but only when you use its actual rules. Do not borrow the "stop early" idea from a sequential tool and apply it to a fixed-horizon spreadsheet.

If the result remains uncertain, report the effect estimate and uncertainty. An inconclusive test does not prove the two experiences are identical.

What to do when the traffic requirement is impractical

Check the tracking, fix broken tasks and investigate customer difficulty with usability work. CXL discusses alternatives for low-traffic websites. This work can guide improvements, but before-and-after observations do not become randomized causal proof simply because traffic is limited.

For a Niwa-assisted product journey, prepare accurate catalogue details, configure the storefront assistant and test real product selection before evaluating the effect. Review order quality and returns alongside conversion. Niwa supplies the customer-assistance workflow; an experiment system supplies random assignment and statistical analysis.

Contact Niwa with your WordPress site and the customer task you want to support. Use your actual eligible traffic and baseline to decide how the outcome can reasonably be evaluated.