This interactive email campaign experiment illustrates how changing the size of each sample alters statistical power, confidence intervals, and the likelihood of detecting meaningful differences between two subject lines.
Conversions:
Rate:
Conversions:
Rate:
Discover how much you would need to adjust one factor while holding the other constant to flip the current conclusion of the test.
P-Value: The probability of observing this difference (or more extreme) if there's truly no difference between groups.
Z-Statistic: Measures how many standard errors the observed difference is from zero.
Effect Size: The actual difference between groups, regardless of statistical significance.
Statistical Power: The probability of detecting a true difference when it exists.
95% confidence intervals
Shaded bands illustrate the 95% confidence interval for each subject line as the sample sizes scale together, while the markers show the observed conversion rates for the current experiment setup. Narrower bands at larger sample sizes indicate higher precision around the same underlying rates.
Confidence intervals use a normal approximation and assume the same observed rates while sample sizes scale.
This line traces how the two-tailed p-value for the observed difference changes as both groups grow proportionally. Crossing the dashed α line shows when the projected sample size would yield statistical significance if the observed conversion rates hold.
P-values are computed with a pooled-proportion z-test using the same projected sample sizes.
H₀: p₁ = p₂ (No difference between groups)
H₁: p₁ ≠ p₂ (There is a difference between groups)
Significance Level: α =
Pooled Proportion:
Standard Error:
Group A Rate:
Group B Rate: