Experiment Design

In one line: an experiment answers whether a change helped, and it only answers honestly if you decided how you would read it before you ran it.

Sample size follows from the effect you need to detect

The single most useful relationship: the smaller the effect, the more traffic you need — and it scales with the inverse square.

Halve the effect you want to detect  ->  FOUR times the sample

That relationship drives most experimentation strategy. Detecting a 10% lift is quick; detecting a 1% lift takes a hundred times the traffic. And since most changes produce small effects, most experiments are slow.

Three quantities determine the duration, and stating them shows you have run one:

Baseline rate. Where the metric sits today. Rarer events need more traffic — a 0.5% conversion rate needs far more than a 30% click rate.

Minimum detectable effect. The smallest change worth acting on. This is a business decision, not a statistical one, and it is the one people skip. If a 0.5% improvement would not change what you do, do not power the test to find it.

Variance. How noisy the metric is. Revenue per user is far noisier than click rate, because a few large purchases dominate — which is why revenue experiments need much more traffic than engagement ones.

Peeking inflates false positives

The most common way a good team gets a wrong answer.

Checking the result repeatedly and stopping when it turns significant is not the test you think you ran. Each look is another chance for noise to cross the threshold, so a nominal 5% false-positive rate becomes much higher — with continuous monitoring, the probability of eventually seeing significance approaches certainty even when nothing is happening.

Two legitimate fixes. Fix the duration in advance and read the result once at the end — simple and it requires discipline nobody enjoys. Or use a sequential testing method built for continuous monitoring, which adjusts the threshold to account for repeated looks and lets you stop early honestly.

What is not legitimate is running a fixed-horizon test and reading it daily. That is the default behaviour of every dashboard, which is why it is the default mistake.

Novelty and primacy

Two time-varying effects that make a short experiment lie in opposite directions.

Novelty effect. Users engage with a change because it is different, not because it is better. A redesigned button gets clicks because it moved. The effect decays over days or weeks, and a one-week test measures the novelty rather than the change.

Primacy effect. The reverse. Users are slowed by having to learn something new, so a genuinely better change looks worse initially and improves as people adapt.

The detection method is the same for both and is worth stating: plot the effect over time rather than reporting one aggregate. A flat line is a real effect. A declining one is novelty. A rising one is primacy or a slow rollout artifact. Reporting a single averaged number throws away exactly the information that tells you which you are looking at.

Trust checks before interpretation

Before believing any result, three checks. Each is cheap and each invalidates the experiment on its own.

Sample ratio mismatch. You asked for a 50/50 split; check that you got one. A significant deviation means assignment or logging is broken, and no result from that experiment is usable — not even a directionally interesting one. This is the first thing a mature team checks and the thing most candidates have never heard of.

An A/A test. Run the identical experience against itself. It should show no difference. If it shows a significant one, your pipeline manufactures effects and every past result is suspect.

Guardrails and segments. A win on the primary metric with a latency regression is not a win. And an aggregate improvement can hide a segment collapsing, which is why the segment breakdown belongs in the analysis rather than in a follow-up.

Testing many things at once

Run twenty experiments at a 5% significance level and roughly one will look significant by chance. Run twenty metrics on one experiment and the same applies.

Two disciplines handle it. Declare the primary metric in advance — one metric decides the launch, everything else is context, and a secondary metric that turns significant is a hypothesis for the next test rather than a result from this one. And correct for multiple comparisons when you genuinely are testing many hypotheses.

Next: running the whole thing against a real problem.