← Back to Episodes

You asked

Why do statistically significant tests not matter?

Tests show significance but don’t impact the business.

Symptom

At 8.4%, the free-to-paid conversion rate is up four-tenths of a point from the control.

Cause

The mechanism is a gap between two standards that sound equivalent but measure completely different things.

Impact

At four additional paid users per one thousand trial users exposed, the incremental revenue from rolling out this variant requires high volume to produce a return that justifies th

Full diagnostic context

Tests show significance but don’t impact the business.

At 8.4%, the free-to-paid conversion rate is up four-tenths of a point from the control.

The mechanism is a gap between two standards that sound equivalent but measure completely different things.

At four additional paid users per one thousand trial users exposed, the incremental revenue from rolling out this variant requires high volume to produce a return that justifies th