← Back to Episodes
Why do statistically significant tests not matter?
Tests show significance but don’t impact the business.
S: At 8.4%, the free-to-paid conversion rate is up four-tenths of a point from the control.
C: The mechanism is a gap between two standards that sound equivalent but measure completely different things.
I: At four additional paid users per one thousand trial users exposed, the incremental revenue from rolling out this variant requires high volume to produce a return that justifies th