← Back to Episodes

Why did my winning test variation stop working after rollout?

But when those same winning elements are rolled out independently, performance frequently drops — the lift came from how multiple elements worked together, not from any one element in isolation.

S: The distortion lives in what per-element significance cannot see: the co-occurrence of other active experiments in the same session pool. C: The mechanism is co-occurrence contamination — multiple experiments drawing from the same session pool inflate per-element lift by including interaction contributions that disappear at rollout. I: This is a test design problem — repeated failed rollouts consuming testing bandwidth and reducing experimentation program ROI without triggering any dashboard alert.