Symptom
The distortion lives in what per-element significance cannot see: the co-occurrence of other active experiments in the same session pool.
You asked
Test results show strong performance from combined variations, with clear improvements in conversion rates when multiple changes are applied together. The experiment dashboard highlights winning combinations that appear to outperform previous versions.
Symptom
The distortion lives in what per-element significance cannot see: the co-occurrence of other active experiments in the same session pool.
Cause
The mechanism is co-occurrence contamination — multiple experiments drawing from the same session pool inflate per-element lift by including interaction contributions that disappear at rollout.
Impact
This is a test design problem — repeated failed rollouts consuming testing bandwidth and reducing experimentation program ROI without triggering any dashboard alert.
Test results show strong performance from combined variations, with clear improvements in conversion rates when multiple changes are applied together. The experiment dashboard highlights winning combinations that appear to outperform previous versions.
The distortion lives in what per-element significance cannot see: the co-occurrence of other active experiments in the same session pool.
The mechanism is co-occurrence contamination — multiple experiments drawing from the same session pool inflate per-element lift by including interaction contributions that disappear at rollout.
This is a test design problem — repeated failed rollouts consuming testing bandwidth and reducing experimentation program ROI without triggering any dashboard alert.