Symptom
At 78%, the experiment win rate is well above industry average.
You asked
Results look strong during rollout but don’t generalize.
Symptom
At 78%, the experiment win rate is well above industry average.
Cause
The mechanism is audience filtering at the exposure stage.
Impact
The economic cost of selection bias accumulates in the gap between what the test predicted and what the rollout delivered.
Results look strong during rollout but don’t generalize.
At 78%, the experiment win rate is well above industry average.
The mechanism is audience filtering at the exposure stage.
The economic cost of selection bias accumulates in the gap between what the test predicted and what the rollout delivered.