Product thinking / DIJION
When should a product experiment stop? Decide before starting
An experiment decision card separating hypothesis, data quality and harm boundaries to prevent moving the goal after seeing results.
Trying a change is not the same as checking results daily and deciding whenever they look favorable. In an illustrative onboarding experiment, write the uncertainty to resolve before launching. Low traffic may not produce a reliable answer quickly. A serious task failure, however, can warrant intervention without waiting for an effectiveness conclusion.
Limit the hypothesis and the change
Changing copy, layout and registration requirements together complicates interpretation. A single change is not mandatory, but the package being tested must be clear. Define eligible users, sessions and outcomes. Prototype observation and a live controlled experiment provide different kinds of evidence; do not treat them as interchangeable simply because both involve a new design.
Separate different stopping decisions
Scroll the table horizontally to see all columns.
| Decision | Trigger category | Next action |
|---|---|---|
| Emergency stop | Data loss or serious task failure | Withdraw the change |
| Invalid result | Measurement or assignment failure | Do not use the result for a decision |
| Inconclusive result | Insufficient or unstable evidence | Replan or discontinue |
Write the observation plan before seeing outcomes
Duration, sample requirements and evaluation method depend on the question; there is no universal number here. Plan an appropriate statistical design when one is needed. Ending early because the result looks favorable can undermine interpretation if the chosen method does not accommodate that stopping rule. Keep any changes to the decision rules visible in the record.
Choose safeguards alongside the primary outcome
Completion can improve while support requests, incorrect records or accessibility barriers emerge. Do not treat those signals as a footnote to the main metric. Specify which trigger requires stopping and which requires investigation. Preserve uncertainty, excluded groups and unexamined conditions when communicating the outcome instead of stretching a narrow result into a general success claim.
Rehearse decisions before launch
- Write favorable, unfavorable, inconclusive and invalid-result scenarios.
- Assign an owner and action to each scenario.
- Check withdrawal in a representative environment.
- Separate the original hypothesis from exploratory observations in the report.
The decision card should allow an experiment to remain inconclusive. Preserving uncertainty is a sounder working practice than requiring every test to produce a winner.