1. Start with a question
A poor question: ‘Let’s test a new banner.’ A better one: ‘Will a personalised offer increase repeat purchases, and will the discount eat up the additional profit?’
- What exactly are we changing?
- What result do we expect?
- Why do we think it will change?
- Which single primary metric will determine our decision?
2. Assign people randomly
The key to a fair test is that the groups should be similar before the experiment starts. The simplest way is to assign users to the variants randomly.
- Do not use ‘the first 10% of the database’ as the control group.
- A user must not be in both the test and control groups at the same time.
- If the audience is very diverse, you can first divide it into broad groups — for example, by country or activity — and then randomise within those groups.
- After the test starts, do not move users between groups without a rule defined in advance.
3. Choose one primary metric
The metric should relate to the business objective. Email opens are useful for diagnosis, but are rarely the final business outcome.
If we are testing… | We could measure… |
|---|---|
repeat purchases | repeat purchase rate |
order value growth | average order value / revenue per user |
retention | returning customer rate |
a discount promotion | revenue, and always profit / margin too |
4. Work out the sample size in advance
With too few people, a test may miss even a real effect. It is therefore better to calculate group sizes before launch.
First define an effect that matters to the business, then calculate the sample size.
5. Give the test enough time
Having enough users is only half the story. The test should cover a normal customer behaviour cycle. That can look very different for frequent and infrequent purchases.
- Define the minimum duration in advance.
- Where possible, include a full weekly cycle if behaviour differs by day of the week.
- Do not end a test just because today’s result looks especially good.
- Account for sales, holidays, price changes and other events that may have affected both groups.
6. Do not check for a winner every hour
If you check the result every day and stop as soon as the difference becomes ‘significant’, the chance of a wrong conclusion increases. The clearest approach for most CRM teams is to define the sample size and minimum duration in advance, wait until both have been reached, and only then make a decision.
7. Make sure the control group does not receive the same treatment
If the control group did not receive the email but received the same discount in a push notification, banner or another campaign, the comparison becomes weaker. Check overlapping communications before launch.
8. Look beyond ‘there is a difference / there is no difference’
- How large was the effect?
- Is it large enough to matter to the business?
- Did profit, unsubscribes, complaints or other important metrics get worse?
- Was there enough data to reach a confident conclusion?
