4.6.5.1 - How to Design a Simple Holdout Test (Difficulty: Advanced | Path: Scale)

Lesson Summary

How to Design a Simple Holdout Test (Advanced)

What is it?

A holdout test is the most basic and powerful way to measure ad effectiveness. You split your *entire* audience (e.g., all of the USA) into two random groups: a Test Group (e.g., 90% of users) who *will* see your ads, and a Control Group (e.g., 10% of users) who are *excluded* from seeing your ads.

Why is it important?

After running the test, you compare the total sales from the test group to the total sales from the control group. If the test group had a 5% purchase rate and the control group had a 3% purchase rate, the 2% difference is the *true lift* or 'incrementality' caused by your ads. This proves your ads are finding new customers, not just targeting people who would have bought anyway.

How to Set It Up (Simplified):

This is easiest to do with retargeting or email list audiences.

1. Take your audience (e.g., your 10,000-person email list).
2. Randomly split it into two new audiences: '90% - Test' (9,000 people) and '10% - Holdout' (1,000 people).
3. Upload both lists to your ad platform.
4. Run your ad campaign, targeting the '90% - Test' list.
5. Crucially, EXCLUDE the '10% - Holdout' list from seeing *all* ads from your account.
6. Run the test for 2-4 weeks.
7. Compare the total sales from the 9,000-person group to the sales from the 1,000-person group (you'll need to scale the holdout group's sales by 9x to compare them fairly). The difference is your lift.

Common Misconception

'An A/B test is a lift test.' False. An A/B test (e.g., Ad A vs. Ad B) tells you which ad is *better* at getting clicks. A holdout test (Ads vs. No Ads) tells you if *advertising itself* is profitable.