Start with the decision, not the method
Write the allocation question first: would shifting a defined portion of budget from channel A to channel B increase qualified customers or contribution? Specify the decision threshold and what action follows either outcome. If no budget decision would change, an expensive experiment may not be the best use of scarce traffic.
Separate platform-attributed conversions from incremental outcomes. A campaign may claim a sale that would have occurred through organic search or direct traffic anyway. A holdout estimates the difference between exposed and comparable unexposed groups, but only if the groups and measurement are sufficiently credible.
Check whether geography is a usable unit
List markets with stable historical demand, meaningful conversion volume and similar business conditions. Check that paid delivery can be restricted to treatment markets, that customers can be assigned to markets consistently and that sales operations are not changing differently across them. Cross-border shoppers, commuting and national brand exposure can contaminate the comparison.
A local service business with two cities may have too little volume to distinguish a modest advertising effect from ordinary week-to-week swings. A retailer with many comparable delivery regions might have a better design. Do not assume a test is feasible because an ad interface allows geographic exclusions.
Estimate the signal before launching
Use historical weekly customers or qualified leads per market, natural variability, conversion lag and the expected effect size. If the business averages three customers per market per week, a 10 percent lift is a fraction of a customer and unlikely to be measurable over a short test. Calculate the minimum detectable effect with statistical support where possible.
Choose an outcome that matters and occurs often enough. Qualified leads may be a practical leading measure when sales close slowly, but they are not interchangeable with customers. Document the trade-off and avoid retroactively selecting whichever metric happens to look most favorable.
Design the control and treatment fairly
Match or randomize comparable markets, preserve the same landing pages, sales staffing and promotional calendar where feasible, and predefine start and end dates. If the treatment is incremental ad spend, hold other campaigns as steady as practical. If the treatment is pausing ads, recognize that customer demand may shift across channels.
Record all planned exclusions and any exceptions. A nationwide email promotion or sudden competitor closure in one market can change the result. Monitor delivery leakage, actual spend by geography and organic brand search so a nominal holdout does not quietly receive the treatment.
Analyze lift with uncertainty and limitations
Compare the treatment-versus-control change relative to a pre-period baseline, ideally with a method appropriate to the data. Report incremental outcome estimate, uncertainty range, test cost, possible contamination and whether the effect clears the predeclared business threshold. A positive point estimate with a wide interval crossing zero is inconclusive, not a victory.
Do not extrapolate a small market test to every market without considering demand and competitive differences. If conversions mature over weeks, wait for the outcome window. Document what the experiment can and cannot establish, especially when channel attribution is incomplete.
Use a simpler alternative when the test is underpowered
For very small budgets, prioritize instrumented landing-page experiments, structured before-and-after analysis with explicit caveats, sales-quality reconciliation or staged budget changes. These can guide decisions without pretending to prove causal incrementality. Keep a learning log and avoid changing several variables at once.
Connect this to the attribution-versus-incrementality and experiment-backlog guides. The expert decision is sometimes to defer a geo holdout until volume is sufficient. Measurement credibility is more valuable than a sophisticated-looking chart built on too few customers.