A Marketer’s Guide to Geo-Based Incrementality Testing

What is Geo-Based Incrementality Testing?

Geo-based incrementality testing is a method of experimentation that splits an audience by geographic location. Based on this approach, ads are served to select regions (the test) while holding out others (the control), allowing marketers to measure the true impact of their advertising efforts. This method is flexible enough to measure lifts to websites, Amazon, brick-and-mortar stores, and other sales channels, assuming sales can be reported by geographic region. Geo testing works across any media with controlled impression delivery, though some platforms are more flexible than others.

Tatari has employed several successful geo-based incrementality tests for our clients. Here’s a breakdown of how it works, and how to get started.

What Can a Geo Test Solve?

The most important consideration when setting up any kind of test is asking: “What is it that we hope to answer through testing?” For most marketers, the objectives fall into two main categories:

1. Measuring Lift: Does this marketing channel drive incremental sales?

Partners often frame this in statistical terms like "achieving 95% significance with 85% power over X weeks at Y budget." While important, many marketers already suspect a channel provides some lift and are more interested in the next question.

2. Understanding Efficiency: How well does this channel perform in terms of efficiency? 

Marketers typically want actionable metrics like CPA and ROAS to optimize their spending. Geo tests designed to address the root of this are usually more involved than those just looking for lift. While any geo test can technically provide these metrics, their accuracy heavily relies on a well-designed and executed test, similar to other measurement methods like A/B tests or marketing mix modeling.

What Type of Data is Required?

Geo tests require your first-party data (e.g., D2C online sales, app installs, Amazon sales, brick-and-mortar retail). These data points should include DMA-level sales reported at least weekly. Providing business context such as seasonal trends, unusual activity dates, and upcoming changes is also beneficial. 

Reporting Construction: Selecting Test and Control Regions

Determining regions to be used in the test and control cells is typically based on a specific client's historic data and the regional sales trends. It is commonplace in geo tests to identify groups of correlated regions to ensure the most accurate measurement. To measure efficiency in a geo test, use one region (Control) to predict the performance of others (Test). This prediction's accuracy depends on:

• Precision of the Control Group: How good is the Control group at matching the Test group's performance? Since the control group is an estimation of the test region’s performance in the absence of advertising, the precision of that estimation is vital for test setup.

• Incremental Sales Volume: How much sales increase (lift) do we expect based on our budget, CPA, or ROAS? An example of this is “with a $10,000 budget and a $100 CPA, we expect 100 units of lift.”

• Potential Measurement Inaccuracy: Will the expected lift be large enough to be reliably measured against the control group’s historic variability?

For example, if monthly sales in the test area are 10,000 units with a 5% margin of error (+/- 500 units), the predicted lift needs to be larger than 500 units to be meaningful. A $50,000 budget with a $100 CPA suggests a 500-unit lift, which could be masked by the model's error.

These factors come together and directly impact the expected Signal:Noise ratio. You ultimately want to run a test that confidently produces results that aren't due to noise but reflect your planned marketing budget.

Other considerations when selecting test regions:

• Product or demographic skew 

• Regional marketing activations

Any major business changes (e.g., pricing, product, etc.)

How Tatari Measures Geo Test Results

Results are measured as the aggregate of your test performance minus the control performance. The control represents the prediction of how your test regions would have performed if there had been no marketing. Because of this, we aren’t able to provide granular network/creative level performance, but we can measure the campaign as a whole. Below are some examples:

This chart represents a time series showing weekly sales of both the test and control. The green box highlights the testing period. We see that once we begin running TV, the test and control diverge and we’re able to see lift week over week.

Pros and Cons of a Geo Test

If you're a marketer weighing how to measure TV impact, geo testing offers unique advantages, but like any approach, it comes with pros and cons worth considering.

Pros:

• No PII (personally identifiable information) required

• Can test any media (TV, digital, direct mail, etc.)

• Can validate drag factor

• Works alongside Tatari’s standard measurement methodologies to triangulate performance

• Advertisers can independently verify the results using their own sales data by geo without the need for 3rd party vendor data.

• Can be used to measure offline and omni-channel performance

Cons:

• Can be quite noisy depending on baseline

• Can only see overall lift by test region (not broken out by creative, time of day, or medium)

• Less effective for long-consideration purchases/conversions (i,e., bank account opening, car purchase, other low conversion rate events). Long-consideration purchases/conversions would necessitate tests that are impractically long and/or expensive to operate.

• Local campaigns aren’t always apples-to-apples with national (e.g., higher saturation/frequency, less flexibility on local optimization, constraints on some linear channels).Test regions may not be indicative of national averages

• Less efficient than a typical campaign due to local targeting CPMs and high saturation (i.e., higher frequency), which is a byproduct of putting enough spend in market to ensure incrementality. These CPMs and higher frequencies differ substantially from how an evergreen campaign would run.

• Doesn’t measure long-term brand/halo effects

Time to set up the test (logistically + mechanically due to cool downs post test, etc.). Time during the test = opportunity cost in terms of sales/efficiency that could have been had from a national/more efficiently operated campaign

Geo testing isn’t one-size-fits-all and may not be applicable to every campaign, but when carefully designed and executed, it gives marketers a reliable, real-world view into how media spend drives incremental sales. That’s why we make sure to partner with measurement experts like LiftLab and Measured, just to name a few.

Want to run a geo test on your TV campaign? Let’s talk.