One of my favourite concepts is Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.” Scott Garrabrant on LessWrong came up with a framework for four different ways proxy measures can fail when you are optimizing for them. To reinforce my learning I wanted to try finding examples in digital marketing and website optimization that apply to the other categories.

“Adversarial Goodhart – When you optimize for a proxy, you provide an incentive for adversaries to correlate their goal with your proxy, thus destroying the correlation with your goal.”

We observe that online sales are correlated with display campaigns being the last touch before conversion (that is, under a ‘last touch attribution’ model). Our proxy for success is increased last touch attribution clicks for our display campaigns, but the true goal is increased online sales profit.

A competing display agency also serving ads for the company has the goal of increasing the amount of money that the company is spending on their ads. This is opposed to the true goal of the company as the more the company spends on ads, the more costs their company has (unless of course, those ads have an amazing ROI). The competing agency, knowing of the proxy of last touch attribution, thus tries to get their ads to show up right before purchase by primarily doing retargeting ads and swamping all users with ads so that they are the last touch before as many sales as possible.

I wrote more on this problem here.

“Extremal Goodhart – Worlds in which the proxy takes an extreme value may be very different from the ordinary worlds in which the correlation between the proxy and the goal was observed.”

We have correlated paid search advertising spend with online sales: The more we spend in a month on paid search, the more profit our company sees (which is of course our true goal). Our monthly spend historically has ranged from $25K-$100K but is usually around $75K.

The CMO sees this data point at a company presentation and is so excited that they free up $1M to spend on paid search advertising. This is more than we ever expected to see our company have in paid search budget! But, according to our model, this is great: we should see so much more profit!

So, we take this money and spend it. Unfortunately, we do not see an amazing increase in profit because of this! Why?

The world where our proxy value took on an extreme value is very different than the world where the proxy and goal was originally observed. The paid search keywords being bought with the additional spend are probably very different than the keywords under which we observed success, as the keywords where we saw success are likely to be saturated. So now there is probably spend on keywords completely unrelated to what our company sells, so users are not clicking and purchasing from our website at as high rates.

Here, what I’m talking about is basically the law of diminishing returns.

“Causal Goodhart – When there is a non-causal correlation between the proxy and the goal, intervening on the proxy may fail to intervene on the goal.”

Every day we observe a spike in visits to the website and subsequent sales around 9:30 PM. So, we conclude, more users on the site at exactly 9:30PM (our proxy) causes sales (our goal).

We maximize users on the site at exactly 9:30PM by blocking users from accessing the site and displaying a popup saying “Come back at 9:30pm!” We blanket the internet with ads for our company that only display around 9:30 PM. This will of course increase sales because 9:30 PM visits cause sales! We expect our company to rake in additional millions!

But… It turns out that we are getting so many users (and sales) on our site at precisely 9:30PM every day because someone in the TV advertising department (who we of course never talk to because large companies are frequently siloed) has purchased ads for a popular news program and is offering a promo code for users who ‘act fast’ and buy within the next 15 minutes.

In this case, there was a non-causal connection between our proxy and our true goal. There was a distinct other event (the TV promotion) that was causing both our proxy and our true goal to increase.

“Regressional Goodhart – When selecting for a proxy measure, you select not only for the true goal, but also for the difference between the proxy and the goal.”

Clicks are correlated with sales conversions. Therefore, to optimize for your goal of getting users with the highest conversion interest to the site, you decide to invest in campaigns that get the highest clicks and divest from campaigns that have lower clicks. Here, your proxy is Clicks and your true goal is Converting Visitors.

However, as many an individual who has encountered a click-baity title can tell you, what makes someone click on something isn’t just conversion interest. It is also entertainment or shock factor interest. Here we are not just selecting for our true goal, but the difference between the proxy and the goal where the difference is entertainment (or shock value, or some other unthought of factor).

Conclusion

Some of the examples might feel a little ridiculous, but I think are things that could happen without trying to mitigate the potential for this occurring for your proxy measures.

  • Choose proxies that are the most tightly aligned with your goals
  • Assess the true causal nature for your proxy (or other events)
  • Have better company communication to ensure you don’t miss some other potential causal factor
  • Don’t fall victim to extrapolation

There are probably many other ways of optimizing for proxies can cause failure: Can you think of other types of failure? Or, can you think of other examples of each type of failure in digital marketing?