Correctly evaluating the effectiveness of marketing is essential for the growth of a company. However, simply observing that "sales increased after advertising" does not allow one to determine whether advertising was truly the cause of the increase in sales. This is because fluctuations in sales can be influenced by many factors other than marketing activities such as advertising, such as seasonality and competitor actions. Causal inference is a method for correctly determining this causal relationship.

In this article, we will explain the basic concepts of causal inference that marketers should understand and introduce practical approaches to verifying effectiveness. On the other hand, although causal inference is theoretically important, it is not easy to prove a complete causal relationship in practice. This article also explains the limitations.

first

This article is the first in a series that will introduce a way of thinking to understand the basics of causal inference in marketing and apply it in practice.

The contents of this theme are as follows:

• Part 1: "The Basics and Importance of Causal Inference in Marketing" (this article):
  Explaining the basics of causal inference in marketing
• The 2nd "Understanding the causal structure of marketing strategies using DAG (Directed Acyclic Graph)":
  Explains the basic principles for correctly understanding causal relationships
• The 3nd "How to use causal inference in marketing practice: Evidence from observational data analysis":
  Learn how causal inference can be applied to practical effectiveness testing

While it is realistically difficult to fully satisfy the strict conditions of causal inference, by understanding the concept through this series and using it appropriately, you can hone your perspective to take your marketing decision-making one step further.

What is causal inference for correctly evaluating the effectiveness of marketing?

Causal inference is a theory for determining whether a certain action caused a certain result, that is, clarifying the "causal relationship." In marketing, it can be used as a framework for accurately understanding how a measure affected sales or customer behavior. This reduces the risk of overestimating the effectiveness of a measure or deploying the wrong measure.

The main purpose of causal inference is to distinguish between "correlation" and "causation." By determining whether the superficial associations shown in the data are in fact cause and effect relationships, it becomes possible to evaluate and improve policies more accurately and effectively.

That being said, perfect causal inference is extremely difficult to achieve in the real world, but that doesn't mean it has no value in marketing - we'll explore why in the next chapter.

The Ideal and Reality of Causal Inference in Marketing

To what extent is causal inference possible given the constraints of reality?

It is said that it is difficult to fully realize causal inference in marketing. The following are some practical challenges that hinder causal inference:

1. Lack of ideal experimental conditions
To perform precise causal inference, it is ideal to use a randomized controlled trial (RCT), a method commonly known as A/B testing. However, in the field of marketing, it is difficult to randomly assign all customers, and there are ethical and cost constraints. In addition, since customer behavior and market conditions are constantly changing, it is not easy to ensure stable comparison conditions.

 2. Difficulties in collecting data
Although various statistical methods have been proposed to solve the above problems, it can be difficult to obtain the data necessary to utilize them. For example, it is often difficult to obtain customer behavior logs for offline advertising, and data such as advertising delivery and sales data is often incomplete due to missing records or errors.

3. Diversity of confounding factors
For example, when measuring the effectiveness of a particular measure, it is nearly impossible to completely eliminate the influence of other measures or external factors that occur at the same time. As we will explain later, these are called "confounding factors (external factors that affect both explanatory variables and target variables)," and in marketing, because confounding factors are complex and diverse, it is said to be difficult to fully realize causal inference.

Practical benefits of causal inference for marketers

Even if causal inference is not fully feasible, marketers can benefit greatly from understanding and using the concept. Here are three practical benefits that causal inference can bring to marketers:

1. Improving hypothesis formation and verification skills
Knowledge of causal inference will enable you to formulate more specific and rational hypotheses for your measures. It will also enable you to select the necessary data and analysis methods appropriately in the hypothesis verification process, enabling you to obtain more reliable results. For example, when considering "how new creative will affect purchase intent," you will be able to recognize the importance of setting an appropriate control group (described below).

2. Improving the accuracy of policy evaluation
By taking a perspective that makes use of the basics of causal inference, you can more accurately evaluate measures. For example, by adopting an analysis method that takes confounding factors into account, rather than relying on before-and-after comparisons or simple correlation analysis, you can be more confident in identifying "measures that were truly effective."

3. Effective collaboration with data analytics experts
Understanding the basics of causal inference will make communication with data analysis teams and external consulting companies smoother. For example, you will be able to ask more constructive and specific questions and make more suggestions when discussing "what data is needed to measure the effectiveness of this measure" and "what causal relationships can be verified."

Basic concepts of causal inference that you should understand in marketing

The difference between causation and correlation

To understand causal inference, it is important to understand the difference between "causation" and "correlation." For example, if we take two elements, A and B, correlation is a state in which A and B are related to each other. Causation is a state in which B occurs because A is the cause.

For example, if there is a correlation between the number of clicks on a retargeting ad and sales, it is dangerous to jump to the conclusion that "retargeting ads increased sales." In fact, if the ad delivery target was a "group that already had a high purchasing intent," they might have made a purchase even if they had not been exposed to the retargeting ad. In this case, the "original purchasing intent of the ad delivery target" is a confounding factor, and not taking this confounding factor into account in causal inference will lead to an erroneous evaluation.

In order to correctly distinguish between correlation and causation, it is essential to understand the causal structure behind the data. If you would like to know more, please see the second article in this series, "Understanding the causal structure of marketing strategies using DAG (Directed Acyclic Graph)Please take a look at ".

Confounding factors and selection bias

Confounding factors are external variables that affect both the policy (explanatory variables) and the business outcome (target variable). Identifying and controlling for confounding factors is important to understand causal relationships.

For example, an increase in sales of a certain product does not necessarily mean that the marketing of that product was successful. Seasonality or competitors' actions may have affected the sales. In fact, there have been cases where an increase in sales of a certain toy was judged to be the result of a TV commercial, but in fact, the commercial was broadcast during the summer vacation, so the "summer vacation" affected both the effect of the TV commercial and the sales. In this way, if confounding factors are overlooked, there is a risk of overestimating the effectiveness of the measures.

Examples of common confounding factors in marketing

• Seasonal factors (year-end and New Year holidays, peak demand periods, etc.)
• External environment (competition, economic indicators, etc.)
• Customer attributes (age, purchase history, etc.)

Selection bias occurs when participants or data are not selected randomly. For example, a company may determine that a newsletter with a coupon is effective because the purchase rate increased after the coupon was sent, but in reality, the newsletter was sent preferentially to customers with a purchase history or active users, so it may contain many customers who would likely purchase even without the coupon. In this way, it is necessary to minimize this bias when designing experiments and data analysis.

Randomized controlled trials (RCTs) and observational studies

A randomized controlled trial (RCT) is the gold standard method for verifying causal effects, and is commonly known as an A/B test. By randomly assigning a treatment group (a group that implements marketing measures) and a control group (a group that does not implement marketing measures), it is possible to estimate causal effects after removing the influence of confounding factors and selection bias.

However, as mentioned above, in the field of marketing, it is difficult to randomly assign all customers, and there are ethical and cost constraints.In addition, since customer behavior and market conditions are constantly changing, it is not easy to ensure stable comparison conditions.

Observational studies are other ways to make causal inferences from observational data when randomized controlled trials cannot be conducted. They use methods such as difference in difference (DID), propensity score matching, regression discontinuity design (RDD), and regression analysis. For more information on these, see Part 3 of the series,How to use causal inference in marketing practice: Evidence from observational data analysis", but here we will briefly introduce the method of finite differences.

The difference-of-differences method is a method to estimate the pure effect of a measure by comparing the "difference in change before and after" between a group that implemented the measure and a group that did not. For example, "Sales increase at stores that implemented the discount - Sales increase at stores that did not implement the discount = Pure effect of the discount" - this allows you to measure the effect after eliminating other factors, rather than simply looking at changes in sales. However, the difference-of-differences method makes the assumption that "if the group that implemented the measure did not implement it, sales would change in the same way as the group that did not implement the measure." In reality, it is difficult to find a group that did not implement the measure that fits this assumption, so it is practical to first verify the effect to the extent possible.

Three common mistakes marketers make: Things to watch out for from the perspective of causal inference

Understanding the basic concepts of causal inference should enable a more accurate evaluation of the effectiveness of policies; however, in reality, this theory is not fully utilized in the field, and people often end up making the following mistakes:

1. The "before and after" comparison trap

One mistake many marketers make is judging the effectiveness of a campaign by simply comparing before and after.

Problem: Ignoring confounding factors
For example, even if a winter coat sales campaign is conducted in December and sales increase compared to August, it is possible that the impact was not actually due to the campaign, but rather due to external factors such as "dropping temperatures" or "end-of-year bonus period." In this way, before-and-after comparisons that simply compare figures before and after the implementation of a campaign often do not take confounding factors into sufficient consideration.

Measures based on causal inference
Causal effects can be estimated more accurately by using methods that control for factors other than the policy, such as the difference-in-differences method mentioned above.

2. The "average effect" can be deceiving

Evaluating the effectiveness of a campaign based solely on the overall average carries the risk of missing unexpected adverse effects or different responses among some customers (individual segments).

The problem: Ignoring differences between segments
For example, if a 20% off coupon for luxury apparel is distributed to all customers, the coupon will have a strong effect on new customers, but on the other hand, the coupon will create "expectation of price reduction (a psychological state of waiting for another coupon to be released soon)" for existing customers, delaying purchases. As a result, even if the average purchase amount appears to have increased, there are cases where a certain customer group is actually attriting.

Measures based on causal inference
By utilizing techniques such as propensity score matching (a statistical method of pairing customers with similar characteristics and comparing the effects; more details are provided in the third article), you can capture causal relationships for each segment and understand the actual situation that cannot be seen by average values ​​alone.

3. The risks of "leaving it to data scientists"

If you leave the analysis to data scientists without first taking into account the practical context and intent of marketing, the data may not be interpreted correctly.

The problem: analytics that don’t directly tie to business goals
For example, a marketer may hire a data scientist to run an analysis to evaluate the effectiveness of a new digital advertising campaign, only to find that the results showed no statistically significant improvement in ad clicks or website visits.

However, what marketers really wanted to know was how the campaign contributed to business outcomes such as the purchasing behavior of loyal customers, "conversion rate" and "sales." Analyzing only the "number of clicks" and "number of visitors" does not allow for meaningful causal relationships in practice, and there is a risk that the effectiveness of the campaign will not be properly evaluated.

Measures based on causal inference
To prevent such discrepancies, it is important for marketers themselves to clearly define the purpose of the analysis and business prerequisites, and set KPIs (e.g., conversion rate and sales for a specific target). In addition, it is necessary to regularly coordinate with data scientists to confirm that the design of models based on the concept of causal inference and the interpretation of results are in line with the intention of business.

Summary

Causal inference is a powerful method for correctly evaluating the effectiveness of marketing, but it must be applied with caution. First, it is practical to carry out effectiveness verification to the extent possible, and in that case, it is important to be able to use the data to make better decisions without getting too hung up on causality.

#2Understanding the causal structure of marketing strategies using DAG (Directed Acyclic Graph)" explains how to use DAG to visualize causal structures and organize hypotheses. Please read it to gain more theoretical knowledge.