This series focuses on the theme of "causal inference," and aims to understand its basic concepts and explore practical approaches to verifying effectiveness. In this third article, we will introduce causal inference techniques that can be used in marketing practices. Let's explore effective methods for evaluating campaigns while understanding the ideal and reality of causal inference.

The contents of the series on this topic are as follows. Please also refer to the other articles to utilize causal inference.

• #1The Basics and Importance of Causal Inference in Marketing":
  Explaining the basics of causal inference in marketing
• The 2nd "Understanding the causal structure of marketing strategies using DAG (Directed Acyclic Graph)":
  Explains the basic principles for correctly understanding causal relationships
• Part 3: "How to use causal inference in marketing practice: Effectiveness verification through observational data analysis" (this article):
  Learn how causal inference can be applied to practical effectiveness testing

Analytical methods for utilizing causal inference from observational data

In marketing, it is not always possible to obtain experimental data. Experimental data refers to data that allows causal relationships to be clearly identified by manipulating certain conditions. However, it is difficult to conduct experiments in a strictly controlled environment for many measures, and analysis is generally performed based on data from events that have already occurred or data influenced by external factors (observational data). Therefore, a method for deriving causal relationships from observational data is important. Below, we will introduce methods that marketers can use to utilize causal inference from observational data, including examples of use and checkpoints.

Propensity Score Matching

The propensity score is a numerical index that represents the probability that a certain measure will be implemented. By utilizing this index, the backgrounds of the group that implemented the measure and the group that did not are as uniform as possible, and by matching subjects with common characteristics, the causal effect of the measure can be estimated.

Usage examples
When measuring the effectiveness of a campaign, customers who participated in the campaign and those who did not are matched based on criteria such as age and purchase history, and the increase or decrease in sales is compared. This makes it possible to determine whether the difference in sales is due to the campaign or to customer attributes.

Checkpoint

• Selection of matching variables:As in the example above, it is important to choose the right variables that can have an impact, such as customer attributes.
• Ensuring sample size:If the number of samples after matching is insufficient, the reliability of the analysis results may decrease, so it is important to collect sufficient data.

Regression discontinuity designs (RDD)

Regression discontinuity design (RDD) is a method to compare groups before and after a certain criterion (threshold). Since users with similar attributes are clustered around this boundary, if there is a difference in the results before and after the boundary, it can be determined that this is due to factors other than attributes (such as measures).

Usage examples
"When measuring the effectiveness of a coupon distribution campaign targeted at customers with a cumulative past purchase amount of 10,000 yen or more, the effectiveness of the coupon can be estimated by comparing purchasers who spend 10,000 yen (with a coupon) and purchasers who spend 10,200 yen (without a coupon), who are both on either side of the 9,800 yen boundary. This is because the cumulative purchase amounts are very close, so it is unlikely that there are any differences due to factors other than the coupon (such as age or interests).

Checkpoint

• Threshold selection:If the threshold is set arbitrarily, it could lead to bias in the subjects, so ideally it should be a natural, unmanipulated standard.
• Sufficient data density:The more data there is around the threshold, the more reliable the estimates can be.

Difference of Differences (DID)

The Difference in Differences (DID) method is a method to estimate the effectiveness of a measure by comparing the "difference in change before and after" between a group that implemented a measure and a group that did not. For example, it can measure the effectiveness of a measure by eliminating other factors, rather than simply measuring the change in sales, such as "sales increase at stores that implemented discounts - sales increase at stores that did not implement discounts = pure effect of discounts." The important thing is that the premise (parallel trend assumption) that "both groups would have changed in the same way even if the measure had not been implemented" holds true.

Usage examples
When a TV commercial is aired in the Kanto region, if sales in the Kanto region increase from 100 to 150 (+50) before and after the airing, and sales in Kansai, where the TV commercial is not aired, increase from 100 to 110 (+10) during the same period, then the net effect of the TV commercial can be estimated to be +40 (50 - 10).

Checkpoint

• Target group selection:As mentioned above, the assumption is that "even if the group that implemented the measure did not implement the measure, sales would change in the same way as the group that did not implement the measure," so it is necessary to select a group that meets this assumption based on past data, etc.
• Consider the long-term effects:It is necessary to devise ways to capture not only short-term effects but also lasting changes observed after the implementation of measures.

Causal Impact

Causal Impact is a method to estimate causal effects by statistically predicting the "hypothetical outcome (counterfactual)" in the absence of a policy and comparing it with the actual outcome. This method is used when it is not possible to find an appropriate control group, as in the difference-in-differences method.

Usage examples
Let's say that when a TV commercial is aired in Kanto, sales in the region went from 100 before the run to 180 after (+80). The difference-of-differences method compares with other regions, such as Kansai, where TV commercials are not aired, but Causal Impact uses past sales data for Kanto to predict "what would have happened to sales in Kanto if the TV commercial had not been aired." If "sales forecast without TV commercials" went from 100 before the run to 130 after (+30), the pure effect of the TV commercial can be estimated to be +50 (80-30).

Checkpoint

• Ensuring data quality:It is important to have sufficient historical data, as this depends on how accurate the predictions are if the measures are not implemented.
• Consider the comparison range:If there are sudden changes or factors that are difficult to predict (e.g. a large-scale promotion by a competitor or a sudden fluctuation such as the COVID-19 pandemic), the forecast may be inaccurate.

Steps to use causal inference

It is important to understand these techniques, but in order to use them appropriately, it is also important to grasp the overall flow. Therefore, below we will explain the specific steps to put causal inference into practice.

1. Clarify your purpose: Define which business metric you want to improve (revenue, customer acquisition cost, LTV, etc.).
2. Collect and organize data:We will centralize customer data, policy data, results data, external factor data, etc., and prepare the data infrastructure necessary for analysis.
3. Formulate the causal hypothesis:Clarify hypotheses about which measures are likely to affect which outcomes.
4. Choosing the right technique:We select the most appropriate method taking into consideration the characteristics of the policy and the business environment.
5. Conduct analysis and consider external support if necessary:Conduct the analysis with your in-house analyst and evaluate the results. If you need to improve efficiency, consider using external professional services.
6. Verification and improvement:Start with small tests and gradually scale up, revisiting your hypotheses based on the results.
7. Quantify and share your results:Show the results in concrete figures and evaluate the ROI. Share the results with the relevant parties and use them to plan your next measures and actions.

By following these steps, you can make your analysis using causal inference more effective. However, there are some points to be careful about when using causal inference. For example, if the assumptions of the analytical model are incorrect, there is a risk that the results will be biased, so care must be taken. Verify the assumptions and results from multiple perspectives to ensure objectivity, and use causal inference carefully while combining it with other data and business knowledge to verify its validity.

The limits of causal inference

Causal inference can be a powerful tool if you have the right preparation and approach, but in a realistic marketing environment, it is extremely difficult to fully implement causal inference. For more details, please see the first article in this series,The Basics and Importance of Causal Inference in Marketing", but the following challenges are listed:

1. Lack of ideal experimental conditions
In order to verify the effectiveness of a policy, intentionally implementing a policy on some people and not on others may not be acceptable from an ethical or cost perspective.

2. Difficulties in collecting data
To make causal inference, it is necessary to collect a sufficient amount of appropriate data, but in reality, there are many cases where data is missing or insufficient.

3. Diversity of confounding factors
In order to conclude that "A was the cause of B," it is necessary to eliminate the influence of other factors (confounding factors). However, in reality, confounding factors are complex and diverse, making it difficult to fully achieve causal inference.

Complementary methods: Improving the accuracy of effect measurement through MMM and path analysis

These limitations in causal inference are a major hurdle for marketers when making decisions. However, one complementary method to compensate for these constraints is to use "Marketing Mix Modeling (MMM)."

To reiterate, causal inference is a theory that clarifies whether a certain action caused a certain result, in other words, "A was the cause, and B was the result." On the other hand, MMM does not aim to prove a strict causal relationship like causal inference, but rather is an approach that uses past data to correlate and estimate the extent to which each element, such as advertising or promotion, contributed to sales.

The reason why MMM is considered to be a useful complementary method is that it can provide useful suggestions for decision-making in real business environments even without ideal experimental conditions or perfect data. In addition, its usefulness can be further enhanced by combining it with the assumptions of causal inference and the methods mentioned above (such as propensity score matching).

How can MMM help with causal inference?

MMM uses past data to clarify the relationship between measures and sales, but it does not necessarily show "cause and effect." As such, it is important to note that MMM itself does not directly prove causal relationships (causal inference), but it is possible to contribute to verifying causal relationships through the following:

1. Visualization of policy effects
MMM quantifies the correlation between measures and sales. This allows you to estimate "how much this measure contributed to sales."

2. Hypothesis Verification
MMM helps to verify hypotheses such as "increasing advertising investment increases sales" or "price changes affect sales." However, this verification requires prerequisites and assumptions, and it is necessary to carefully evaluate whether these hold as causal relationships.

3. Use in Simulations
MMM makes it possible to model measures and scenarios and simulate their effects. It can be used not only to predict outcomes but also to quantitatively evaluate the impact of measures and improve the accuracy of decision-making.

Measuring effectiveness through path analysis

Path analysis is a method of visually representing the relationship between explanatory variables and target variables in a path diagram, and analyzing the process through which measures affect sales.

By adopting this in MMM, it is possible to verify the process step by step, whether a campaign directly affects sales, or indirectly affects sales via brand name searches, brand recognition, etc., allowing for a more detailed campaign evaluation. In other words, although MMM is not causal inference in the strict sense, it can be said to be a useful method for evaluating the effectiveness of campaigns while taking into account causal relationships by incorporating path analysis and other ingenuity.

Examples of MMM using path analysis

XICA's MMM solution "MAGELLAN" uses this methodology, making it possible to perform detailed analysis of how a specific measure has affected other factors. MAGELLAN was created after more than five years of research and development, and is now used by more than 5 companies. The following is a summary of MAGELLAN's features.

Designing analytical models that can verify hypotheses

• By utilizing MAGELLAN's "Analysis Model Diagram," you can visualize your entire marketing activities according to the hypothesis you want to verify, and systematically organize the customer behavior process from awareness to purchase.
• Clarify the relationship between various elements such as measures, sales and other results, prices, and external factors (such as weather, seasonality, macroeconomic trends, and competitor trends)

Comprehensive measurement and forecasting

• Visualize not only the direct effects of marketing activities, but also their indirect (ripple) effects
• Ability to analyze short-term effects as well as medium- to long-term effects spanning several years
• Supports optimization of budget allocation according to objectives such as maximizing sales and minimizing budgets

Support from experts

• A dedicated team of data scientists and marketing consultants will accompany you
• Providing thorough support at every stage of the project (goal setting, data collection, modeling, analysis, reporting)

For more information about the MMM solution "MAGELLAN", click here

Summary

Causal inference is an important approach to gain a deeper understanding of the effectiveness of policies, but in practice, it is often difficult to fully satisfy the conditions. Therefore, rather than placing too much faith in causal inference, it is effective to use realistic methods such as MMM in combination with causal inference to verify effectiveness.

MMM itself is not causal inference, but you can take the first step in causal inference by verifying the effectiveness of your measures. To make your marketing decisions more accurate, you can use causal inference and MMM appropriately to hone your practical analytical skills.

Key points to remember

• Causal inference is useful for verifying the effectiveness of policies, but has many limitations in practice
• By utilizing MMM, it is possible to verify effects by incorporating elements of causal inference.
• Combining causal inference and MMM for more accurate decision-making

If you would like to incorporate the concept of causal inference into your marketing analysis or decision making, or if you are considering introducing MMM,Consult with our expert teamWe provide scientific approaches and solutions using data science to help you realize more effective marketing strategies.

Learn more about XICA's solutions