Why do supposedly data-driven initiatives fail? Uncovering "customer truths" with behavioral economics

Does data make all marketing decisions rational?

Many marketers have been conducting data analysis with such expectations. However, the reality is not so simple. In the real world, you are likely to be faced with "irrational" phenomena on a daily basis that you cannot explain, asking "Why is this happening?" For example,

• Free Magic Power: "Just writing 'Free Shipping' can skyrocket your conversion rate..." Why does user reaction change so dramatically when shipping costs are included in the product price, even though the total amount remains the same?
• The paradox of choice: "Even though we've expanded our product lineup, the number of purchases is decreasing..." Logically, the more options there are, the higher customer satisfaction should be, but in reality, the opposite is often the case.
• The gap between favorability and purchase"Brand surveys show high favorability, but no purchases are made..." This phenomenon of a mismatch between emotions and actions has troubled many marketers for many years.

"Why did that meticulously designed initiative fail?" "Why did the creative that was supposed to be superior in the A/B test not produce the expected results in the actual run?" These are questions that I think all marketers who deal with data have trouble with. Furthermore, not being able to explain the "why" makes it difficult to make repeatable decisions and is a major obstacle that hinders the organizational growth of marketing departments.

Data analysis is essential as a means of "visualizing" customer behavior, but it only shows the results of "what happened." Data alone cannot tell us the background or context of "why it happened."

What's needed here is a perspective that can help us understand how people make decisions. Behavioral economics is a field that has tackled this very question head-on. In this article, we'll explain, along with specific analytical methods and examples, what clues the perspective of behavioral economics can provide for phenomena that cannot be explained by data alone.

Behavioral economics reveals "human-like" decision-making

People make decisions in a "human" way, not rationally

Traditional economics is based on the premise of "homo economicus," which states that people always make rational decisions and choose the option that provides the most value to them. However, in reality, people make what might be called "human" but irrational decisions every day, such as being sensitive to avoiding losses or choosing something that is labeled "No. 1 in popularity."

Behavioral economics focuses on these real-life patterns of human behavior and studies them systematically.Daniel Kahneman,Richard ThalerBehavioral economics, which has developed through this, has revealed many of the laws underlying such "human-like judgment."

When people buy something, there is not always a logical explanation for their decision. It is not uncommon for them to be driven by emotions or intuition that are difficult to put into words, such as "because it felt good" or "because it somehow seemed attractive." There are several typical examples of such psychological tendencies that are unique to humans.

• Ambiguity effect : Avoiding options with high uncertainty, such as a lack of information, and preferring more certain options
• Scarcity heuristic : Scarcity and urgency drive action
• Status quo bias : There is an unconscious resistance to changing the status quo.
• AnkarinGu effect : The information presented first strongly influences subsequent decisions
• Decision fatigue : Too many options lead to decision avoidance
• bandwagon effect : The fact that "other people are doing it" motivates action

Three reasons why "liking" something and "buying" it don't match

Why does high favorability not result in a purchase? One possible reason is that emotions and behavior are not necessarily directly linked.

• Time axis deviation:Even if you like something now, you may not act on it when the time comes to buy it later. This is because emotions are fleeting, but purchasing is often a planned act.
• Psychological costs:Even if you like something, the hassle and anxiety it takes can put a damper on your purchase. Psychological barriers such as "cumbersome membership registration," "complex payment," and "anxiety about returns and exchanges" inhibit behavior.
• Comparative effect:What may be attractive in isolation may appear inferior when compared to other options, and how it is evaluated relative to competitors and alternatives will influence the final purchase decision.

As you can see, there is a gap between "likes" and "impressions" and actual purchasing behavior. So how should we understand and analyze this gap?

The power of behavioral economics to understand human nature

This is where the perspective of behavioral economics, which delves deeply into the relationship between emotions and behavior, can be useful. Data shows us the "result" of a purchase. However, what we really want to know is the "reason" behind the behavior: "Why did it happen?" Using the perspective of behavioral economics makes it possible to clarify the background to behavior that cannot be explained by data alone. For example, in cases like the one below, we can understand the human psychology behind mere data, which can improve the accuracy of our marketing hypotheses and interpretations.

• The measures were not as effective as expected: Expressions that emphasized loss, such as "to disappear" and "to lose," may have caused psychological resistance.
• Even if the actual conditions are the same, reactions can vary: Differences in wording and presentation, such as "20% OFF" and "20% OFF for a limited time," or the order in which products are presented or a "Most Popular" label, can make a big difference in purchasing intent.

This perspective from behavioral economics can be applied to a variety of marketing analysis contexts. In other words, behavioral economics is extremely effective as a lens that adds a "human touch" to data.

Practical Guide: Applying Behavioral Economics to Marketing Analysis

In order to apply the perspective of behavioral economics to actual marketing analysis, it is necessary to find ways to quantify these psychological factors and make them treatable as data. This conversion process is a key element in connecting behavioral economics and data science.

4 steps to quantify psychological factors

Step 1: Translating abstract psychology into concrete behavior 

First, we convert abstract and difficult-to-understand psychological characteristics and tendencies into concrete customer behavior. For example, if the psychology of "wanting to avoid losses" (loss aversion bias) is used, we can observe behavior by saying, "Purchasing behavior is affected as the discount expiration approaches." If the psychology of "feeling reassured when other people are buying it" (bandwagon effect) is used, we can replace it with, "The more reviews a product has, the more it affects conversion rates."

Step 2: Convert into a measurable variable 

These behaviors can then be converted into measurable numbers, specifically defined as variables that can be analyzed, such as:

• Scarcity heuristic → Number of times "in stock" or "limited quantity" is displayed
• Anchoring effect → The difference between the "list price" and the "actual purchase price"
• Decision fatigue → "Number of products displayed" and "Number of times the filter function is used"

Step 3: Get the real data 

The numerical values ​​defined above are extracted from actual data. For example, the necessary information for analysis is gathered based on actual user behavior, such as "product comparison behavior patterns" from website log data and "price range trends" from purchase history.

Step 4: Test your hypotheses (e.g., multiple regression analysis) 

Finally, we will use the variables we have prepared to see how psychological factors affect actual purchasing behavior.

One way to do this is with a statistical approach such as multiple regression analysis. For example, if you have a hypothesis that "if too many products are displayed, the purchase rate will actually decrease," you can verify the magnitude and direction of the effect by expressing the relationship between the number of products displayed and the purchase completion rate in a mathematical formula.

This equation is called a "regression equation." A regression equation is a model for numerically capturing the factors that affect outcomes such as purchase rate. Below, we will explain the model equation and its meaning when using purchase rate as an example outcome.

Purchase rate = β0 + β1 × number of products displayed + β2 × number of comparison actions + β3 × length of stay + ε

• β0 (base purchase rate):Even if all other factors (such as the number of products displayed and the length of time spent on the site) were zero, this represents the starting point that a basic purchase rate can be expected, and in other words, the "ease of purchasing in its raw state."
• β1, β2, β3 (strength of influence of each factor):It specifically shows the "strength of influence" of each factor, i.e., how much it increases (positive value) or decreases (negative value) the purchase rate.
  • For example, if the value of β1 (number of products displayed) is negative, this is evidence supporting the hypothesis that "displaying too many products makes them less likely to be selected."
  • If the value of β3 (time spent) is positive, it clearly shows the relationship that "the longer a user stays on the site, the more likely they are to make a purchase."
• ε (unexplained "error"):This refers to the "human whims" that models cannot capture. For example, you might receive a sudden phone call while you were thinking about making a purchase, your mood might change, or you might suddenly be drawn to a different advertisement. These countless coincidences often influence your final behavior. This ε gives us a realistic perspective that our analysis is never perfect.

While this may seem difficult at first glance, it can actually be verified without specialized statistical software using Excel's "Data Analysis" function. For example, if the result obtained in Excel shows a "coefficient for the number of products displayed" of -0.02, this means that "each additional product displayed reduces the purchase rate by 2 points." Similarly, if the "coefficient for the number of comparisons" is -0.15, this means that "each additional comparison reduces the purchase rate by 15 points." This can provide suggestions that lead to specific improvement actions, such as "narrowing the number of products from 20 to 15 could potentially improve the purchase rate by approximately 10 percentage points."

The important thing is not to simply look at correlations (connections between numbers), but to judge the validity of the interpretation by looking at the psychology and behavioral mechanisms behind them. Also, this suggestion goes beyond simple UI improvements; based on the insight that "customers are looking for choices they will not regret rather than optimal choices," it has the potential to influence the entire way of thinking about marketing, including product strategies and sales floor layouts.

Please see the following document for detailed instructions on how to implement the program and how to interpret the results.

▶ Related materials:A guide to multiple regression analysis in Excel that empowers marketers

Two approaches to designing customer decision-making

Here we introduce two approaches to designing customer decision-making, applying the insights of behavioral economics to all of your marketing activities.

Design of meaning, selection, and memory

Behavioral economics is also effective in designing policies, products, and services. The following three perspectives are particularly useful as "thinking patterns" that marketers can use.

• Design of meaning:It gives customers the context of "Why do they need this?" By giving the product a social significance or an element that they can relate to, it clarifies the reason for their actions. For example, by presenting an eco-friendly product in a personal context, such as "for your children's future" rather than "for the Earth," you can strengthen the motivation to purchase.
• Design of choice:We make it easy to choose. By designing the number of options, sorting order, and comparisons (such as gold, silver, and bronze), we can naturally guide customers to make the desired choice. Displaying "recommended" options and setting price ranges are also part of the selection design.
• Memory design:How you design the emotional peaks and end of an experience will influence how people perceive your brand. Creating emotional peaks in follow-up experiences after membership registration or purchase will help you build long-term relationships.

In this way, by incorporating the three design perspectives of "meaning, selection, and memory," behavioral economics can evolve marketing activities into more human and effective ones.

Nudge theory: Practicing natural behavioral guidance

"Nudge" (English: nudge)" means to give a gentle nudge, and was coined by Professor Richard Thaler (2017 Nobel Prize in EconomicsNudge theory, proposed by Robert G. Beck and others, is known as a practical approach in behavioral economics that aims to encourage people to act in a natural and desirable direction without taking away their freedom. In marketing, it can be used in the following cases:

• When you're not sure what to choose, if there's a label that says "Everyone's choosing this!", you'll naturally want to choose it.
• By placing the purchase button in a position, color, and size that is easy to see, you can naturally encourage the user to take the "final step."
• By deliberately listing overpriced menu items, the middle price range appears "just right."
• If the "Receive email notifications" option is checked in the initial settings, many users will proceed.
• Place signs saying "This item is often purchased together with this product" near rice balls, sandwiches, etc., and place drinks and soups nearby.
• Adding messages like "Only 3 left" or "Popular colors almost sold out" to products with limited sizes or stock

Nudges are not coercive, but rather a method of providing natural encouragement while respecting the customer's autonomous decision-making. This allows for effective marketing that takes into account the human behavior of customers. However, it is important to note that excessive guidance or designs that substantially narrow the customer's options run the risk of damaging trust.

Summary: Marketing that combines data and human understanding

Behavioral economics is a field of study that focuses on factors that are difficult to see in data, such as emotions, context, and memory. Even with the widespread use of AI and advanced analytical tools, the ability to interpret "why people behave in a certain way" remains up to marketers, and a perspective that addresses "humanity" will become increasingly important in future marketing.

However, understanding people alone is not enough to unravel these complex customer psychology and lead to repeatable decision-making and business growth. The power of science, in the form of precise data analysis and data science, is essential.

XICA combines data science and consulting, supporting corporate marketing decision-making based on its knowledge of statistical analysis and modeling techniques that it has cultivated over more than 10 years, and its practical analytical design know-how gained through working with over 280 companies. What we value is not analysis completed by data scientists alone, but a collaborative process in which we maximize and refine the "hypothesis power" of marketers, backed by a deep understanding of the market and customer insights.

What is required for future marketing is a balance between "a perspective that understands people" and "the ability to utilize data." XICA works closely with clients to dig deep into hypotheses, build optimal analytical models, and implement "growth engines" that will have a definite impact on their businesses. The power of data and understanding people, combined, are the keys to truly moving customers and creating an organization that "continues to win."

▶ Related articles:Introduction to hypothesis thinking for marketers: The basics and practice for using it in practice and getting results