Before analyzing advertising effectiveness from viewer ratings: Can you trust viewer ratings?

The viewer rating survey is conducted by Video Research Co., Ltd. on 900 households in the Kanto region. The survey investigates whether the program was watched by the surveyed households or not, or more precisely, whether the channel on which the program is broadcast was on or not. But can a survey of only 900 households tell us anything about 180 million households? The conclusion is that we can find out with a fairly high degree of accuracy.

Do you know what "reasonably accurate" means?

This time, we will provide a theoretical explanation for those who use viewer ratings to analyze marketing and promotions, on how much trust can be placed in these figures.

For those of you who are busy, here is the conclusion about the accuracy of the viewing ratings:

If you try to accurately express how accurate the viewership is,

• Suppose the viewership rate for a certain program among surveyed households was 10.0%,
• The viewership rate for a certain program among all households is
• More than 95% chance
• Between 8.0% and 12.0%

It's a long story short, but this is the statistically accurate way to express it. Since we don't survey all households, there will inevitably be a difference from the true viewer ratings.

• The difference between the true viewership rate and the viewership rate of surveyed households
• With a very high probability (95% or more)
• Keep within a certain range

To do this, we surveyed the necessary number of households, which turned out to be 900 households.

The researcher's argument: We're studying a sample group with the same characteristics as the population!

Although we have suddenly come across a statistical term, I think that marketers have often heard the word "population" and many of them use it. In a viewer rating survey, all households in the survey area are the population, and the 900 households surveyed are the sample population. There are about 180 million households in the Kanto region, so an accurate viewer rating can be calculated by surveying all 180 million households. However, the survey is extremely costly, so it is not feasible to survey all households. Therefore, those who survey viewer ratings claim that they should survey a portion of households that are likely to have similar characteristics to all households (the population), that is, the sample population, and that "since they have similar characteristics, let's insist that the survey results of the sample population = the survey results of the population."

How accurate can you say the viewership rate is if you survey 900 households?

How accurate are the ratings? This is a difficult question to answer because by studying the ratings of a sample population,

• The true viewership rate (the viewership rate when all households are surveyed) is
• With very high probability
• Survey household viewership rate
• It falls within ±○%

That's because. In other words, the true viewer ratings and the viewer ratings of surveyed households will not be equal (there will be an error). Therefore, we will make the error smaller. There is a rare chance that the error will be large, but there is a 95% or higher probability that it will be kept to a very small error! That's what I mean.

How much error can there be?

The extent of the margin of error in the ratings is stated on the Video Research website. They are so serious that they explain the margin of error themselves.Things to note when viewing ratings | Top 10 weekly programs with high household ratingsAs you can see from the link, the amount of error varies depending on the viewership rating of the surveyed households. It would be nice if the error was about this much for a 900-household survey, and this much for a 1,200-household survey, but that's not the case. This can't be helped because the household viewership rating is included in the formula for determining the degree of error that can occur. In a 900-household survey, it seems that the viewership rating that is most likely to have a large error is 40-60%. ±3.3%. By the way, the viewership rating for the final episode of "Hanzawa Naoki" was announced to be 42.2%. That drama was broadcast in 2013, but if it had been broadcast after 900, when the 2016-household survey began, there is a 95% or higher probability that the true viewership rating would be within 42.2% ±3.3% = 38.9% to 45.5%.

How many additional households should we survey to improve accuracy?

A 95% probability of plus or minus error of 3.3% is unacceptable! I want the accuracy to be improved! For those of you who think this way, I would like to think about how many households should be surveyed. Ultimately, what we want to know is how many households should be surveyed, so let's call the number of surveyed households n. There are two ways to improve accuracy: "with a probability of a" and "with an error of ±b". These are the probability and the error range. Let's call them a and b, respectively.

For example, if n=900, then a=0.95 will result in an error of ±0.033. Those who are serious about learning statistics should read a beginner's book, but I will just present the conclusion here. For a, the values ​​0.95 or 0.99 are often used in statistics. It's a kind of convention, but we will use this for now. It means that you can choose either 95% or 99% probability that the error will fall within a certain range. Now, for the long-awaited formula, just substitute it into the formula below without thinking about it. Choose either 95 for a 1.96% probability, or 99 for a 2.58% probability.

If you want to have a 20% chance of finding the true viewership rate of a program with a 99% household viewership rate within a margin of error of ±1%, you would need to solve the formula below, and the number of households you would need to survey would be 10,651. That's a huge increase, and it would be incredibly difficult!

Summary

I think there are quite a few marketers who use viewer ratings to measure the effectiveness of their advertising, but I think it's best to approach your analysis with the understanding that viewer ratings will have a certain amount of error.

You can also use this way of thinking when you conduct your own research, so it might be a good idea to apply it and work backwards from the level of accuracy you want to achieve to decide the number of samples to use.