Research or Ratings SAMPLE is EVERYTHING!!

Recently we've seen lots of discussion over samples in ratings. Much of it relates to Arbitron rolling out the PPM and we've seen some industry concern over the sample both in size, return and balance for ethnics and demos. No doubt we'll see more confusion on the sample with PPM rolling out because PPM is really a different type of research than we have traditionally used for ratings. With the diary we are looking at a snapshot of one weeks listening as recalled in a diary vs the behavior of the respondent over weeks and weeks with no 'recall' necessary. If you go to our reports in McVaymedia.com you can get a lot more information and analysis on the differences between PPM and Diary methods here.

Before you work with any research or ratings data the most important analysis tip we can pass on is LOOK AT THE SAMPLE. How far off of the population estimates is the return? You may also want to look at how the sample returned in zip codes or key ethnic areas. Know who they got of hold of. Of course Arbitron does reveal the numbers, it may be a little harder to find on the Max or Arbitrends menus but it's there and you should look at it before you celebrate or wonder what happened. The same goes for any perceptual or music testing you do.

In order to get realistic results Arbitron and many other studies balance the sample by weighting. This places more value on demos or areas where they have lower or higher returns than in the population estimates. It does affect the reliability of the results and the confidence levels. While it might not be a tool that sales wants to dive into trying to sell with the numbers, programmers need to make sure the sample is reliable before you make 'big' conclusions based on it. It's also more of an issue in the smaller markets where often see a lot of weighting with a small sample and some pretty big wobbles in the numbers. Are you feeling real good about gains that were more the result of a few diaries weighted way up that will disappear in the next report? Or are you looking at a downturn that's really an illusion? No research is perfect you have to look deep into the results and the sample before you act.