Phones - Research and the Elections

As we wade through the endless stream of political spots on every media with all their accusations there is another story behind this fall election in the U.S. that could be a major factor.  The telephone based polls that candidates and the media has used to track the public since the 40s could end up being very mis-leading.

Nearly every analysis of the polls shows a Republican leaning wave moving the House and maybe even the Senate to a Republican lean.  But, those polls are mostly done on by calling land line households since reaching out to cell phone users for research is costly and difficult. The national polls are more likely to use cell phone households, but on the local and state level it's still pretty costly and not used that much.  Much of the polling data being analyzed for this mid-term election could be off the mark.

In radio we know the cell phone only households and their effect on research.  For years we've struggled with incomplete data in the under 40 age cells and watched our samples decay.  Arbitron and Nielsen have both retooled their sample systems and now do a better job including cell phone households.  Still, we struggle with not enough cell phone sample, but at least we have a little more accuracy.

The Pew Center has reviewed the cell phone only households for the political polls and released the following data:

You can see that the Cell Phone only households are more likely to lean Democratic changing the pools by around 2 percent.   When you factor in that most research has a 3-5% variance and then add the 2% Democratic lean many of the Republican margins could suddenly be a lot closer than expected.

Could we see an election here where polling is suddenly in question?  Much like the famous Dewey - Truman election in the late 40s where polling predicted Dewey would win.  Back then phones were not in all households and when the non-phone households voted the polls were suddenly skewed.

We have relied on phone based research for nearly all our data on the audience over the years.  This fall we may see in a big way that the phone is flawed.  Cell phones, VOIP, Skype and so many other ways of personal communication have made the land line a thing of the past.  We - and the politicians - can't rely on it anymore for data on the audience.

Audience Research - The Future - Summary

With so much new media plus changes in entertainment and communications, as well as a revolution in our measurement systems we have many more questions than answers.  Asking the same old questions to a sample gathered in the ancient days of land line phones in a format that takes lots of precious time to execute will make it real hard to find the answers to so many questions.  

We have the tools and opportunity to blaze new trails in researching the audience.  But, like so much in radio we have to get our heads out of the 80s and welcome the new world.   Hopefully as we roll out of the recession some of the gains will find their way into learning about the audience in their new world.

The keys lie in moving more towards managed panels for our research base, moving more towards on-line resources with their rich databases.  The next step is to overhaul our questions to learn more about behavior instead of just reciting recalled images.  To learn more about the audience on these levels we need to look at new data gathering systems to get past the surface levels and also use new analysis techniques to get to the core of real audience actions.  We'll also need to move away from talking to them on the phone and interact with them in their world which as moved to the digital world of the web, cell phones, and smart phones/tablets.

It's not like we are really blazing a new trail here by ourselves.  A lot of other marketing research and industries have already started down this path.  Let's learn from them and start looking at the audience's behavior and how we can be a stronger part of it.

Audience Research - The Future - Data Collection

It's obvious that we are in the middle of a revolution in communication.  With cell phones and the Internet we suddenly have a whole new platform that is evolving in a full digital world following Moore's law.   The law is built on the capacity of integrated circuits which doubles every 2 years - and with it we double the capability the the circuits and their applications.  We no longer talk on our cell phones, we text, surf the Internet, blog, tweet, Facebook, take pictures, shoot movies, and now have millions of apps that do almost anything.

In the Radio Research world a lot of what we do still relies on talking on the phone or filling out pen and paper forms - like the diary.   Our measurement is moving away from this model with the PPM, but is our perceptual and music research following along?

When you think about the telephone perceptual it's a lengthy world.   We have to read off long lists of stations, music mix descriptors, personality names, image descriptions and if we want open ended comments someone has to transcribe them.  It limits the amount of information we can get from them and the time they will stay on the phone.  We also have that interaction with the interviewer which sometimes holds people back from their real actions and thoughts.

Moving perceptual research online opens up a lot of doors.  We often get a lot more open ended responses with very valuable impressions and images, we can cover long list of images, stations and other questions much quicker.  There are also lots of animation options that will allow us to use dial type scales and other ways to collect data that gives us more insight than just the 1-7 scale. Imagine what we could do with the I-Pad touch screen!

Online research also moves at the pace of the responder - not the pace of the music test or the interviewer, the sample can come back to the questionnaire/music list at their schedule.  And of course we can add audio, video and pictures/images into the mix with ease.

Having tried out a few on-line perceptual studies we've seen the potential.  While we have all done a quick survey on-line there is a lot more that can be done in this environment.  Some innovative researchers working with other product lines have started to push the envelope in questionnaire design and the potential is very impressive.

It's a whole new world of potential and possibilities that we haven't even tapped into .  Perhaps as we start to dig into audience research again in our budgets we'll have the courage to innovate instead of just replicate.

In our last post in this series we'll summarize the thoughts early next week.   Please feel free to add yours.

Audience Research - The Future - The Questions

The usual pattern in most research studies really ends up with the same questions.   We start with their 'cume' behavior/recall and then narrow it to most listened to stations then we move on to the body of the questions .  We craft music blends for them to evaluate, ask about Morning shows, other personalities, maybe at work listening, contests, marketing efforts and maybe a few questions about the web site (have you been there).   Sometimes we are on a format search where the music mixes or other descriptive moments take more time in the questions.  There's always the pressure of keeping it under 25 minutes on the phone so we really don't have that much time.

Notice that the only area that is about their behavior is in the Cume and which stations they listen to the most.  An even there we may see the most marketed station win even though we look down their answers and they don't really know much about that station in the other parts of the study.  Where do they listen to radio the most, and what are those stations?  How passionate are they? Are they super radio users or just casual tuners?  Do they jump around as soon as the music sets end?

Also what kind of people are they?  Are they innovators that latch on to new concepts or products or are they laggards who are just figuring out innovations that happened years ago?  Or are they in that mass in the middle and where do they rank in that segment?

A few more questions about them before we dive into their perceived listening habits would really help us learn a lot more about what their actual behavior might be.  Right now we are mostly learning what they recall about our products and not a lot about how they use them.  Recall worked in the diary days - now it's about real behavior.  We have to find the secrets to making our products perform on a different plane

Many products and industries have used this type of research for years - dating back to the 60s and while it's gone through a number of evolutions it's still in use today.  Finding the key links in the sample or population and building products that use the keys to open doors to the bigger groups has been a big element for many industries.  

Now how do we communicate with the sample?  A lot has changed in the last few decades let's take a look in our next segment later in the week.  

Audience Research - The Future - The Sample

The land line phone and a random sample from the general population taken from the phone number listings has been the base for research since the late 50s.   Since radio really didn't get into ratings till the 60s and audience research in the 70s we started in the telephone era and for the most part haven't evolved much.

Now the world has obviously changed and the tides of radio research are also on the verge of changing.

First we have the land line telephone - which is quickly dying.   We are now at over 25% of the population who don't even have a land line and in some demos it's nearly 40%.  The number of Cell Phone converts grows every day and we can't call them on their cells.   Arbitron has begun the shift to try and integrate more and more cell phone households into the sample frame, but has our perceptual and music testing research?  It's not become a widespread practice, because it's hard to try and set up systems to reach cell phone only households for a survey done 1 or 2 times year.

2nd the audience has clearly changed the way they communicate.  Written snail mail and actually talking on a phone (cell or land line) are both evolving to new worlds.   With the internet we communicate in a whole new world from email to text messaging to social networks and many more evolutions to come at the speed of the internet where every 18 months we see innovations all over the place.  But, how do we communicate or research our audience?  While we have dabbled in on-line music testing, some station database questions, and maybe a few have actually done an on line perceptual test most of the time we still talking on the  phone and punching the responses into a CATI software program.

The new answers may well lie in managed panels.  Really a phone listing is a form of a managed panel of households that have phones.  In this case the panels could be built from a wide collection of on-line sources and serve as a sample base to draw from.   There are managed panels all over the place already that we could tap into and use that base to reach out on many platforms on line to gather the data.  We have started to use them for some online music testing and a few questions from our station databases, but our valid concern is just talking to our fans and not the whole market.

We also have to take a look at the new world of PPM.   This is really a managed panel of the market that is carrying around the meters for months and even years.  It's different from the old random draw of the sample that took place every week.   Here the sample is pulled with only a few replacements every week or month.  Instead of getting a snapshot of a few people's listening habits for a week we get a full length movie of them over 6 months, a year and maybe even 18 months.

Radio research used to rely on the random sample and in today's world trying to poll everyone doesn't yield enough specific information for us to work with.  More and more we need to stratify the sample and narrow the vision to those we have a chance to reach with our products when we do audience research for perceptual and music surveys.

There are panels out there, the key is grooming them so we can use them to research our product.  Many others have already started.  NASCAR, PPG, the auto companies, and many of the major retailers already work hard to grow panels that they can rely on to help them keep momentum with the audience on their products.   We need to start to find ways to make this work on much bigger scale than just our newsletter email list.

We also need to start asking new questions to learn more about behavior and not just about what they remember of our products.   We'll cover some thoughts there in a few days.   Feel free to chime in and contribute your thoughts on these topics.  Everyone's help will be needed.

Audience Research - The Future

Over the last 4-5 years doing audience research for music testing and perceptual studies has been an expense that has in many cases been deemed too expensive.   With the economy struggling, lots of competition from new media, and the need for every manager to cut expenses research seemed to fall off the table.   Clear Channel pretty much dismantled it's in house Critical Mass Media and left BA research on the table.  Other groups curtailed a lot of their work in studying the audience and many smaller owners also cut back.  

The reality here is that in many cases we've gone a long time guiding our ships in ever changing seas of new competition, audience evolution, and life style changes that no doubt have affected how the audience uses the product and their passion levels for it.  Now the economy is improving and perhaps we are getting ready to return to doing some audience research.

A lot has changed with a completely new measurement system in many markets.  The old sample system of using  mostly land line phones to gather the panel is suspect as so many people have turned off their land lines for cell phones.  Even Arbitron is using different systems to gather the sample to include non or limited land line participants in all markets.  We also have new opportunities using the internet for research that we have only done a few experiments with.

Much of our perceptual research in the past was built completely on recall behavior.  After all our whole game in the diary days was built on recall.  It really didn't matter if they really listened to our morning show every day, if they had high recall and impressions of the show we often got credit for listening.  If they could remember it to write it down - game over.  Making the impression still matters a lot, but we also need them to listen.

It's time to scrap the old questionnaires, the old auditorium tests, and the call out studies and come up with some new systems to find answers to new questions.  But, is anyone up for the experimentation it's going to take?  Over the next 10 days let's take a look at some opportunities to remodel our research here - I'll be bringing in some ideas and thoughts, but if you have any please feel free to contribute with your comments.  

First up - The Sample in just a few days.  

Thanks

Wow Cell Phone Households - Big JUMP

The Center for Disease Control has released a preview of their bi-annual study on Cell Phone Households from the last 1/2 of 2008.   Of course it grew again and now over 20% of the households in the country have no land line.  Another 14.5% take almost all of their calls on the cell phone despite having a land line.  

That's nearly 35% of the country and if you break it down by demos and over the last 6 years it's a pretty amazing graph: 

In 18-34s is around 40% - nearly half and at the rate it is growing it will probably hit 50% of this demo before 09 ends.   Even when you move to the 45+ cells it's around 15%.  

Keep in mind that this study was from last year and with the recession hitting so hard as the year wrapped up no doubt the growth of Cell Phone Only households will pick up even more.  Cutting out the 20 or 30 bucks a month to the land line is really pretty painless - gee now I can add the data package and not even notice it.  

Aside from the trends in the audience here we also have to realize that this greatly affects our ability to have somewhat accurate audience ratings and research.  Connecting with Cell Phones for research is very hard or impossible as we all know seeing the shrinking samples in our audience research.   We also have to look at the audience research we do in music tests and perceptual studies.  If they are based on telephone samples they are missing at least 20% and probably closer to 40% of the population and in the younger demos it's over 50%.  

If you look at the early snippets from the study you can see that the cell phone only world is different on many levels.  We'll see more in a few weeks when the whole study is released.  You can read the preview here. 

Radio's Outdated Research Systems

We really don't have any index or measure of how many stations are doing any 'formal' audience research, but it's probably a safe bet that few are.  The days of doing an annual perceptual study and a full auditorium library test are most likely fading, or in many markets completely gone.  No doubt the economic slump isn't allowing the 'luxury' of gathering the audience's impressions of our products.  Clear Channel used to have a division (Critical Mass) with hundreds of people devoted to perceptual, call out and telephone marketing, now it's pretty much dismantled.  No doubt many in the top 50 markets are flying without any 'radar' looking out at the audience and gathering their impressions, preferences, and the reasons why they listen.  

Perhaps with all the data we have from PPM what do we really need research for anymore?  With PPM the game has changed from a recall strategy.  Now we actually see what the sample listens to minute by minute every day without any recall on their part.   We can see when they tune out and trace back to see why?  But, do we really know?  Did they leave because they hated the song or jock?  Or did they have to go to work and leave the car?  Did they pick up the kids and they flipped on another station?  

While we can finally see an accurate picture of what they listen to we really don't know why they listen, or why they don't.  We used to get some of the picture from perceptual and music testing, but we also had to tailor our research to the reality (from the past) that the audience had to recall their listening - not have it measured automatically.   Before we often won if the audience just remembered our call letters, frequency or slogan - now they actually have to listen for it to count.  

For many stations the perceptual and music testing has been taken over by the station's database.  The folks who are big enough fans to visit the web site, bother to sign up for another database, and wait for their mailbox to fill up with our newsletters.   We can test our music with them, ask a few usage questions, and maybe even find out who they think will win American Idol. But, these little bites of data we get from a small universe of very dedicated P1s is a very distorted picture of market and the audience.  We really can't see the much larger P2 groups or fans of other stations that could also be users of our product.   No doubt the low cost of doing music testing and asking a few questions of the super P1s in your database is worth it, but we have to realize it's not a very complete picture.   

The advent of PPM, the changes in the way the world communicates, and the economic realities are all combining, making the old styles of research outdated and ineffective.  

Spending 50K (or more) on a perceptual study that only reaches the households that will bother to answer the phone and hang with a 25 minute questionnaire are gone.  While we used to justify using the telephone with 'that's the way Arbitron does it' it doesn't work anymore.  Even Arbitron is trying to use address based samples to gather the sample and in PPM they pretty much keep the same panel for up to 18 months.   

Not only do we lack the budget in today's world - is the data still giving us an useful picture?  

In PPM we are now looking more at behavior - not recall.   To be effective we actually have to understand how they behave, not just what they remember or what we've drilled into their brains.   But, we've pretty much quit asking the questions in an organized and strategic format.   And it's happening at a time when we have a complete overhaul going on in the media landscape.  This is like shutting down all the weather monitoring systems in Oklahoma in February for 8 months - we'll just keep looking out the window for those pesky tornadoes.  

It's time to get our heads out of the sand and start listening to the audience again, but we also need a new model.   The perceptual studies from the 70s/80s are too slow, cumbersome and expensive in today's world.   On top of that we are still asking lots of questions that date back to the recall days of measurement.   

It's time for innovation in research, but is anyone working on it?  

The Year of 25-34s FINALLY!!!!

In this week's Arbitron Advisory Board meetings build up it looks like the main issue on the table is SAMPLE. Advisory Board leader Chuck DuCoty sums up the agenda in his press conference call - 'Sample Sample Sample.' Chuck (a fine programmer before he crossed over to the GM and now COO chair) hits it on the head and now it looks like Arbitron is at least ready to go to work.

Arbitorn Exec Pierre Bouvard has now declared 2008 as the 'The Year of 25-34s.' This after declaring that Arbitron's recent campaign to get 18-24 returns up was improving. But, most of that was centered around the PPM markets in Philly and Houston. Even after PPM rolls out there will probably be around 200 markets still looking at diary returns, with it's weak data collection system, and need for more annual sample (remember the PPM sample mostly stays around for a year or more and diary keepers are only in for a week).

So let's take a look at the Spring 2007 National Sample report from the Arbitron Radio Nationwide pdf on their website (look at it here). In the sample report here are the Spring 07 sample results:

Arbitron has a long ways to go even if the goal is only to get into the 70 index range for Men 18-24 and the news is even worse in 25-34 Men. Realize that that means they are only delivering a C- sample (if we were grading it in a High School format where 75 is a C). Just barely passing the class and being able to stay on the team - not enough to get accepted at anything but a trade school.

Pierre claims that the PPM sample in Philly is now above 70 in 18-24 Men but there they are pulling out all the stops. They are going door to door!!!! Also adding a lot more cash to carry the PPM and calling and calling till it hurts. Will they make this level of effort without PPM on the line? In your market?

The sample index problems we see here are not new. It's been an issue since the 80s - over 20 years. At first it was just those pesky 18-24 Men, but now it's even worse in 25-34 and Women have also joined the non response team.

You have to reach out beyond the land line telephone. This group are the first to move to cell only or ignore the land line households. They are quickly heading to 50% unreachable by Arbitron's land line sample system.

It's time for new answers and I bet Chuck would agree that we are not seeing many options that have much of a chance to impact this crucial area.

Regular readers here have see a few posts on some ideas - I'm sure there are more out there. Somehow they have to get to the surface very soon. The agencies are already looking at Arbitron with a raised eyebrow (to say the least) and all the PPM developments and publicity only make the diary system even more suspect. Add the bad sample to the weak diary data collection and this is quickly becoming a guess - not an estimate.

The Random Sample - No Longer Random or Accurate

For years in research we have relied on the "Random Sample." It's the core of statistical based research that relies on a sample instead of a census. We all know the theory that in a world of limited options we only have to talk to sampling of the population to get a confident estimate of their behavior.

When the Telephone came along sampling became easier. Suddenly research became pretty simple. Get a phone bank together and get a list of the telephone numbers in the community then put together a random formula to grab the names and call away. At first it was easy as everyone had a phone, very few had private or unlisted numbers, answering machines were rather complicated new fangaled equipment and caller ID wasn't available.

Now we are in a world where the land line phone is a relic. Over the last 20 years the database has become torn apart by more and more unlisted numbers, don't call lists, over tele-marketing, caller ID, answering machines, and the biggest issue THE CELL PHONE. So where do we go from here in research?

Perhaps the first realization we need to make is that a simple random sample is really no longer available. In most of our research we stratify the sample in some manner, perhaps making sure we hit a sample goals. In Arbitron we have ethnic groups, population distribution across the market and demographic groups involved in various sample goals. In our radio product research we also limit ourselves to target audiences and P1 preferences based on the goals of the project.

So why are we still clinging to the idea that we need to have a random sample in research for it to be accurate? What we are really doing in nearly all reserach today is building panels. Groups of respondents that fit some picture of the world we are looking at - realizing we can't see the whole picture on any subject in today's world.

I bring all of this up because we need to start looking at a different model for our panels. It will have to start with Arbitron which is still clinging to the old land line phone database.

Now Arbitron hopes to get a little wider view from the shrinking land line phone view from going back to home address based databases in combo with the land line database. By comparing the 2 they can find out who are the cell phone onlies and try to get them into the panel by going around the phone.

The problem here is that while we all probably have a mail box it also suffers from many of the same problems we have in using the telephone. We screen the mail closely and quickly sort non important items and Email competes with the mail box for a lot of communication needs.

We're going to have to reach out and build a complete database and it's going to take more than an outdated phone and address list. We need to communicate with the audience and the sample on the internet - it's their new community and finding sampling tools here needs to be explored quickly. It's a fast moving world and unless Arbitron starts to experiment here and move a lot quicker we will continue to live in the past with the audience estimates we sell with.

There are many experiments to tackle including building panels using email, text messages, social networks, and reward based incentives. Yet, we seem to find the radio research community looking at the web options with an evil doubting eye. Look at Neilsen's web site research where they are purely soliciting the panel online. You apply to join up and are rewarded if chosen to put a collector on you browser to relay your activity back to Neilsen for their data. In their TV research they gather the panel by door to door solicitations, the phone and via email. And once you are in the panel you stay for years much like PPM.

How about building a big panel that we pick from for the diary system that will still be in place for 150 to 200 markets for at least the next 5-10 years. If we had a nicely proportioned panel of 10,000 in many of the markets where we have 1000 diaries in a report we could draw from that sample and suddenly it would be easy to find enough males or Hispanics or 25-34s. We have a group that knows how to complete the survey, is better compensated (since we don't have to go to all the expense of random dialing through the whole database), and reliable to respond. A lot better answer than striving for a Random Sample in a world where no sample is random any more.

A Few DEMO Facts

As 2008 gets underway everyone is another year older and when it comes to target audiences and demos some interesting facts come up:

• If you take the definition of the boomers as being born from 1946 to 1964 the first boomers are now 62 and the youngest are 44.
• The only cell with any boomers left in it within 25-54 is the last cell 45-54!!!!
• The peak year of the boomers was 1957 and now they are 51 years old - in just 3 years they will be out of the 25-54s.
• Around 38% of the boomers are already out of the 25-54 cell.
• The newest members of 25-54s were graduating from high school in the 90s.
• If you were graduating with the dawn of grunge (1992) you are now 34.
• In 2009 Led Zeppelin's first LP will be 40 years old. In 1990 a 40 year old piece of music was from 1950 - the end of the big band era.
• The last boomer will turn 50 in 2014.

We used to think of 25-54s as all boomers and the formats that did well in that demo were boomer formats for the most part. Classic Rock/Hits, AC, and Country all had good appeal with boomers and are often the first choice if you want to win 25-54. But the reality of 25-54s is changing the Generation X and Y are taking over the demo and the music of the 80s and 90s is starting to be the mainstay soundtrack for the new 25-44 demos. Soon there will be no boomers left. What formats will be the most valuable then? Have we even thought about this or even started to create new formats that will appeal to the new 25-44s?

This could be the golden opportunity if you are looking for new ways to win in 25-54s.

New Rock Music for 25-54s

While Classic Rock is no doubt a great format and we've seen a whole bunch of winners playing the 'music that's stood the test of time' it's a tough format to keep alive in the long run. The problem is New Music. Yes there are so many great songs that seem to never completely burn out in the library, but it's still a challenge to make Freebird sound exciting enough to hang out for a spot break to hear it for the millionth time.

There are new songs out there that could be played. Just look at the new Springsteen E Street Band reunion. It debut at #1 in sales and hung in there for 2 weeks. The song is a very well done Springsteen tune that in the 80s would have been zipping up the charts. There's a ton more of the vintage artists that have tried to keep putting out new music and there's also a bunch of artists that really are compatible with rock. Why couldn't we play Coldplay if we play U2 or perhaps even taken on John Mayer or Dave Matthews if we played the Eagles? We play old Ozzy all over the place, but little happened with the new Ozzy outside of all the active rock airplay which is really not directed fully at the boomer 40 plus audience that Classic Rock targets.

This same audience is paying thousands to see the classic artists perform in concerts and they are really the last ones buying CDs today. Part of the problem lies in the positioning of Classic Rock. Just like we saw in the name 'oldies,' which meant only songs from the 50s and 60s to the audience - Classic Rock means only songs from the 70s and 80s. It's been sold to the audience that we only deliver those decades and new music, even from these artists, is not in the package. So when we play it the audience rates it low on the scale. We could build a station that blends the classics and new music from compatible artists but it will take a whole new platform. Someone has to try.

A while back we revamped the Fox in Grand Rapids to include currents and still focus on a Classic Rock base. It took a whole revamp of the positioning and imaging, but in the end we were never able to make it fully work. The station gained a lot at first then fell behind as the marketing dried up just as we needed it and internal changes pushed the station off target. Had they stayed the course it was likely it could have been a breakthrough station. It's an approach that needs some research, fresh marketing, and you have to be very careful with finding truly compatible newer music.

If you look at other 25-54 formats like AC or Country they have NO PROBLEM playing current songs and getting compatible new artists in the mix. Why can't rock?

If you want to get some other views visit Lee Abrams' blog here. While Lee is seeing it from the XM perspective the reality is still there. BTW his views are a bit buried in his blog entry this week - you'll have to wander through Lee's airplane tales (always entertaining to those of us who fly and wish we could just take off in our own rig) and a few sales pitches for the XM channels, but it's always a worthwhile read.

Research or Ratings SAMPLE is EVERYTHING!!

Recently we've seen lots of discussion over samples in ratings. Much of it relates to Arbitron rolling out the PPM and we've seen some industry concern over the sample both in size, return and balance for ethnics and demos. No doubt we'll see more confusion on the sample with PPM rolling out because PPM is really a different type of research than we have traditionally used for ratings. With the diary we are looking at a snapshot of one weeks listening as recalled in a diary vs the behavior of the respondent over weeks and weeks with no 'recall' necessary. If you go to our reports in McVaymedia.com you can get a lot more information and analysis on the differences between PPM and Diary methods here.

Before you work with any research or ratings data the most important analysis tip we can pass on is LOOK AT THE SAMPLE. How far off of the population estimates is the return? You may also want to look at how the sample returned in zip codes or key ethnic areas. Know who they got of hold of. Of course Arbitron does reveal the numbers, it may be a little harder to find on the Max or Arbitrends menus but it's there and you should look at it before you celebrate or wonder what happened. The same goes for any perceptual or music testing you do.

In order to get realistic results Arbitron and many other studies balance the sample by weighting. This places more value on demos or areas where they have lower or higher returns than in the population estimates. It does affect the reliability of the results and the confidence levels. While it might not be a tool that sales wants to dive into trying to sell with the numbers, programmers need to make sure the sample is reliable before you make 'big' conclusions based on it. It's also more of an issue in the smaller markets where often see a lot of weighting with a small sample and some pretty big wobbles in the numbers. Are you feeling real good about gains that were more the result of a few diaries weighted way up that will disappear in the next report? Or are you looking at a downturn that's really an illusion? No research is perfect you have to look deep into the results and the sample before you act.

Step Away From The Keyboard

You probably know something of the history Lee Abrams' impact on Rock radio with his development of the AOR/Superstars format in the 70s and 80s. Today he's at XM leading their product develpment and still dispenses some useful insights on his blog www.leeabrams.blogspot.com .

One of Lee's most impactful themes was 'the balance of art and science.' Lee used this theme a number of times addressing the programmers of the 70s and 80s and it still rings true today. Great radio stations are not completely built on the science of crunching numbers, analyzing research, and working behind the computer screen. They are built by the creativity of the whole team. The air staff's work behind the mic, the imaging between the songs, the promotions in the streets, and having a creative music mix.

While the research data, the music scheduling software, and digital studios are useful tools. The research allows us to hear the audience's reaction and opinions and the music software and digital studios do help us consistantly keep the product on the air they are not content. Yet we often see PDs working dilgently at their screens breaking down web metrics, tweaking the music log, fine tuning the station to the smallest details - instead of spending time focusing on the ART side of the product.

Radio programming is a 'team sport.' It takes the creative resources of everyone on your staff and to use those creative resources you have to put on your coaching hat. Great winning coaches have great skills with the 'Xs and Os' but they also know how to motivate the talent on the field. While the plan may gain from spending time behind the keyboard the real execution comes from leading the staff. Taking the time to brainstorm, motivate, and coach the airstaff, imaging team and promotion folks is where you can make a real impact on the product. Great stations do not come from the computer screen they happen on the air.

On Line Research

Been preparing a Panel for the Canadian Music Week conference in Toronto comparing on-line and traditional phone based research systems. For most of you who have never been to a CMW convention Canadian Music Week is a unique gathering of Canadian artists and broadcasters every March in Toronto. There are many sessions for both the broadcasters and the musicians and lots of showcases - it's a lot more like the Country convention in Nashville than the typical NAB or R and R meeting we have in the U.S.

Our panel for the On-Line/Traditional session includes Jeff Vidler from Solutions Research in Canada, Mark Kassof from Kassof Research, Tom Meyer from Media Score on line research and Julie Adams VP/GM/PD at CHDI in Toronto. I've worked with Jeff, Mark and Tom on various projects in Canada and with Tom and Jeff we have stations where we do all on-line research. Perceptual studies, library music tests (300 plus titles) and monthly call out style testing is all done on-line.

There are a lot of advantages to doing research this way including:

• Sample Size - In most music tests we are lucky to have 80 people in the room, but most on-line projects are at least 300. The much larger sample allows us to slice and dice the data a lot more with confidence in the results.
• Respondents dictate the pace - Since all the respondents are doing the study on their computers at their pace they can stop for dinner, replay the song or audio/video element and take the test at their pace. No more people in a music test finding themselves 3 songs behind or rushing through answers on the phone. People actually have time to think about their responses and it shows in the data.
• Phones - Even though Arbitron and BBM still use phones to gather the sample it's quickly becoming an outdated way to do research. We can't use cell phones and so many people really don't communicate much anymore on their land lines.
• New presentations on the questions - On line we can play audio, video and even use some java/flash tricks to make the questionnaire more interesting and interactive. No more repeating 'on a scale of 1 to 5 with 5 being love it and 1 meaning you hate it...' We can put a little volume control up and tell them to turn it up or down.
• Costs - It's clearly less expensive to do research on-line than using call out, phone banks and auditorium tests.

The biggest obstacle is gathering a sample. With the phone we have a directory that probably includes at least 60% of the phone numbers in the market so we can just grab a bunch of numbers and start dialing. On the Internet it's a little harder. There are millions of email addresses and since the user defines the name there's no system to it. The addresses are also all over the world which won't help you in the middle of Kentucky. We have found more and more databases of email addresses to use. In Canada, with a smaller population base, it's a little easier to find email databases that are sorted by geographic area (postal codes) and ages. For the monthly music testing on-line we build a database from on-air/web promos and for the library (bigger) music tests we use a combo of station and outside databases.

Actually the results have been very useful. The music scores are a little more passionate than we see in the auditorium tests, with less unfamiliarity and higher burn scores. In the perceptual studies we see a lot more longer and more detailed responses to open ended/comment style questions. You can see that when the audience has a little more time to respond and can do it at their pace we get more data and they are clearly a little more 'into it.'

This is the future of research. Some may argue that this system is not as random, doesn't match the systems the ratings firms use, and that we don't have as much control over who we are talking to. All valid points, but having a lot bigger samples, moving away from the outdated phone line, lower costs and getting a lot more data are also big advantages that need to be considered. We all know we need to re-think and re-invent radio to survive in the new media world. We're going to need research and data to do it and a lot of it. On-line systems will have to become a part of the plan sooner than later.