METHODOLOGICAL NOTE

Effective evaluation of a compiled list is the only way to be sure that the list is consistant witht he objective of your survey.

Lists have become a very popular sample source in today's research environment. Low incidence of a particular target group, a tight budget, or data collection time constraints are just some of the many reasons contributing to this trend. Unfortunately, the use of compiled lists is fraught with potential hazards. It is of crucial importance to develop and implement an exacting evaluation methodology in order to minimize these hazards.

The importance of this evaluation process becomes increasingly obvious as one considers the sheer number of compiled lists available. There are literally thousands of consumer and business lists to choose from, for example:

• Magazine subscriptions
• Mail questionnaire responders
• Phone numbers from a white page directory
• Convention attendees
• Warranty lists
• Businesses that advertise in the yellow pages

Given this wealth of choices, it's fortunate that there are a number of attributes common to all compiled lists. These attributes conveniently fall into a framework of four basic questions:

• How is the list built?
• How is the list enhanced?
• How is the list maintained?
• How is the list structured?

It is important to note that there is often a fine line between the attributes these questions represent when dealing with lists in any but the most theoretical way. A procedure that may represent the enhancement of one list may be part of the building of another. Regardless of the blurred distinctions that are not uncommon in the research industry, carefully considering the answers to these questions when evaluating any compiled list will help you decide if it will provide the best sample for the particular project you are working on.

It's not that one list is intrinsically better than another. Each list is different and must be evaluated individually in order to make sure it's consistant with the objective of the study in question. The truth is that a list that works perfectly well for one study may be unsuitable for another.

How is the list built?

As with any sound evaluation process, we will start at the bottom and work up. Every list has a basic foundation upon which it is built and we begin with this underlying structure. Thus, the first question to be resolved is: what is the primary source of the data records? In other words, how does someone get on the list?

For some list sources, the answer to this question is quite obvious. For magazine subscription lists, the initial source upon which the list is built is simply those who subscribe to the magazine. For association lists, the initial source is the members of the association. The InfoUSA list (widely used for listed household samples) is built from white page directory listings.

Not all lists, however, have obvious sources. Take the list titled "Business Frequent Travelers". Where exactly do the records come from? Is the list based on attendees of a business convention for shoe salesmen, or from a national mail questionnaire? There is a difference. What about a list titled "New Homeowners" that is based on mortgage applications. With today's low interest rates, many homeowners are re-mortgaging, yet they are not new home owners. It is these types of lists where the basic foundation can have an impact on your survey objective.

Once you have determined the primary source, the next step is to determine what information comes with each individual record (i.e., name, address, phone number, etc.). For the InfoUSA list, this is everything you see in a white page directory listing - the name (of the person listed in the phone book) and the associated address and phone number. For the frequent business travelers list (from the shoe convention), it might just be the shoe salesman's name and company name. Why is it necessary to know this information? If the sample is for a telephone study, it would be nice to know up front whether a phone number is available!

Evaluating a list's primary source is very important for one reason -if the foundation upon which the list is built is not sound(for your particular application), then the rest of the compilation process is inconsequential.

How is the list enhanced?

Assuming the list met your requirements in terms of its basic foundation, you can proceed to question number two: how is the list enhanced? By this, we are referring to what is added to the basic foundation. This could include the addition of other data sources, thereby increasing the number of records on the list, or adding more information to those records already on the list.

Most lists you will come across are from a single source and receive no enhancements to the basic record (other than what came from the initial source). Multi-sourcing is prevalent among compilers of large lists who are trying to achieve high coverage rates. For example, a list compiled from white page directories, which misses unlisted/unpublished telephone numbers, may be supplemented with subscription lists in order to increase coverage of all households. A business list compiled from yellow page Directories can be supplemented with a manufacturing list (an industry that does not typically list themselves in the yellow pages).

In addition to multi-sourcing, compilers will sometimes add more information to the existing records. While some information will be exact, other pieces of information may be statistically modeled. Depending on your specific application, a modeled figure may or may not be adequate. For example, it's not uncommon for household income information to be modeled. From a personal point of view, it's nice to know that our exact household incomes are not public knowledge. From a researcher's point of view, however, it doesn't insure 100% accuracy when targeting a specific income group.

The bottom line is that these enhancements can increase the value of lists.  However, the precise manner in which these enhancements are implemented will affect their value for a particular application.

How is the list maintained?

Things change - people move and make career changes, interests and hobbies change, products previously owned may be discarded for the newest version, etc. Although a list may be an accurate static representation, we live in a dynamic world. Depending on the care taken by list compilers, it is possible that a significant percentage of a list could be obsolete by the time it is available. The end result of this could be inefficiences in data collection efforts as well as the distortion of survey results.

When you examine what procedures a compiler is using to maintain/update their list, you need to consider how frequently the maintenance is being conducted, and what records are being updated/verified.

Certain major list compilers exercise rigor in the upkeep of their lists with regular maintenance cycles. For example, the entire list may be compared to and corrected by the National Change of Address (NCOA) file on a monthly basis. However, it is not necessarily true that all list compilers devote the resources necessary to perform ongoing maintenance.

The point is that asking questions about the maintenance frequency and scope is an important component of your list evaluation process.

How is the list structured?

Lists are structured in a variety of ways. How the records are ordered on the list can make a difference when the sample is selected. Records can be arranged in alphabetical order by state, last name, industry type, etc. Alternatively, ascending ZIP Code order is a popular way of structuring a list. Depending on the specific list structure, an nth name sampling methodology may or may not accurately represent the entire list. Obviously, this order has to be taken into consideration when pulling the sample.

Unfortunately, getting a list vendor to tell you how the list is structured may be very difficult. Often list brokers will not know the answer to this question. The primary focus for most list brokers/compilers is on direct mail applications. Terms like "statistical representation" or "nth selection" may have little or no meaning in their day to day business. Don't be surprised if the person on the other end of the phone doesn't appreciate your need for this information.

Summary

Again, it's not a matter of one list being intrinsically better than others. Use these questions to evaluate how well the list will meet the specific needs demanded by each particular project. One set of answers that would be fully acceptablefor one study might mean disaster for another. Taking the time up front to effectively evaluate your list options will lead to a more representative, cost effective survey, and, ultimately, a happier client.

As always, please feel free to contact us at 215¬653-7100 if you have any further questions about this or any other sampling issue.