Not ensuring that the geographic definition you specify for your sample corresponds to the ultimate client's objectives can be both time consuming and costly. In the thousands of jobs we process each year, we've noticed that the geographic frame's definition is often the culprit when a project becomes problematic. Difficulties involving this process are often caused by misconceptions about, or the misuse of, a particular geographic parameter or definition. Sometimes, the difficulty can be traced to inherent problems or idiosyncrasies of the geographic parameter itself.
This Methodological Note will describe the three most common types of problems giving rise to inefficient or inaccurate sample frames: conformity, stability and terminology.
Conformity
Conformity refers to how well a geographic definition fits the area which the researcher would ideally like to sample. Often it's possible for the two to match exactly. Sometimes a near match is all that can be accomplished. Listed sample will, for the most part, conform reasonably well to most areas. RDD sample, however, is another story.
One of the painful truths about designing RDD samples is that telephone exchanges will rarely, if ever, conform to the actual boundaries of the geographic area you want to sample. Since RDD sample frames are defined on the basis of area code/exchange combinations, it's important to understand how your target geography relates to the actual area served by a particular set of area code/exchanges.
For example, an irregularly shaped geographic area may be defined based on its constituent census tracts. However, it may not be possible to generate an RDD sample of households that reasonably represents the target area without including many households outside the boundaries. This conformity problem is more common as target geographic areas become smaller. The following example of a GENESYS Census Tract Coverage Report details the telephone exchanges serving a tract defined area. The report shows the exchange serving the tract area, the total households in the exchange, the number of households in the tract area with that exchange, the percentage of households in the area with that exchange, and the cumulative percent coverage of the tract area.
|
Census Tract Coverage Report
(Bronx Community District - 20 Census Tracts) |
|
Exchange |
Total HH |
HH in Area |
% |
Cumulative Coverage |
|
718367 |
2745 |
1396 |
51 |
9 |
|
718584 |
2504 |
1244 |
50 |
17 |
|
718365 |
2884 |
1407 |
49 |
26 |
|
* |
* |
* |
* |
* |
|
* |
* |
* |
* |
* |
|
* |
* |
* |
* |
* |
|
718884 |
3725 |
45 |
1 |
100 |
|
Total |
68064 |
15792 |
23 |
100 |
n this case, the relevant telephone exchanges do not conform very well to the census tract-defined frame. The "best" exchange (718367) has only 51 percent of its households in the census tract area. This means that 49 percent of respondents would be expected to reside outside of the tracked area. In addition, this exchange covers only 9 percent of the total households in the area. The bottom-line here is that a simply defined RDD sample may not always provide the optimum solution and that other alternatives may need to be explored.
Stability
Another inherent problem with some geographic parameters involves their relative stability over time. Some are extremely stable and are very unlikely to change: the geographic definitions of states and counties are good examples of stable definitions, as are census-defined areas such as census tracts. However, other frequently used definitions can and do change at any time, with little warning.
ZIP code redefinition takes place at the whim of the Postal Service and its thousands of local postmasters. In fact, changes are so frequent that the Postal Service publishes a monthly update of ZIP code changes. Obviously, this has implications relative to a "simple and straightforward sample specification" for subsequent waves of a ZIP code-defined sample frame.
Other commonly used geographic definitions are also subject to frequent changes. A.C. Nielsen's Designated Market Areas (DMAs), used for TV measurement areas, are redefined annually. Metropolitan Statistical Areas (MSAs) are a contiguous set of socioeconomic ally integrated counties which are supposedly defined by the Office of Management and Budget based on a set of objective criteria. However, this designation process has been politicized with the result being a constant expansion of MSAs as well as an increase in the frequency of changes, splits, etc.
Congressional Districts (CDs) are created and maintained by a state-level, politically controlled process. Major redistricting occurs following each decennial census, but changes can occur near election time, due to legislative or judicial decisions.
The definitions of other parameters are, for the most part, reasonably stable, but are open to radical changes at certain times. Census tracts are a good example of this, where major changes occur every ten years, just before and after the census takes place.
When in the design and specification stages of certain survey research applications, especially tracking or repeat studies, one must take special care in assuring that the current geographic definitions can be replicated in the future, or that the current definition matches that of previous studies.
Terminology
Most of the difficulties encountered in the process of defining a geography result from the misuse of terminology. The term "metro area", for example, means different things to different people. Sometimes it may be a generic description of a "city", other times it's used to make a loose or a vague reference to an MSA.
Another commonly misused term is "market", as in "the New York market." There is no single, specific geographic definition of a city's market. There are, however, MSAs and DMAs which can be used for this purpose. Which of these possibilities is best for a particular project needs to be decided on a case-by-case basis. A unique, client-defined geography may also be used.
It's important that the researcher has a very clear understanding of the geographic terms he or she uses to define a study's geography. It's just as important that the researcher use terms in the ways that are most generally understood within the research industry, which may be very different from the common conventions in your client's industry. (Please see the reference section for a glossary of terms commonly used in defining geographies.)
While not strictly a problem of terminology, mistakes are often made when cities in different states share the same name. For example there is a Springfield, IL MSA, Springfield, MA MSA and a Springfield, MO MSA. It's a good idea to be specific when defining this type of geographic area by including city and state name.
Planning for the future
Many of the above difficulties, especially as they relate to tracking/periodic studies, can be minimized if the initial specifications are made with a little thought as to how potential changes in the underlying geographic definitions might impact future waves.
Although much of the above information references RDD sampling, it should be stressed that the same types of problems present themselves when using list-based samples. Changes in the underlying definitions, ZIP codes, for instance, from one wave to the next of a tracking study could result in an unwitting change in the geographic sampling frame.
Conclusion
Given the pressure that most researchers are under, it's easy to understand how habitual behavior can become a survival technique. Sometimes there just isn't enough time to think. However, we would like to urge you to not engage in rote geographic definition.
It's always best to review any geography you are using with your sample supplier - even if it's one you're familiar with. Of course, you should always ask lots of questions when using a geographic definition that you don't have much experience with. Your sample supplier should be able to answer all the questions raised above and provide you with specific cautionary advice. The best suppliers will raise these questions themselves when working with you during the initial stages of sample definition. Don't be surprised if not everyone does.
GENESYS Sampling Systems creates a database of highly detailed geographic information for every area code/exchange combination in the United States on a semiannual basis. Please feel free to direct to us any further questions or concerns about this or any other sampling issue. You may contact us at 1-800-336-7674.