Sampling and Samples — Five Critical Issues

Otterbein’s review of developments in cross-cultural sampling since 1976 (CAM ~2 ) requires correction in five respects. The errors in his presentation affect his discussion of random sampling versus the standard samples, including both the HRAF Quality Control Sample (QCS) and the Standard Cross-Cultural Sample (SCCS). 1. INCOMMENSURATE SAMPLES. Otterbein does not favor the use of srandard samples (SCCS or QCS) but rather &dquo;replication of results&dquo; by drawing multiple samples from a large sampling universe. In his view, all major cross-cultural studies should draw a new sample. He seems unaware of the advantages of more stringent internal replications of results within large samples such as the SCCS, as in replicating correlations in different regions of the world. For example, Burton and I (1984; White and Burton 1988) have used regional replication to great advantage in our testing of major hypotheses. In contrast, Otterbein’s &dquo;incommensurate samples&dquo; approach was used in the period from 1949

samples (SCCS or QCS) but rather &dquo;replication of results&dquo; by drawing multiple samples from a large sampling universe. In his view, all major cross-cultural studies should draw a new sample. He seems unaware of the advantages of more stringent internal replications of results within large samples such as the SCCS, as in replicating correlations in different regions of the world.
For example, Burton and I (1984;White and Burton 1988) have used regional replication to great advantage in our testing of major hypotheses. In contrast, Otterbein's &dquo;incommensurate samples&dquo; approach was used in the period from 1949 to 1969, and due to the difficulty of developing new codes (or coding all variables used by previous authors) few if any of these studies actually replicated earlier findings. A more serious drawback of these studies was that their authors were rarely able to test altemative theories when they used a sample different from that of previous authors who had presented competing hypotheses.
There are valid reasons for drawi ng a new sample in cross-cultural research, as in the case of Otterbein's ( 1985 ) study of warfare-For warfare, as wi th certain other topics, the proportion of case studies which provide data on the topic is low. Consequently, a high proportion of missing data in a standard sample (SCCS or QCS) may pox a problem. Otterbein's sample design called for sampling one case for each sampling stratum, with replacement sampling for every case with inadequate data on warfare. (One could also do replacement sampling wi thin the framework of a standard sample, of course.) Otterbein seems to assume that all major cross-cultural studies, like his study of warfare, deal with topics that are not well covered in most echnogmphies. Studies of topics that are poorly treated in ethno-graphics do tend to call these topics to the attention of ethnographers. This may have the effect of raising ethnographic standards of reporting, but these are not the only types of &dquo;major&dquo; cross cultural studies. Otterbein's sampling design should not be universally emulated.
Even if new samples were preferable for each newcross-cultural study (which they clearly are not), Otterbein offers aflaweddesign for probability sampling. The purpose of probability sampling is not simply to guard against bias in sample construction. Problems of representation (bias) in sample construction are fairly easy to correct, once biases are known, by comparing the sample with known population distributions. Even random samples sometimes need such corrections.
The purpose of properly executed probability samples is to provide, from the evidence of the sample itself, estimates of the confidence limits or standard errors of estimates of proportions, means, correlations, regression coefficients, or other statistical measures. Standard errors are crucially dependent on computation of the variances of observations within each sampling stratum. The problem with Otterbein's design, in common with that of the QCS, is that no variances can be computed wi thin any of the sampling strata, since only one case is chosen (&dquo;randomly&dquo;) within each stratum. This nullifies the advantages of probability sampling for purposes of statistical estimation.

LIMITATIONS OF (STRATIFIED)
RANDOM SAMPLES FOR CULTURAL COMPARISONS. The proper way to do probability samples is to have either no strata (simple random samples-orSRS) or few strata relative to the number of cases (SRS within each major stratum).
However, in the case of comparative research, simple random sampling of many cases per stratum results in the selection of disproportionately many &dquo;similar&dquo; cases in certain regions where there are many societies of the same general type (e.g., Bantu Africa, Malayo-Polynesia Oceanea, etc.).
Should the societies chosen in such overrepresented regions be counted as i ndependent cases or reduced to a smaller number of effectively independent cases? This is, of courx, Galton's problem. The way that i t has been handled in the literature on cross-cultural sampling is to choose only one representative for each distinctive cul ture typeoften using Mur.
dock's classification of societies into 60 world areas (QCS) or his more elaborate classification into 186-200 cultural pro. vinces (SCCS). Such strategies are highly efficient in two statistical senses: 1) by maximizing between--cluster heterogeneity in the sample they are known to provide more accurate estimates of standard errors than simple random samples used without statistical estimation techniques that correct for Gal ton's problem; 2 ) they thereby provide a greater &dquo;effective&dquo; sample size for the coding effort. That is, coding all cases chosen in a simple random sample represents a considerable wasted effort when the effective sample si~e is considerably reduced by a poor choice of sampling design: in this case, the simple or stratified random sample.
Representative samples that have higher effective sample size(QCS or SCCS) do allow the use of statistical techniquessuch as randomization tests and autocorrelationthat provide valid estimates of standard errors, even for nonpnobability samples. The validity of sample representtion (but not its validi ty as a true probability sample) is enhancedwheniccan be assumed that the choice of one case per stratum is unbiased. ' The SCCS is commonly assumed to achieve this by choice of the best-described case for each stratum, while the QCS restricts the sampling frame to best described cases and makes choices among alternates randomly. Neither approach is self-evidently superior to the other in terms of representation, nor is either one a true probability sample in terms of advantages (Continues on page I I )