A future cities survey research design for policy analysis

Abstract Strategies for the analysis of public policy have been a major focus of interest among social scientists during the last decade. This paper introduces a survey research design and perspective which differ from most current strategies for public policy analysis. The design, which is basically an ex post facto correlational design, employs an unconventional sampling technique to derive policy-impact statements from a small sample. The policy analysis perspective of this research is focused on making prescriptive rather than descriptive statements. The specific research is the “URBIS” project, a study which is evaluating the impact of automated information systems upon the operations of local governments. After briefly discussing the general logic of public policy analysis, this paper details the methods and critical issues of this research design. The paper suggests that the URBIS research strategy might be viewed as one general model for public policy analysis.

The URBIS Group, Public Policy Research Organization, University of California, Irvine, CA 92717, U.S.A.

(Received 2 February 19%)
Abstract-Strategies for the analysis of public policy have been a major focus of interest among social scientists during the last decade. This paper introduces a survey research design and perspective which differ from most current strategies for public policy analysis. The design, which is basically an ex post facto correlational design, employs an unconventional sampling technique to derive policy-impact statements from a small sample. The policy a&y& perspective of this rest&r& is focused on making prescriptive rather than descriptive statements. The specific research is the "URBIS" proiect, a study which is evaluating, the impact of automated information . .
-_ systems upon the operations of local governments. After briefly discussing the general logic of public policy analysis, this paper details the methods and critical issues of this research design. The paper suggests that the URBIS research strategy might be viewed as one general model for public policy analysis.

PUBLIC POLICY ANALYSIS
Generally, public policy analysis attempts to specify the relationships between some phenomena which are classified as "public policy" and some characteristics of the environment within which that policy exists. "Policies" are the decisions a government unit makes regarding a course of action or inaction-to spend X dollars on service,, not to fluoridate the local water supply, to undertake a military action against another country, to reorganize a bureaucratic unit. The analysis of such public policies can be usefully characterized on two dimensions: research perspective and research design.

Research perspective
The perspective of empirical public policy research normally is descriptive. That is, the committment is to describe and explain that which "is", and one of two explanatory approaches usually is employed. One approach attempts to explain the existence of public policy. Some researchers, including Allison [ 11,Crecine [2], Snyder and Paige [3], Moynihan [4], Wildavsky [5] and Wanat [6], focus on the processes internal to the governmental units which produce policy. Other researchers who take the same general approach, such as Dye [7], Hayes [8] and Long [9], examine the extra-  [13], Cook[l4] and Gibbs 1151.
At a fundamental level, however, policies are behavioral positions which are consciously taken in order to achieve some particular impact. Thus, it is odd that empirical public policy analysis is seldom directed explicitly to the question: What public policy (P) should be undertaken to achieve desired impact (I) given certain conditions (C,, . . . , C. ) ? It might be that this question is avoided in social science policy research because it is viewed as "normative." The thrust of positivism is to avoid the issue of what ought to be. And yet public policy inquiry can, in principle, be both empirical and prescriptive. The policy question above can be reformulated in a form of the classic scientific mode of explanation: If P, the I under conditions C,, . . . , C..

Research design
Public policy studies employ a variety of research methods. The unit of analysis varies in level from the individual to the nation, or even to the supra-national organization. Data can be collected from direct observation, from interviews, from archival or aggregate data, or from unobtrusive measures.
A critical methodological decision in public policy research concerns the number of units of analysis. The number of units can range from one (the single case study) to a handful (comparative case studies) to a very large number (the "demographic approach'?). Normally, as the number of units increases, the ability to generalize and to test hypotheses is enhanced. Conversely, small sample analyses often are selected where hypothesis generation is important, where a richly textured, extensive data base is desired, or where limited units of analysis are dictated by research feasibility. Trade-offs among these factors are characteristic of most decisions regarding the number of analytic units.
Despite these variations in level of analysis, number of units of analysis, and sources of data, most examples of public policy analysis share a basic research design. They are ex post facto analyses.5 The existing relationships among phenomena, conceptualized as variables, are taken as given. One measures these variables at one or more points in time. Then the analyst assesses linkages between the (dependent) variable to be explained and the (independent) variable(s) that might explain it. Thus the object is to describe/explain those configurations of variables which exist in the population (or a representative sample) of the units of analysis.
The experimental research design,t advocated by Campbell[22] for evaluating social reforms, is the basic alternative to ex post facto designs. Ideally, conditions are controlled so that only one independent variable (the "treatment") is manipulated to examine how it affects the dependent variable. This means that the researcher can control (usually by random assignment) the exposure of units to treatments, limiting the problems of "selfselection" to treatments and limiting the effect of unmeasured variables. More fundamentally, the experimental design allows the researcher prior control over the configuration of variables and the hypotheses being examined.
While experimental designs are desirable, there are few examples of their use to analyze public policy as noted in Campbell and Stanley [20]. The obvious difficulty of such designs is that the natural setting of governments and their environments do not allow prior manipulation of systemwide variables by the researcher. Even the natural occurrence of circumstances facilitating "quasiexperimental" design& is rare and unexpected, as noted by Campbell and Stanley [20] and Caporaso and Ross [26]. For these reasons most public policy research is ex post facto.

The URBIS research strategy
This paper presents and analyzes the perspective and design of the URBIS [Evaluation of Urban Information Systems] project[27] as a strategy for public policy analysis. The perspective of the research is to provide empirical and prescriptive policy analysis. The research attempts to deal imaginatively with the policy-impact configurations that exist in current city governments in order to specify those policies which could be enacted and those impacts which could be expected in a set of "future cities." That is, it aims to identify the mix of policies which would facilitate desired impacts, given certain characteristics of the task and the environment.
The design of the research is an amalgamation-of ex pod facto and quasi-experimental designs; of large sample and small sample analysis; of aggregate, survey and unobtrusive data. The most unconventional aspect of the design involves the sampling strategy, which is illustrated fully in the sections below.
Briefly, the design is as follows. Policies (and policy mixes) relating to automated information systems are considered to correspond to "treatments" in an experimental design. Different mixes of these treatment are expected to vary in how they affect impacts-that is, the local government unit's decision making, service delivery and work environment. In order to better estimate the policy-impact relationship within an ex post fucto design, tExposltions supporting the advantages of experimental research designs may be found in Fisher[l9] and Campbell  the units of analysis are selected by a factorial sampling strategy. This provides a systematic array of treatments which, controlling for other conditions, can be analyzed in relation to impacts.

THE UBBIS RESEARCH PROBLEM
Conceptual framework for the problem The URBIS research seeks to provide practical advice to local decision makers regarding choices they can make related to automated information systems. The research can provide this advice if it discovers the relationships between impacts on the one hand, and local government policies related to automated information systems on the other hand, given certain environmental characteristics. Thus, the research involves three major sets of variables: controllable policies, policy impacts and policy environments. Figure 1 illustrates the pattern of relationships to be investigated and some of the major variables in each set.

Controllable policies
"Policy" means a position that decision makers can take with respect to certain behavioral or organizational arrangements that the government can control. Thus the research seeks to identify the controllable policy variables most important to effective automated information systems and to specify preferred strategies with respect to them. The controllable policies, within which the information system operates, are either organization-based or technology-based.
These policies governing the organization and management of information technology within the municipal government are called organization-based policies. Some illustrations of these policies include the following: Centralization of control over computer hardware facilities, system use, and data.
Management support of the information system. User involvement in the information system. Technical training of EDP and user personnel. Charging for computer usage.
Technology-based policies are characteristics of the EDP technology configuration. They can be viewed as intervening variables between organization-based policies and the policy impacts. They can vary as to pattern and amount, but these variables are not subject to such direct control and the organization-based policies. For example, sharing data or computer equipment ordinarily would not occur in the presence of organizational restrictions requiring dedication of computer equipment or data. Similarly, it would not often occur in the absence of centralization of control over facilities and data. These technology-based characteristics are considered policies because they can be altered over time by the action of decision-makers. Some illustrations of technology-based policies are: Sophistication of computer equipment and software. Degree of computer resource sharing. Degree of consolidation and integration of data bases between functions.
Extensiveness of use of automated information in a government function.
The local government's organization-based and technology-based policies tend to determine the specific policies applied to a given information processing task, such as budget preparation or traffic ticket processing.? However, within each city, certain policies may vary across tasks, e.g. the degree of user involvement for designing automated budget preparation may vary from that for designing traffic ticket processing. Thus, although task-specific policies are generally identical to the organization-and technology-based policies, some policies may be found to vary across tasks within the same city.
Wee the text discussion below and footnote t on p. 205 for an explanation of the information processing task (IPT) as the primary unit of analysis in this research. fThe chorce of "impacts" over "outcomes" represents a difficult compromise over what and where to measure. One might measure effects at the point of interaction between governmental service-provider and citizen-client. Or, one might go directly to citizens and measure their perceptions of the effectiveness of government services. Then, one would attempt to link these effects on the variation in these perceptions with variations in policies.
The examination of outcomes in the community has not been undertaken, on several grounds: (a) only a few information processing tasks, such as utility billing, are directly related to citizens and therefore there would not be much to study; (b) in most cases, the impacts of an information system would not be appropriately evaluated by talking directly to citizens since the general citizenry could not accurately comment on the type of information received nor the work environment of government officials; and (c) a reasonable approximation of the technology's impact on government services (though not on social justice or quality of life) can be obtained using information available within the government.

Policy impacts
"Policy impact" means an effect of information technology in the government administration and is distinguished from "policy outcomes," which are effects in the community environment. Ultimately, local govemments must be evaluated on the basis of their policy outcomes, in particular, the social justice and quality of their provision of public goods and services to the citizenry. However, the URBIS research does not directly examine the government-citizen interface. This is because information technology currently is used primarily as a means to improve the internal operations of local governments, and therefore, the effects of information technology are most directly linked to the government's decision-making and service-delivery activities.
In particular, information technology might enhance the capabilities of the government to make informed decisions and to manage the provision of public goods and services. Improvements in these aspects of government activity also might produce improved community outcomes. At the level of community outcomes, the role of information technology is indirect and inextricably combined with the effects of other programmatic factors. Thus, the URBIS research framework limits its empirical focus to policy impacts, which are more directly linked to the use of information technology. This focus on policy impacts, rather than ultimate policy outcomes, is modest, but seems the most appropriate perspective for examining the importance of current information technologies.$ Since the government's policies for information systems primarily influence the performance of departments and agencies which use information technology, policy impacts are determined by observing the relationships between the policy variables and the behavior and beliefs of those using the technology. The impact variables of interest include impacts on decision making, service delivery and work environment. Impacts on decision making are estimated by measuring such factors as the range, amount, accuracy, availability, utilization, and perceived contribution of information provided to users. Impacts on service delivery are examined by estimating the costs of performing tasks, the time users take to perform tasks, the degree to which automated information is perceived to facilitate provision of service, and the extent to which users now perform services not performed prior to automation. Impacts on work environment are assessed by measures of user job satisfaction, perceived time pressure, task routinization, task supervision and promotability requirements.

Policy environments
Since the same information system policies are not likely to be preferable under all circumstances, the research is also concerned with environmental variables which measure the non-EDP policy factors that significantly affect information systems. The aim is to discover how information system policies are shaped by the social and the political environment in which they occur.
Three major sets of environmental variables seem important to the research. The political and administrative variables include such attributions, and the professionalism of elected officials. The community attributes include such factors as population size, city growth rate, city age, and the socio-economic heterogeneity of the population. Extra -community -based policy instruments refer to policies outside the city, such as the level, types and restrictions attached to federal funding, which might shape local policies.

Prescriptive policy
The perspective of prescriptive policy analysis is to answer the question: What public policy (P) should be undertaken to achieve desired impact (I) given conditions (C ,, . . . , C,)? The preceding sections have identified the variables used to characterize the concepts of policy, impact and condition and have explicated the basic nature of the linkages between these variables. Even if research can accurately and comprehensively characterize the linkages, policy prescription is contingent upon a determination of the desired impact. For example, expenditure control might be a desired impact for financial management in City,. Assume the research reveals that an on-line, real-time computer application in the finance unit increases the capabilities for budget and expenditure monitoring, independent of other measured conditions (e.g. centralization, user involvement in design). In this case, City, could be advised to implement an on-line system in the finance unit.
It is often the case, however, that policy prescription will be more complicated. In some cases, the desired impact might be linked to different mixes of policies and might obtain only under certain conditions. In such situations, prescription will be conditional and probabilistic, rather than simple and straightforward. It is also likely that some desired outcomes will conflict with others. The typical objective of political decision makers is to balance competing goals rather than to maximize a single desired impact. The use of computers by a supervisor to monitor the work of subordinates might enhance the supervisor's decision making capabilities; but the subordinates might perceive that the quality of their work environment has deteriorated. When desired impacts conflict, this research can specify the trade-offs, given certain policy mixes and environmental conditions. But the research will not specify either the particular impacts, or the balance of impacts which ought to be preferred. Thus, in a strict sense, we shall provide information about the impacts of policy alternatives rather than prescribe choices where impacts are in conflict.

Problems in common with other policy studies
As the conceptual framework suggests, this research problem has much in common with most policy analyses. First, there is a large number of policies to be evaluated. To give useful advice to decision makers, one must address a significant number of the choices they face. Thus, a design for policy analysis should allow for the assessment of a number of independent, dependent and control variables. Second, most research questions which involve multiple independent variables also face problems because these variables are highly interrelated. This makes an assessment of their independent impact difficult. Policy designs should anticipate highly related policies (dependent variables) and should attempt to facilitate an assessment of their independent effects. Third, this research involves multiple levels of analysis-from the government system down to the individual user. As with many policy analysis problems, a design must be capable of systematically relating multiple levels of analysis.
A fourth common problem is that some of the policies to be investigated are rare. The most interesting policy innovations sometimes are adopted in a small number of places, if they are adopted at all. For example, charging for computer usage is an interesting policy. If few municipalities utilize direct charges for computer usage, then this policy becomes more difficult to evaluate. An adequate study design attempts to identify and include those sites in which such rare policies have been instituted.
Finally, a design must be feasible in terms of both cost and the conditions to which subjects will agree. If we wanted to do a policy study of six particular policy variables and were unconstrained, we might draw a large number of cities at random and then assign a policy combination to each city at random. Moreover, we might supply each city the resources necessary to pursue the assigned policy combinations for a few years, regardless of the consequences. Since such an experiment is impossible, we must settle for whatever approximates it in the natural world.

3.THE RESEARCH DESIGN
The URBIS research design addresses these foregoing problems through a fusion of research methods that constitute a modest design innovation: an ex post facto design aimed at policy prescription; a small sample in combination with a highly-stratified sampling strategy focused on multiple policies; the selection of particular cases which have extreme values on the key policy variables; the choice of a unit of analysis that facilitates multiple levels of analysis; and a mixture of data collection and data analysis strategies. The relation among these methods can be seen in an overview of the research plan.

Research plan
The general plan is a two-phase research program. Phase I is a census survey, by self-administered mail questionnaires, of chief executives and data processing installation managers in U.S. cities with a population of 50,000 and above.t The Phase I survey instruments provide measures that characterize each city's policies towards information technology and its political/administrative environment.
Phase 11 is a cross sectional study with a sample of 40 cities and utilizes personal interviews, self-administered questionnaires, census and archival data. Cities then can be compared "at one point in time" on each variable of interest. The cities are selected through a two-stage sampling strategy which stratifies the population of cities on the major policy relevant variables. Within each city, multiple levels of analysis are distinguished including: individual users, six information processing tasks common to the 40 cities, the operating departments which perform the tasks, the data processing installations, and the municipality. Data are collected from 50 respondents at varying levels in the administrative hierarchy of each city government, and the questionnaires provide measures of policy impacts on each government's decision making, service delivery and work environment.

Policy prescription for future cities
The research design is aimed primarily at permitting the researchers to prescribe policies for "future cities" rather than to describe the policies followed by the current population of cities. Specifically, the research seeks to discover the effects of various configurations of policies. Therefore the design departs from conventional surveys in two ways: (1) it samples at the extremes of policies rather than randomly, and (2) it samples primarily on policy variables rather than on environmental variables.
A conventional survey, using a random sample of cities, would permit accurate description of the population of cities in terms of current policies, associated impacts, and environmental conditions. Such a design probably would be adequate for policy prescription if the sample were of sufficient size, if the policies of interest were randomly distributed in the population, and if the effects of each policy were easily distinguishable. However, large samples and conventional random samples do not facilitate the intensive study which is required in an area where previous policy research is lacking.
While a small random sample would facilitate intensive analysis, it has several drawbacks in distinguishing the effects of interesting policy mixes. First, since some important policies occur only rarely in the population, they would probably be excluded from a small random sample. Second, other important policies tend to co-exist and therefore the independent effects of such policies would be difficult to distinguish with a random sample. In tPhase I also gathered data for all counties with a population greater than 100,000. This Phase I data for cities and counties is a rich data base for analyzing the current state of local government computing. Representative analyses include: Kraemer  order to differentiate the impacts of these policies it is necessary to examine cases where the policies do not co-exist. To the extent these cases of non-coexistent policies are rare, they too would tend to be excluded from a small random sample. Third, the impacts of policies might be confounded by eivironmental factors. To discern the impact of particular policies, it is valuable to maximize the variation on the policy variables. Then, if the policy does have an impact, it will be discernible from the "noise" generated by other factors. Since random samples are not likely to maximize the variation, they reduce the researcher's ability to distinguish policy effects from these confounding effects. The key to achieving these requirements of the research design is the sampling strategy.

Stratification to maximize policy variation
The research employs a severe form of stratified sampling designed to fashion what we can find in the natural world toward an array of policies whose impacts are important to evaluate. As such, the technique of stratification departs significantly from its conventional use. Stratification, a well-accepted aspect of sampling theory, normally is employed to increase the power and efficiency of a sample, as noted in Leege and Francis [ 171 and Kish [30]. However, as sample size becomes smaller, the feasibility of extensive stratification normally diminishes. Hence, small samples rarely are stratified on more than one to three variables.
This research stratifies on a substantial number of variables (6) despite a small sample size (40), because the purpose of stratification here is to assure an adequate distribution of cases on variables of major theoretical interest, rather than to maximize efficiency. And, it is to assure some independence among policies that might be highly related in the existing population.
The population of cities is stratified on the six variables deemed to have substantial theoretical impact and policy relevance. These six are as follows: Degree of automation. Sophistication of EDP system. Degree of integration. User involvement in design of EDP applications. Decentralization of EDP. Charging for EDP services.
The stratified sampling process involves four stages. First, the universe of cities is defined and bounded to yield the sampling population. Second, the population is dichotomized on the basis of six policies. Third, sites are further partitioned to represent the extremes for each policy variable. Finally, cities which maximize the differences on the policies are. selected. Each stage is now elaborated as it was actually implemented, along with specific decisions that must be made to apply the process for other studies.

Definition of the population
The original universe of cities from which the 40 sites were chosen is composed of all 403 incorporated cities over 50,000 in population in 197O.t Values for all stratification variables were obtained for all but 16 of these cities through the following estimation procedure. Demographic and expenditure data obtained from the U.S. Census was regressed on the stratification variables for the cities responding to the URBIS census survey (about 80% of all cities). From these regressions, equations were formulated to estimate the values of non-responding cities for each of the stratification variables. This procedure yielded a population of 387 cities with either real or estimated values.
This population was limited further by excluding cities with fewer than 12 computer applications. We set this lower bound on the degree of automation because most policies are relevant to our analyses only when the government has a minimal degree of automation. This limitation yielded a final population of 310 cities.

Stratification on policies
The population of 310 cities was partitioned into 2" = 64 strata by making each policy variable dichotomous. That was done by constructing an index for each variable. All values of the index smaller than a selected cutting point (dependent upon the distribution of values) on the index were treated as low values of the variable, while all values above the cutting point were defined as high values. Thus, on any variable, the population of cities was split into two groups; the six splits define the 64 strata.
The indexes were constructed from appropriate census questionnaire items using weights chosen judgmentally by the investigators and scaled from 0 to 100. Cutting points were selected from inspection of the distribution of cities on each stratification variable. If the distributions had been normal or bimodal at each extreme, the median would have been chosen as the cutting point. However, inspection of the distribution of cities on the URBIS stratification variables led to the decision to dichotomize each variable at the third quartile. Figure 2 illustrates the kind of skewed distribution each index tended to approximate. Thus, dichotomization tended to group cities on each index into those with common policies and those with relatively rare extremes.
After all six indexes had been constructed and all sites labeled 0 to 1 on each of the six indexes, then every site was characterized by a sequence of six zeros and ones, such as "I 10010," which indicates that a particular site is in the high half of the population of the first, second and fifth variables, and in a low half on the third, fourth and sixth variables. The 64 strata were labeled by these sequences of O's and 1's. tFor example, if there are some but not many (say less than eight) empty strata, one would employ the above strategy. But, if the iirst randomly selected replicate contained an empty stratum, one might reject it and keep selecting replicates until one with no empty stratum was found. With still more empty strata, say 8 to 12, one might systematically search for a non-empty replicate.
SIf there are 12 to 24 empty strata. one might best select a l/2 replicate (32 strata) or 40 balanced strata at random and then fill each empty stratum in it by moving a site closest to that stratum into the empty stratum. In this case, it is necessary to define "close." Suppose 001011 is the label of the empty stratum. There are six adjacent strata whose lables are obtained by changing (in turn) each 0 to a 1 and each 1 to a 0. Among these six strata we would choose the one containing the most sites. Suppose its label is 001111; then, we would choose the site in that stratum which has the smallest value on the fourth index and move the site to the empty stratum. Of course, if the most populous adjacent stratum were obtained by changing a 1 in the empty stratum to an 0, then the site moved would be the one having the largest value of that index.
If more than 25% of the strata is empty, we would probably give up the idea of stratifying on six variables and stratify on only five, thus partitioning the population into 2' = 32 strata which, hopefully, would be mostly occupied. If we were primarily interested in determining properties of the population of sites, we would simply draw one site at random from each of the selected strata. Since our primary interest is to determine the relationships among a special array of policy configurations, we propose the non-random selection of elements from the extreme cases within each stratum, The purpose is to increase the likelihood that any effects of the policies will be discernible, especially given the "noise" from other variables that inevitably cloud the measurement of the policy-impact relationship. It is more important to know whether or not a policy has an effect, than to discern the full range of effects possible from minor variations in the policy.
In choosing a site from a stratum. therefore. we would select the one site that has its indexes for the policy variable nearest to their extreme values. The process is to calculate, for each site, the six differences d,, dz, d,, d4, d,, d, obtained by either subtracting the index from 100 if the index is above its median, or subtracting zero from the index if the index is below its median. The site whose largest d had the smallest value among all largest d's in the stratum would be chosen.
To illustrate the process for only three variables, suppose there are eight sites in the 001 stratum with indexes as shown in the left three columns of the following Table 1. The three right-hand columns are the corresponding d's. Examining the eight triples on the right, we find that the fourth one has the smallest maximum (19). Hence the fourth site would be chosen from that stratum.
While we suggest the above strategy for selecting sites from cells, the URBIS study randomly selected sites from cells for two reasons. First, most cells were sparsely populated, usually having two to four cities. Second, index values within cells often so closely approximated one another that extreme value selection was not considered useful.
tThe "object unit of analysis" is the unit whose behavior is to be explained. It is distingutshed from the "subject unit of analysis," which is the unit whose characteristics/behavior are observed or measured. See Eulau [32].
SThere are about 300 such IPTs covering the full range of services provided by most municipal and county governments. However, this study does not attempt to deal with all IPTs provided by each government. Rather, tt is confined to a smaller set of six IPTs, or "IPT types" common to all 40 cities. As elaborated in Danziger [33]. most information processing tasks fall into six general "types" based on their function in relation to information. The types are distinguished by the primary modality of their use: (a) Recordkeeping: Activities which primarily involve the entry, updating and storage of data, with a secondary need for access; the computer facilitates manageable storage and easy up-dating for nearly unlimited amounts of Information. (b) Calculating/Printing: Activities which primarily involve sorting, calculating and printing of stored data to produce specific operational outputs: these utilize the computer's capabilities as a high speed data processor. Activities which involve reorganization, reaggregation and/or analysis of data; the computer is used to link data from diverse sources or to summarize large volumes of data as management and planning information. (f) Process ControL : Activities which approximate a cybernetic system; data about the state of a system is continually monitored and fed back to a human or automatic controller who steers the system towards a performance standard; the computer's capabilities for real-time monitoring and direction of activities are utilized.
"Task" as the unit of analysis To maximize variation on the policies of interest within the cities selected, it is necessary to identify theoreticallymeaningful units of analysis. A focal point for our substantive interests is the city's automated information system. Information systems in cities are inherently complex because they are essentially a collectivity-of people as users and providers; of information processing activities (i.e. data collection, storage, retrieval, manipulation, summarization and analysis); of equipment and facilities comprising a data processing installation; and of organization and technology-based policies that control the other elements. Because of their complexity, information systems cannot be analyzed as a single entity. Rather, they must be analyzed in terms of some elemental units.
A further complication is that the policies which shape these automated information systems can affect various levels: (1) individual users or providers, (2) the data processing installation, (3) the user department, (4) the city government and (5) intergovernmental units. Therefore, the design needs to accommodate these multiple levels of analysis. The problem, as conceptualized by Eulau and Prewitt [31], is to find a suitable unit of analysis which enables us to measure variables at other levels of analysis and then reconstruct these variables at the level of the analytic unit. While such a procedure requires careful specification, it is the most appropriate technique for examining phenomena that must be measured at different levels of analysis. It is also possible, of course, to collect and manipulate data so that more than one primary object unit of analysis? can be examined. Thus in the URBIS design, cities are the primary unit for purposes of sampling, individual role-takers are the units providing most of the data, and both the "Information Processing Task" (IPT) and the types of role-takers are the primary object units of analysis.

The Information Processing Task
An "information processing task" is a term used to signify a particular type of activity undertaken by the municipality. The activity has an identifiable functionthat is, it fulfills a specifiable objective. It explicitly involves information processing and might be automated. For example, most cities regularly issue a payroll which involves translating records of hours worked, pay rates and payroll deductions into a payroll check. Thus "payroll processing" can be designated an IPT. Similarly, the searching of wanted-warrant records on a criminal suspect being questioned by a patrol officer in the field is the IPT, "patrol officer support.7 These information processing tasks represent the primary unit and level of analysis in the research design. They allow meaningful characterization of variations in outcomes relative to various mixes of information policies with regard to information systems. For example, "patrol officer support" may or may not be automated and a variety of computer applications could support this task. It makes sense, therefore, to evaluate the speed, cost, effectiveness, work environment and information availability for individuals performing this task as a function of the technology-based policies.
For some theoretical interests, the role-taker (or an aggregate type of role-taker) rather than the IPT will be the object unit of analysis. Hence, we might compare the relationship between measures of perceived quality of the work environment and sophistication of EDP for different types of role-takers (e.g. clerks vs managers) or for role-takers with different professional orientations and different length of job experience.

Multiple levels of analysis
The IPT also allows comparability of a task across cities. It is evident that city-wide data processing policies are identical for all IPTs performed in the municipality. Installation-specific policies and technologies generally are similar for those IPTs and users serviced by the same data processing installation. Operating department policies are similar for all IPTs and their users within that department. Thus, each IPT has certain IPT-specific attributes (e.g. the types of computer applications assisting this task), as well as a number of contextual variables at the department, installation and government levels. The fact that these attributes are common to any single IPT allows comparison of the users of a particular IPT across municipa1ities.t Figure 3 illustrates five levels of analysis along with the types of relevant variables at J Fig. 3. Levels of analysis for URBIS.

Data collection
Each method of data collection has certain weaknesses [33]. Therefore, a strategy of the URBIS project is to integrate a variety of data collection techniques.
Data will be collected from about 50 respondents in each of the 40 cities. URBIS investigators will gather documents, conduct semi-structured personal interviews, gather lists of users for systematic sampling and deliver self-administered questionnaires to each respondent.
iIt is unlikely that IPT findings can be used to generalize about all IPTs in a department or city. The behavior and experiences of the users of six IPTs are an inadequate basis for making general inferences to department or municipal impacts. This is because it is unlikely that these users will comprise a representative sample of all users of information processing within a department or municipality.
STechniques similar to the multitrait-multimethod matrix will be used for evaluating the scales and indexes. See Campbell and Fiske[21].
Managers and technical staff of the data processing installations will be the sources of most measures of the policy variables. Chief executives, departmental managers, supervisory staff and line personnel doing the relevant IPTs will be the major sources of most measures of the dependent policy impact variables-user behavior, beliefs and attitudes relating to decision making, service delivery and work environment.
The particular IPT largely will determine the primary role-takers who should be sampled. For example, a "criminal incident report" task is utilized by the patrolman, detective, precinct captain, police chief and others within the police department. Given the particular IPT, all those occupying the formal roles involved with the task will define the interview population. Each role will be weighted by the number of individuals occupying the role and by a rovgh index of the importance of that role to the IPT. Systematic sampling procedures then will be used to select a sample of specific users within each ro1e.S There are other data sources beyond the selfadministered questionnaires. First, Phase I questionnaires provide a varied data base reflecting the general characteristics of the city's EDP operation, its administrative and political style, and the chief executive's perceptions of data processing. Second, U.S. census and other archival data will be utilized to tap attributes of the political, administrative and s&o-economic environment of the municipality. Third, a number of observations are planned, such as obtaining sample reports and computer printouts. These will enable us to assess the quality of information or types of services provided. Fourth, unstructured interviews and observations of the investigators during field work will add a mixture of what Bruyn [34] calls "participant-observer type" data.

Data analysis
Data analyses will consist of two major analytical endeavors. The first will be the testing of hypotheses suggested by the current state of knowledge and experience about information systems. Figure 1 illustrates the basic pattern of hypothesized relationships among the major classes of variables.
We shall not collect enough data to examine the entire Fig. 1 structure as a single model. Generally, our analysis will examine each functional relationship independently through a variety of analyses, emphasizing regression analysis. By working through the structure systematically, some of the hypothesized connections will be supported and others will not be supported by the data. The overall result should be a considerable specification of the structure. This will enhance our understanding of how the variables relate to each other and will identify variables which seem to be particularly important.
The second major analytical endeavor will consist of inductively examining all of the data in an effort to express the general pattern of relationships more elegantly. This will involve a series of iterations in which we try to develop more general constructs by combining conceptually related variables into more general scales and indexes. This will be accomplished by techniques such as factor analysis.$ These more general scales and indexes will be interrelated to search for significant patterns of relationships not already revealed by the tests of specific hypotheses. Thus, in addition to hypothesis testing, data analysis will employ alternative techniques to develop summary scales and indexes which can be interrelated. The patterns of relationships discovered through this process should identify the most significant features within the mass of data.

1SSUEs RAISED BY THE URBIS DESIGN
In the evolution of the URBIS research, several important design issues were examined extensively. A discussion of these issues might further illuminate the logic of this design for policy research. The issues can be specified in a series of questions which fall into three groups: A. The risk of extreme stratification. A. The logic of extreme stratification 1. Sampling variables on extreme values. The selection of the extreme-values case in each stratum is guided by the logic of the design and it directly benefits data analysis. It is quite likely that a large number of factors affect the dependent variables of decision making, service delivery, and work environment. Investigation of the linkage between particular policies and the dependent variables must deal with the confounding effects (the "noise") of these other factors. It seems reasonable to assume that if our cities are near the extremes on each policy, any relationship between policy and impact is likely to be more discernible. If we are near the extremes of each policy and are unable to detect a relationship, we have reasonable evidence for inferring that no clear policy impact linkage exists.
In assessing the relationships between major variables, tThere will not be enough degrees of freedom to rely upon individually calculated error variances. *It is possible that, having selected units near the extreme values on policy variables, these units also will be near the extremes on the outcome variables. This would further enhance our ability to estimate the nature of relationships. §A11 such analyses operate under the assumption that the relationship is basically linear. However, even if it were curvilinear, a random sample would tend to underestimate the linear component relative to an extreme case sample.
'Not only should we be able to determine the existence of a non-linear relationship, but we should be able to estimate the approximate shape of the relationship. contingency tables (e.g. 2 x 2 tables) and most types of analyses employing frequency counts are inappropriate. The fact that we have balanced on certain policies insures that a 2 x 2 table intended to explore dependence between two of the stratifying policies would come out: Thus, we would detect nothing, even though the two policies might be highly correlated. It is possible to do simple contingency analysis between a stratified variable and an impact measure. But, the differences in proportions between categories will be somewhat inflated given that cases toward the median are under-represented. Most variables will be quantified by constructing index numbers and scales which approach an interval level of measurement. Our typical mode of examining the relationship between two variables will be to find the best straight line describing the relationship and to judge whether the slope is statistically or substantively significant. For statistical significance, the primary technique will be the use of a typical (median) error variance, found when impact variables are regressed against the stratifying variab1es.t For substantive significance, we will rely on (1) the relative strength of relationships, (2) the proportion of explained variance, (3) an evaluation of the general pattern of relationships and (4) our judgments regarding the meaningfulness of these relationships from a theoretical standpoint.
It might seem that if variables are represented by extreme values, it would be problematic to assess relationships across the entire range of values. In fact, the extreme-values sample will be particularly good for estimating the linear component of a relationship between two variables-better than a random sample. This virtue obtains because in estimating slopes one does well to sacrifice observations near the center in favor of observations on the extremes.
For example, assume that the data for an impact variable I plotted against a policy variable P had the appearance shown in Fig. 4(a).S In essence, our analysis would estimate a straight line going through the means of the two groups and would declare that the line represents the relationship between Impact and Policy.0 When extreme cases are selected for analysis, it might be that something odd occurs in the center. If, for example, the population (from which Fig. 4(a) is sampled) has the appearance of Fig. 4(b), then our linear estimate would be poor. However, such a poor estimate is quite unlikely for two reasons: (1) since we have a census of the whole population on a large number of variables, it should be possible to discover whether the distribution of policies in the population has an odd shape and (2) since the cities will not be at the extreme on every variable, some cities will have values near the median on any particular variable. Due to this partial spread of values, the sample is likely to resemble Fig. 4(c).
A random sample on the same scale might look like Fig.  4(d); and the whole population might look like several variations of Fig. 4(d) imposed together on Fig. 4(c). In sum, the sample should have the population wellbracketed, and non-linear relationships should be detectable during Phase II data analysis, as well as by comparison with the Phase I census data.7 2. Generalizing to the population. While the extremevalues sample will be excellent for estimating linear relationships, it will be less satisfactory than a random sample for estimating parameters of the existing population of cities. It should be clear that the "future cities" perspective is not concerned with estimating population parameters. However, when necessary, one can estimate them reasonably well by regressing an appropriate variable against the stratification or other Phase I census variables.
The basic strategy for estimating population parameters from the stratified sample of 40 cities can be illustrated by a simple example which assumes linearity of relationships. Suppose we want to estimate the mean of a variable V in the population of all cities (that is, all N cities that we have included in our census). We observe V in the 40 cities of the sample and calculate a simple regression of V on the six policy variable indexes, say PI, Pz, . . . , P6. It will have the form: This would, of course, be the estimate of V for the population of existing cities. If we wanted to estimate V for the population of "future cities" which would result from adopting certain of the six policies, then we would simply calculate V by giving the P's the appropriate values (perhaps 100 for some and zero for others).?
3. Why stratify on so many variables? The design attempts to stratify on as many variables as possible. The sample is not constrained to mirror the distribution of key tIf a variable V happens to be definitely non-linear with respect to one or more P's, then we would employ a method more complex than simple linear regression. We shall have ample opportunity to detect nonlinearity because our selection of 40 cities will pick up a number of instances in which a P value is not far from its median. A brief treatment of non-linear correlation and regression is provided by Blalock [36].
$This is often referred to as the "ecological fallacy." variables in the existing population. On the contrary, the design aims to identify a special and (probably) unrepresentative array of configurations. The purpose of this unrepresentative sample is to better estimate the independent impacts of policies that are often highly related or often rare. The stratification of policies is structured by means of a 2' factorial design. In the ideal research world, k would equal all those policy variables which seem especially important and all those exogenous variables which, according to the analysis of the population of cities, seem to have a strong effect on the impact of the policy variables. The decision that k equals six rather than four or ten is based on the constraints of the real research world. It is our assessment that approximating a l/2 replicate of a 2" factorial design is the maximum variable stratification feasible, given the size of the population and a 40 city sample. And, sample size has been established by the tradeoff between the requirements of adequate data gathering for each type of information processing task in each city and the availability of resources for research. . The brief answer is that with careful specification, one can construct variables at one level of analysis from data collected at other levels. Given our interest in policies related to information systems, the sampling procedure is based on city-wide policies. Most of these policies are likely to have comparable impact on all units of analysis. For example, sophistication of the EDP system is a contextual variable for the operation of each IPT and should affect each IPT in the same manner. The strategy does make the assumption that in cities where a policy is generally high or low, it will be rather high or low for the specific IPTs examined. Few of the stratifying variables are, by their nature, likely to have substantial within-city variation. And of these few variables, there is an even lower likelihood that they vary across the particular IPTs examined.
It might be, however, that a stratifying variable does not have the same value for each object unit of analysis. For instance, in a certain city it might be that the user involvement in design is high on four IPTs but low on two others. The key point is that data-gathering techniques are flexible and allow the generation of a rich data base. Thus the level of user involvement will be measured specifically for each IPT and thus there will be valid data for the relevant analyses. In this manner, the analysis will avoid the fallacy of assuming that contextual variables apply uniformly to individual cases.$ 2. Is a small sample adequate for large N analytic methods? An obvious concern for those familiar with survey research is the rather small sample size. It is appropriate to question whether a 40-case sample is adequate for useful analysis. There are several levels at which one can respond to this concern. A broad "quasi-response" is to note the trade-off, in most social science research, between intensive case study analysis and a large N study. The latter normally allows for more rigorous analysis and generalizations at the cost of a substantial loss in the richness of data and insight. The URBIS design, in the attempt to generate a rich data base while expanding the N towards the budgetary limit, is a compromise likely to make proponents of each approach uneasy. But, to the extent the analytic methods are valid, it seems defensible as what Verba [36] calls a "disciplined configurative approach." The design has been discussed continually in terms of 40 cases. This is. for many analytic purposes, a conservative estimate. The primary object units of analysis are normally IPTs, of which there are 6 x 40 = 240 cases. If there are differential impacts between IPTs, the number (240) ~111 drop. For every key variable that is measured at the city-level of analysis and does not vary across IPTs, the number will drop. But, only under the most stringent assumptions will analysis be limited to N = 40. These same observations are equally true when types of role-takers are the object unit of analysis.
Even if we take the most limited case where N = 40, it IS possible to argue that the sample is not as small as it seems. In examimng the relationships between variables, we shall look at the effect of other variables. One method is the use of one or more control variab1es.t If we use physical controls (for example, if we take high/low EDP expenditure as a third variable in contingency analysis), the number of cases in the cells is likely to diminish quickly.
However. our techniques for analysis normally will employ more complex mathematical and statistical methods of control. If correlation techniques are used, multiple and partial correlations provide statistical methods to control or include third, fourth, and so on, variables without such severe sample size problems. In most cases, we expect to examine the relationships between variables by means of regression analysis. It is obvious that careful and insightful interpretation of regression coefficients will be required, particularly where multicollinearity or degrees of freedom become problematic. But, we argue, cautious analysis of such a sample is quite valid. Moreover, the benefits of sampling extreme-values cases must be reiterated. Since the more extreme scores are more useful in linear estimation, it is "as if" we were analyzing a substantially larger sample than the actual number of cases implies. In sum, it seems reasonable to assert that even where the sample is small, it is data rich and it is able to support rigorous analytic techniques.
C. Assessing the impacf of policies 1. Giren other factors a$ecting decision making, service delivery, arid work enzlironment, can the consequences of informutiorl policies be identified? This research assumes that organization-based and technology-based policy instruments related to information systems are only a subset of the factors affecting the quality of decision tOn the techniques of elaboration, see Rosenberg [38].
$The uw of wlthu-government respondents rather than citizens means that the data will not exphcitly measure the direct benefits or costs to those citizens. However, it seems reasonable that if IPT users are clearly affected by Information policies, it is possible to mfer how cltlzen-clients might be affected. making, service delivery and work environment in local governments. How are we to assess the impact, if any, of the information system variables among this broader array of factors? Specifically, are the survey data adequate to determine whether policy-impact relationships do exist? Several aspects of the Phase II design attempt to be responsive to this problem.
First, we assume that the "signals" indicating a policy-impact relationship will be augmented by examining cases near the extremes on each policy. Second, impact measures are generated by interviewing those with the most direct experience of the information systemthose governmental actors who are closely related to each IPT. Thirdly, by focusing the respondents' attention to particular Information Processing Tasks, more valid measures of performance can be obtained. Since the respondents will be those most familiar with the information system and since the question stimuli will be quite focused, their responses should be near the policy-impact interface.+ The fourth characteristic of the design that facilitates assessment of the policy-impact relationship also concerns the restriction to a few particular IPTs. With a limited number of well-defined foci, the research can be sensitive to other major, non-information-system variables. Insight into each particular IPT enables us to identify plausible rival hypotheses to the impact of information system policies. By measuring these other variables, the analysis can control, or at least account for, their effect. If, for example, one examines the IPT of "patrol officer support" involving search of wanted warrant records, one indicator of decision making would be the proportion of times an officer, having stopped a motorist, checks the warrant file. While various characteristics of the police information system are likely to affect this measure, other variables are also quite important. For example, the existence in City, of a well-articulated standard operating procedure (e.g. concerning certain conditions under which the wanted warrant file should be searched) might have a substantial influence on the behavior of officers. In sum, we suggest that it is possible to assess the relationship between information policies and impacts, despite a complex, "noisy" environment. In particular, the design employs careful measurement of responses by those most knowledgeable about well-defined Information Processing Tasks.
2. Is it valid to link information system policies and impacts at one point in time? Most cross-sectional analyses, including the Phase II design, must assume that the variables related on the basis of current values have had a rather similar relationship through time. One might expect, for example, that a city with a large number of on-line terminals decentralized to user departments would be characterized by more extensive use of automation than a city with batch operations. But it might be that in one city many on-line terminals were only recently installed and are not yet affecting extensiveness of use. One can imagine instances where the current configuration of policy variables affecting impact variables is different from that at earlier time points. In such instances, inferences about policy-impact linkages based on current patterns might be misleading. This problem is especially important to research examining local government EDP, because some EDP environments evolve rapidly. Moreover, there normally is a time-lag between the implementation of an EDP policy or technology and a measurable impact on decision making or service delivery.
Still, there is good theoretical and empirical evidence supporting the assumption that, ceteris paribus, one of the best predictors of what a city currently is doing is what it did several years ago. First, most policy seems to develop incremental1y.t Even with the changeability that characterizes data processing, it seems that within city variation over time will be less than between city variation. This is a safer assumption if, as Phase II design proposes, we get out on the extremes of the various policies to be investi~ted. Cities currently at opposite extremes on one of the policies are likely to have differed in the same way several years ago. Second, focusing on specific IPTs and interviewing EDP and user personnel facilitates sensitivity to those situations where there has been a significant alteration in the policy-impact linkage during the recent past. Third, Phase I census data can be compared with Phase II data to locate radical changes which have occurred during the past two years.S 3. Can an "ex post facto" design mercome i~e~roble~ of aeon-seiecfio~ of policies (f~euf~e~f~)? One of the major characteristics of an ex~r~ental design is the ability to randomize the introduction of treatments to subjects (the units of analysis). With this capability, the confounding effect of any other variable is randomly distributed. In contrast, most macro-level social science research begins with natural settings, where the introduction of the treatment to subjects cannot be controlled. The problem is that there are likely to be important interaction effects between the treatment and other characteristics of the unity of analysis. In particular, each unit (in our framework, each city) has selected and maintained those treatments (those information system policies) which are most congenial with other important variables (other im~rtant characteristics of the local political system and its environment). Thus, one expects to find systematic effects between treatments, other measured variables, and other unmeasured variables. The complexities of these interactive effects make analysis quite difficult.
It is reasonable to assume, for example, that a city is more likely to adopt and maintain data sharing arrangements if other conditions are congenial-if it has a centraiized computer installation, if operating departments are not independent and territorial, if the chief executive stresses cooperative arrangements. Where these conditions are not present, the survival and/or success of data sharing is problematic.
It must be stressed that this design does not fundamentally alter the problems related to treatment effects. Although the policy configurations are arrayed so that they tend to be uncorrelated, they are self-selected in each city. Moreover, there are no special controls against the confounding effects of variables other than the stratifying variables. Imperfect control of these other variables is attempted, as in most multi-variate social science research, by the use of statistical controls. While the design cannot overcome treatment-pity interactions, it tThis is well documented in Lindblom [39], Cyert and March [40], Allison[l], Wanat [6], Crecine [2], and Davis et al. [41].
Sit would be desirable to extend Phase II into a longitudinal (panel) study because the design currently does not provide evidence for strong statements about change dynamics or lagged effects. But, cross-sectional measurement and analysis are appropriate for developing valid generalizations about the linkages between policy and impact variables.
does seem that such effects are dealt with more adequately than in most natural setting analyses because of our attempt to select cases which independently distribute the major policy variables. It should be added that self-selection will have predictable and interesting effects. Because cities selecting their own policies are more likely to find them congenial, the treatment-city interactions will be significantly reduced and the main effects correspondingly exaggerated. In other words, an experiment that randomly assigned policies would often assign a policy to a city where it is not compatible. This would tend to reduce the observed main effects. Thus, the relationships we find between policies and outcomes will be stronger than what would be found if the policies were randomly assigned.

A GENERAL DESIGN FOR POLICY RESEARCH
Experimental designs often are advocated as most appropriate for evaluation and policy research. While the internal validity of the experimental design is attractive to policy scientists, there seem to be several drawbacks to the approach. Most significantly, it is usually not feasible to investigate the more si~ificant policy questions within their natural settings as an experiment. Although some investigators have been especially perceptive in identifying situations adaptable to natural experiments, these policy studies remain an exception.
The difficulty of utilizing the experiment to evaluate social policies has resulted in the common use of ex post facto designs. However, studies which attempt to describe the relationships between public policies and their antecedents or their hypothesized consequences have other difficulties. While normally less obtrusive and more widely applicable than experiments, these descriptive studies often fail to identify polity-impact relationships. We believe this is partly due to an over-concern with describing the present, resulting in infrequent examination of cases with rare or extreme policy differences.
In order to incorporate the evaluation of rare policies, the "future cities" design examines a sample which looks quite unlike the present population. The sample is stratified on each major policy of interest in order that its independent impacts might be better estimated. The extreme cases of different policies are selected to amplify the signal of any policy-impact relationship. Also, the sample is large enough to allow the utilization of large sample data analysis strategies; yet it is small enough to facilitate the collection of an intensive and rich primary data base. While the future cities survey design is not without problems and risks, it is a potentially useful approach to other research questions. Finally, it should be noted that the logic of the future cities design is not restricted to policy analysis. In most research problems where the substantive interest is to assess hypothesized relationships between certain key variables, the strategy of extreme stratification and extreme cases might be employed.