Research Methodology in the "Public Administration Review," 1975-1984

Public administration was in an early stage of development when Luther Gulick' called for a "science of administration." Gulick's exhortation became a source of heated and continuing controversy that centers around positivist versus alternative views of appropriate research methodology.2 This paper examines the last ten years of Public Administration Review (PAR) methodology and suggests directions for its future development. Two general methods are used in this study: historical and statistical. Past and current assessments of research methodology in public administration are reviewed in order to set the context for this analysis. Also, published research in PAR from 1975-1984 is analyzed statistically to provide a baseline for evaluation of the state of research methodology. In the final portion of the paper we generate some recommendations for future directions within the field.

Public administration was in an early stage of development when Luther Gulick' called for a "science of administration." Gulick's exhortation became a source of heated and continuing controversy that centers around positivist versus alternative views of appropriate research methodology.2 This paper examines the last ten years of Public Administration Review (PAR) methodology and suggests directions for its future development.
Two general methods are used in this study: historical and statistical. Past and current assessments of research methodology in public administration are reviewed in order to set the context for this analysis. Also, published research in PAR from 1975-1984 is analyzed statistically to provide a baseline for evaluation of the state of research methodology. In the final portion of the paper we generate some recommendations for future directions within the field.

Clarifying Definitions
At the outset we need to clarify some definitional ambiguities. What, precisely, do we mean by research methodology? Do quantitative methods delimit the scope of this subject matter? Do we include activities both academic and practical? Answering these questions requires some judgment, but the literature on social inquiry offers useful guidelines.
In search of a working understanding of methodology, we rely on Kaplan's discussion of this concept in The Conduct of Inquiry.3 He distinguishes several senses of methodology: (1) techniques, the specific procedures used in a given science; (2) honorifics, a ritual invocation attesting to concern with meeting standards of scientific acceptability; (3) epistemology, involving the most basic philosophical questions about the pursuit of truth. It is the first of these senses that has the greatest bearing on this inquiry.
In addition, our primary concern is methodology used in academic research; that is, in the conscious effort to advance knowledge about public administration. Methodologies for administrative research, such as program evaluation, client surveys, and productivity MAY/JUNE 1986 * The adequacy and quality of public administration research has historically generated significant controversy. This study reviews the research methodologies of Public Administration Review articles from 1975-1984. It concludes that recent public administration research: (1) has been predominately applied; (2) has not been cumulative; and (3) has lacked adequate institutional support. Recommendations are offered to improve research methodology in public administration by focusing intensively on core issues, institutionalizing research, and introducing specific methodological improvements. measurement, are focused on generating knowledge about the problems of particular organizations or programs and are excluded from the scope of this study because of their essentially instrumental orientation. They are oriented to the practice of administration rather than to the study of administration.
As the foregoing suggests, methodology and research are closely linked. Methodology exists to guide the conduct of research; methodology is reflected in research. As a practical matter, therefore, our assessment of public administration methodology is also necessarily an assessment of public administration research.

Recent Critiques of Public Administration Research
Recent discussions of public administration research methodology have been characterized by two predominant streams. The first is concerned with the degree to which research is adding to a verifiable knowledge base that we can use to improve public administration as an applied science. The second stream is concerned with methodology issues, that is, the type of research questions that we can pierce with our methodologies, and whether our methodologies produce useable knowledge.
Several recent works have looked at different bodies of research in public administration from the standpoint of their contributions to knowledge. Garson and Over-man4 reviewed public management research, as a subset of public administration research, for the years 1981-1982. They concluded that the research was fragmented, noncumulative, and underfunded. McCurdy and Cleary5 analyzed abstracts from public administration doctoral dissertations published in the Dissertation Abstracts International for 1981. They found that the vast majority of dissertations neither dealt with significant issues nor were conducted in such a way as to produce findings in which one could have much confidence. They concluded that the lack of methodological progress, as evidenced by the low quality of dissertations, is due to inadequate standards among leading public administration programs as well as to the nature of the field itself. Jay White's recent replication of McCurdy and Cleary's6 analysis found that dissertation research is not published and therefore not communicated beyond the dissertation committee. He concluded that whatever the reasons which explain the lack of publication (e.g., poor quality, lack of interest in publishing the dissertation), dissertation research does not appear to be a major source of knowledge in the field.
General research methodologies are essentially restricted to logical argumentation, legal briefs, or empirical analysis.
The second stream of discussions is concerned with the methodology issue, and has been joined by a number of public administration theorists, most notably Catron, Denhart, Hummel, and White.7 White8 argues that most critiques of public administration research have been grounded in models of research predicated on positivism, indicative of the natural and mainstream social sciences. He argues, however, that public administration research has not been viewed in light of two other modes of research-the interpretive and critical modes. He suggests that growth of public administration knowledge be interpreted in light of all three modes of research.
This paper is only able to shed light on this second question indirectly. Its primary purpose is to assess how PAR methodologies measure up against mainstream social science research. Whereas Garson and Overman looked at contract and grant research and McCurdy, Cleary, and White looked at dissertation research, we look at another subset of research in the field-PAR articles.

Research Methodologies in Use
Research articles published in PAR from 1975-1984 are the population for this analysis. Symposia articles, Professional Stream essays, review essays, and special issues were excluded from the domain of analysis. Included are 289 articles. Each was coded on 12 variables, about half reflecting purely descriptive information and the others requiring some interpretation of the contents of the article. These variables are discussed

Analytic Categories
Seven variables provided primarily descriptive information, some of it purely for identification purposes, about each of the cases: year of publication, volume, issue number, author(s), author's organization, general subject area, sources of research support. Four other categories were used to record information about the methodology used in the study.
Research stage is a taxonomic variable derived from earlier work by Rogers and Agarwala-Rogers.9 It represents the stage of social science research, reflecting the purpose for which the study was conducted. These research stages and purposes are summarized in Table 1 area, source of research support, stage, methodology, and methods of analysis. The distribution of research by topic, as shown in Figure 1, confirms the broad distribution of research in the field. No topical area accounts for more than 20 percent of research, but six areas represent more than 10 percent each: administrative theory, public management, public policy, planning, personnel, and finance. Though the data are not shown in the figures, 80 percent of research was problem oriented rather than theory oriented. The low levels of research support are apparent from Figure 2. Eighty percent of the articles failed to identify sources of institutional support. The most important category of sponsorship was "other" which consisted MAY/JUNE 1986 primarily of research funds provided to faculty by their universities. The National Science Foundation supported the largest amount of published research, but it was identified in only about five percent of the articles.
Most articles reported research at an early stage of development, as reflected in Figure 3. Seventy percent of the articles dealt with either problem delineation or variable identification. Although over 20 percent of the articles focused on the relationships among variables, only about five percent of the research was conducted at the three most advanced stages. Figure 4 indicates that the general research methodologies are essentially restricted to logical argumentation, legal briefs, or empirical analysis. Methodologies often associated with interpretive or critical theory, i.e., historical or descriptive approaches," were infrequently represented. Mathematical models or comprehensive literature reviews were employed in less than three percent of the articles. Somewhat surprising is the relatively large proportion (52 percent) of empirical research. Figure 5 indicates, however, that a large share of this Changes in research methodology over time. As a means for identifying changes in public administration research methodology, the data were categorized into two five-year periods, 1975-1979 and 1980-1984. The broad distribution of research by topic during these two periods did not change appreciably, but shifts occurred in the importance of research topics ( Table 2). Administrative theory, citizen participation, planning, and personnel all declined in significance as a proportion of total research. Finance, intergovernmental relations, and public policy increased significantly as focal areas for research.
Few differences for the two periods were found for research sponsorship or research stage. In contrast, general methodologies changed significantly, with much greater emphasis on empirical analysis from 1980-1984. The increase is accounted for almost entirely, however, by expansion in use of case studies ( Figure 6).

Discussion
A question raised by this analysis of Public Administration Review articles is the extent to which this evaluation of the state of research methodology is generalizable to the field as a whole. Other public administration, political science, and management journals which deal with public administration research have not been included in the database. However, given that PAR is a publication of the major professional society whose goal is to advance the art and science of public administration, it is reasonable to assume that strengths and weaknesses of the field are reflected in it.
We think it is useful to note the factors that led us to choose PAR research articles for this analysis. First, PAR has been recognized as the major public administration journal among political scientists as well as public administration scholars.'4 Second, PAR research articles are peer reviewed. This process assures that articles meet broad professional standards. Third, program prestige ratings are highly correlated with publications by faculty in PAR. In their study of reputation and productivity of public administration programs, Morgan, Meier, Kearney, Hays, and Birch,"I found the coefficient of determination between reputation and publication in PAR was around .65. This statistic testifies to the prestige of PAR among academics in the field.
While some might argue for a more inclusive sample, we believe that a purely random sample would produce similar results. As evidence, we cite the substantive correspondence between the original analysis of doctoral dissertations by McCurdy and Cleary and the reanalysis,  Admittedly, PAR is not a mirror image of public administration research. It has a dual set of objectives aimed simultaneously at communicating with practitioners and advancing the science of the field; and this dual set of objectives obviously has implications for the content of PAR articles and the representation of research methodologies published in the journal.
Beyond serving these professional goals of the Society, editorial policy during this period may have had some effect on the content; for example, editorial policy might have been aimed at achieving some kind of MAY/JUNE 1986 balance between academic and practitioner articles. Therefore, generalizations from this research must be made with appropriate caution.
With these cautionary and explanatory notes in mind, we believe three evaluative statements can be drawn from the foregoing findings about public administration research. First, public administration research is primarily "applied" rather than "basic." Nearly threefourths of the articles dealt with either problem delineation or variable identification; less than one-fourth dealt with theoretical relationships among variables. Moreover, the research lacks detachment from immediate and instrumental concerns. Most of the articles reporting on empirical research were of either the case study or cross-sectional survey variety; few articles involved field experiments, structural equations, or longitudinal studies. Finally, the underlying purposes of doing research tend to be problem oriented, which limits development and testing of empirical theory. Eighty percent of the articles in PAR were problem rather than theory oriented. Problem-oriented research tends to reduce the chances that the conditions for sound theory will be met. 16 Second, public administration research has not been Third, public administration research lacks adequate institutional support (university and extramural funding, organized research institutes, collaborative groups, external rewards). The primary indicator of support for research was financial. Eighty percent of the articles failed to identify sources of financial or other institutional support. We believe this omission is not due to poor reporting, journal policy, or author ingratitude. Rather it is an indicator of the low level of such support.
To compare the level of support for public administration research with another professional field, we analyzed reported support for articles published in the Academy of Management's two publications, the Review and Journal, for calendar year 1984. From a total of 105 articles, 40 percent acknowledged some type of support, 22 percent reported receiving extramural support and another 18 percent received assistance from their universities. This is twice the proportion of articles in PAR.
Thus, given the assessments above, we find ourselves The field has not channeled its research efforts; its scope of interest seems unlimited; it has not developed a rigorous methodology; it has been pretty blase about definitions; it has not agreed on any paradigms or theorems or theoretical systems; it has not settled on any stylized jargon or symbols; with a very few experimental exceptions, the field has not been modeled or mathematized into an 'adminimetrics.' In order to advance the status of research methodology in public administration, we believe several changes are necessary: 1. Focus on core issues in public administration. As noted above, public administration is a remarkably diffuse field encompassing contributions from many disciplines. It is also a relatively "small" field, in terms of scholars pursuing its study, when compared with similar fields such as business administration. Public administration, therefore, may need to focus the scope of its scholarship if progress is to be made in understanding phenomena within its general domain. Two core issues could provide this focus: The study of characteristics that distinguish public administration from other administration. This could be attacked as both an issue of political philosophy and as an empirical issue. An approach based on political philosophy would concentrate, as suggested by Woodrow Wilson, on those "public purposes" which define public administration. Political-administrative system interface. The second anchor we propose for a redefined public administration core is the study of phenomena at the interface of the political-administrative system. Among the issues that would be the object of research given this definition of legitimate concerns are: (1) responsiveness or nonresponsiveness to the political system; (2) legitimacy of the administrative system in carrying out its politicallymandated functions; (3) legislative oversight of administrative agencies; (4) representativeness of administrative agencies; and (5) administrative reform, e.g., civil service reform for increasing the responsiveness of administrators to both the executive and the public.
Criteria of research significance. As a compliment to focusing public administration research on core issues, more attention needs to be given to doing cutting-edge research. Both of the analyses of abstracts of doctoral dissertations in public administration arrived at the Although the dialogue about criteria for defining and evaluating the significance of a research topic to the field has not advanced to the stage of producing agreement about such criteria, the development and application of standards about research significance would be a major stride in public administration research methodology.
2. Institutionalize research. It is apparent that public administration research is very much a product of norms and incentives institutionalized within the field. Substantial advancements in research methodology can occur if new norms and incentives are legitimated. These changes would need to include: Upgrade the importance of research in faculty roles.
At the micro level, considerable progress can be made by better developing the capacity and incentives for public administration faculty to do high caliber research. For existing faculty, vehicles such as the Interuniversity Consortium on Political and Social Research (ICPSR) summer program could be used to upgrade faculty research skills. Faculty incentives can be influenced by institutionalizing research as a promotion and merit criterion. The faculty tenure and promotion policies of individual universities could be significantly reinforced by NASPAA's adoption of standards which defined research as a faculty responsibility and required that PA faculty be substantially engaged in teaching, service, and research prior to accreditation. A recent study by Joseph Uveges indicated that NASPAA standards have had a modest impact on MPA curricula and program autonomy. 19 Thus, some evidence indicates that the leverage of NASPAA standards might contribute to institutionalizing research. Increase importance of PhD programs in schools of public administration. A change related to upgrading faculty research roles involves increased emphasis on research-oriented PhD programs. After reviewing abstracts for 142 doctoral dissertations written in 1981, McCurdy and Cleary20 concluded that weaknesses in public administration research methodology were partly a function of inadequacies in doctoral-level research training.
Develop research unit-PA program ties. An expanded emphasis on the value of research would be greatly facilitated by stronger ties between formal research units and public administration programs. Universities considering creation of public administration units Number of Articles should give serious consideration to funding research units at high levels relative to the instructional programs, for example, one-half the program resources. This objective would be advanced by seeking to upgrade the role of governmental research bureaus so that they respond to local needs but in the context of general research interests within the field. Increase funding for public administration research. Without financial resources, adequate research about public administration cannot occur. This is an issue that needs to be resolved collectively by the profession and leading public administrators. The National Academy for Public Administration might initiate a dialog with Congress about the needs for and benefits from research on public administration. Given the scale of the modern administrative state and its centrality in society, Congress might consider creation of a National Institute along the lines of the National Institutes of Health. Within the current fiscal climate such proposals would appear to be inopportune, but the scale of problems of modern public administration could easily justify a moderate amount of earmarked funds likely to repay the initial investment in a few short years. MAY/JUNE 1986 3. Specific methodological improvements. In addition to changing incentives and norms, specific improvements are needed in research methods used by public administration scholars.
More extensive use of meta-analysis. One of the most important of these changes involves steps to make research more cumulative. Kronenberg's2I earlier call for a public administration proposition inventory was one means for dealing with this problem, but it has not been implemented in the 15 years since it was suggested, probably because it was dependent on a large-scale, collaborative effort. An alternative means for making research cumulative is wider use of meta-analysis. Metaanalysis refers to the set of methods used for cumulating knowledge across studies. These methods include literature reviews, counting statistically significant findings, and averaging results across studies.22 Also, the empirical analysis indicated that literature reviews were reported relatively infrequently in PAR, and few studies advanced to mature stages of social science research. Both of these findings suggest the need for more attention to meta-analysis. An ancillary benefit of greater use of meta-analysis is that it can also be valuable for integrating results across different academic fields, which is a particularly important objective for an interdisciplinary enterprise such as public administration.
Case study methodology. Case studies have been stereotyped as a "method of last resort," "exploratory," and an "attractive nuisance."23 Given these critical views about case study methodology, it might be appropriate to call for a significant reduction in the use of case studies in public administration research. Instead, we believe case studies will continue to be a popular method given the subject matter of the field and, furthermore, that a call for abandonment of case studies fails to consider a revisionist view that has developed in the past decade.24 Although major improvements still need to be made in case study research, recent refinements in the conduct of case studies have increased their validity.25 Considering their widespread use in the field, public administration scholars might undertake further refinements in the case study methods as a means for enhancing public administration research and generally contributing to development of -social science methods.
Qualitative methodologies. Another specific area for improvement is the use of qualitative methodologies within public administration. A grasp of qualitative methodologies is becoming increasingly important as "interpretation" and "rhetoric" regain prominence and respectability in the social sciences. 26 The empirical analysis confirmed public administration's already strong preference for qualitative research, albeit a diminishing one, but it also questioned the adequacy of the researcher's grasp of the tools and craft associated with qualitative methodology. There has been a small explosion of materials in recent years about qualitative research,27 and public administration scholars need to become both more proficient practitioners of this craft and contributors to the advancement of these methods.
Advanced quantitative methodologies. The call for better qualitative methodology is not a slap at its opposite number-more appropriately its complement -quantitative methodology. Although the empirical analysis indicated a significant increase in the amount of quantitative research in public administration, the techniques used were primarily confined to simple correlation and linear regression analysis. This represents an advance in the field's application of quantitative techniques, but public administration still lags behind other social sciences in the application of advanced statistical techniques. Thus, public administration scholars need to make more substantial use of causal analysis, structural equation models, and longitudinal statistical methods and to develop working competence with new statistical methodologies sooner after they become available to social scientists than they do currently.
Two examples of advanced statistical techniques that would significantly enhance opportunities to investigate research questions characteristic of the field are Box-Jenkins time series models and covariance structural modeling. Box-Jenkins is a technique for modeling changes in a time series of data to test the effects of specified interventions. Although it has not yet been widely applied, it has already been used to study such issues as the policy implications of economic change and the effects of the CSRA merit pay intervention on organizational performance. 28 Covariance structural modeling, commonly known as LISREL,29 is a causal data analysis technique that is much more powerful than path analysis, which became popular in the 1960s. LISREL permits simultaneous estimation of the relationship between observed measures of latent independent and dependent variables. LISREL has begun to appear with increasing frequency in sociology and management journals for research problems involving social and individual behavior.
Some attention needs to be given not only to specific techniques which might be integrated into the field, but also to how those techniques are acquired by students and scholars. An earlier study of empirical research in public administration and political science found a high degree of methodological stability over time. Scholars kept using familiar, traditional approaches rather than learning new methods.30 Thus, the field needs to develop support systems, for example, research workshops and doctoral consortia at professional conferences, to facilitate learning. Such support systems are equally applicable and necessary for public administration scholars interested in positive, interpretive, or critical research modes to develop and stay abreast of appropriate research methodologies. Quite clearly such steps are only a partial answer. However, they are not only necessary for moving the field to the forefront, but for improving average research craft in public administration.

Conclusion
Even if all the suggestions outlined above could be implemented instantly, it would take several years before their consequences would be noticeable. However, while some of our suggestions require collective or institutional action, many can be implemented by individual scholars in the routine practice of their craft. For example, individual scholars can stay with research issues over the long term, improve the methodologies associated with case studies, and increase the application of more advanced statistical methodologies appropriate to the problems of public administration. The acceptance of these suggestions by the public administration community could serve to advance both the science and the art of public administration.