Musings on a Model: CalSim II in California’s Water Community

Computer model results are becoming more prominent in water policy deliberations in California. CalSim II is the most prominent water management model in California, and has become central to a variety of water management and policy issues and controversies. This paper reports on the results of an extensive set of loosely-structured interviews with members of California’s technical and policy-oriented water management community regarding the use and development of CalSim II in California. The interviewers reflect on the thoughts of interviewees and how such interview activities can further policy-effective modeling and technical activities for water management. CalSim II is a complex model of a complex part of California’s changing multi-purpose water system. As such, analytical controversies and misunderstandings are inevitable. Ideally, a model and its associated data would perform an additional service as a forum to resolve technical controversies and continually improve quantitative understanding of the system. While CalSim II is generally seen as a significant improvement over previous models, a wide variety of ideas are suggested for improvements.


INTRODUCTION
Computer models have become increasingly important in the management and planning of California's water resources. The California Department of Water Resources (DWR) and U.S. Bureau of Reclamation (USBR) jointly developed CalSim II to model the State Water Project and Central Valley Project (SWP and CVP, respectively), which form much of the state's surface water storage and inter-regional water delivery infrastructure (DWR 2004).
CalSim II is a simulation model of the CVP and SWP storage and distribution systems that utilizes a linear programming solver in each time-step to route water through a network given user-defined constraints and priority weights. Developers of CalSim (the generalized water resources management model software underlying CalSim II) also developed the Water Resources Simulation Language (WRESL), which acts as an interface between the user and the solver, time-series database, and relational database. CalSim II simulation of the operations of the CVP and SWP systems includes physical, institutional, and regulatory constraints and an objective function composed of priority-weighted operational penalties. California's current regulatory environment is very complex; and that complexity is represented in the model by four regulatory layers: State Water Resources Control Board's Decision (SWRCB) 1485 (D-1485) and SWRCB Decision 1641 (D-1641); Central Valley Improvement Act (CVPIA), Section 3406 (b)(2); and the California Bay-Delta Authority's Environmental Water Account (EWA). While (b)(2) requires that the conditions under D-1485 be known, EWA requires that condition under D-1485, D-1641, and (b)(2) be known. Because the regulatory environments are interdependent, CalSim II simulates each regulatory condition sequentially for one entire year, before moving on to the following year. This sequential simulation of environmental conditions is commonly known as regulatory layers of CalSim II.
While USBR and DWR developed CalSim II for project-related purposes, CalSim II's actual uses have been wide-ranging. As the single official model for California's two largest water projects, CalSim II and its results affect statewide and Central Valley water operations and planning, and are often at the center of technical and policy controversies. Resolution of controversies often requires an initial airing of concerns from all parties, as it is usually difficult to address informally stated technical problems. This is especially true when many parties are involved, representing a wide range of interests and expressing a variety of concerns. As computer model results have played increasingly important roles in policy and planning decisions, technical concerns (and their policy manifestations) have impeded the development and use of serious modeling tools for water management in California. The original and central purpose of this research was to gather the uses, thoughts, and concerns of a broad cross-section of the California water community regarding CalSim II with the intent to facilitate discussion and assessment, and perhaps address these issues more productively. This project also provided background information for a peer review panel convened in November 2003. http://science.calwater.ca.gov/pdf/CALSIM_Review.pdf (Loucks et al. 2003).
Feedback was collected during interviews of 89 individuals who are involved in the management, planning, decision-making, analysis, and/or modeling of California water resources. Information gathered during the interview process includes existing and potential uses of and questions for CalSim II, reasons for selecting this model, views on its strengths and weaknesses, views on alternatives to CalSim II, and features that people would like to see in alternative operations and planning models or in an improved CalSim model. The report to the CALFED Science Program and its peer review panel presents the detailed methodology and summary of these interviews (Ferreira et al. 2004). The collected uses, thoughts, and insights regarding CalSim II should be useful for: • Purposes of external review, • Identification and prioritization of further model development activities, • Education and outreach activities that would make the model (and modeling in general) better understood and more useful, and • Better practical understanding (and perhaps ultimately better scientific understanding) of modeling and its complex role in water management in California.
Technical discussions usually benefit from open airing of technical concerns. We hope this paper and its underlying report provide such benefits.
For the California water community, this paper provides a concise overview and insights regarding the roles, problems, and concerns for water operations planning modeling. For researchers and technical managers, this study presents and illustrates a qualitative field method for gaining a better understanding of complex and controversial technical topics and highlight some challenges for the development and use of CalSim II and other models of the California water system.

METHODOLOGY
Surveys and interviews are commonly undertaken for research purposes. However, unlike most interview research, the intent of this particular project was not quantitative or scientific hypothesis-testing, but qualitative and applied, to extract from a broad selection of California's water management community their impressions, concerns, and uses for the CalSim II model. As such, a loosely structured interview process with emphasis on clear and verified interviewee statements of their ideas was developed (Bailey 1978). No attempt was made to assess the frequency of ideas contributed by interviewees. The frequency of responses or thoughts was not relevant for most intended purposes of this research. These interviewee-validated statements of uses, impressions, and concerns are available in the appendix of a report (Ferreira et al., 2004), with a systematic consolidation of these comments being presented in the report itself for easier reading and use by model developers, reviewers, users, and policy-makers.
Since most members of the interview team are also active in computer modeling of California water management (albeit not using CalSim), special potential problems and opportunities arose. As such, the process had some elements of a classical participant-observer study (Gans 1967;Whyte 1955), especially as reported in the latter part of this paper. The potential problems of having interviewers interpret interviewee responses based on the interviewers' experiences were mitigated by having multiple interviewers (for all but one interview), with all interviewers reviewing draft summaries of each interview for verisimilitude with interviewee statements. These draft summaries of each interview were then returned to interviewees with ample opportunity for correction or expansion. The interviewers' prior acquaintance with modeling water problems in California provided advantages for understanding many of the points made by interviewees, facilitating consolidation of comments in the report, and hopefully communicating these thoughts for later use.

Interview procedure
The research team developed and followed standardized procedures to arrange and conduct the interviews. Ninety-five individuals from California's water community, including staff from both DWR and USBR (the agencies that created, own, and manage CalSim II) and individuals affiliated with consulting firms, water districts, environmental groups, and universities, were contacted, of whom 89 agreed to participate in the interviews. Potential interviewees were selected from the members of the California statewide long-term planning (California Bulletin 160-03) advisory committee, from discussions with individuals known to be active in development and use of CalSim II, and from suggestions provided during the course of the interviews. A summary of interviewee affiliations is presented in Table 1. After being contacted, interviewees received a document describing the purpose of the CalSim II interviews, the questions they would be asked during the interview, procedures for interview write-up and review, and policies for attribution. The research team conducted interviews either individually or in groups, with group sizes ranging from two to five. In total the team conducted 65 interviews (16 groups and 49 individual) between April 30 and August 28, 2003. Each interview followed a common questionnaire (available in Appendix A of Ferreira et al. 2004), focusing on how the individual and his/her organization currently use CalSim II, would like to use the model, or plan to use the model in the future, as well as more open-ended questions to solicit the individual's full range of thoughts and suggestions regarding CalSim II. The responses were extensive, varied, and at times contradictory. Typical interviews lasted one hour, but some lasted as little as half an hour or as long as two and a half hours.
In all but one case at least two interviewers were present for each interview. Each member of the interview team took hand-written notes during the interviews, none of which were tape-recorded. After the interview was completed, the team wrote a summary and sent it to the interviewee or "lead interviewee" for group interviews. The interviewee then had two weeks to revise and/or extend the summary of his or her interview. Each interviewee had the option of having some, all, or none of the interview summary included in the remarks "Not For Attribution." All of the summaries of the interviews with DWR and USBR personnel were designated "Not For Attribution." Interviewees also had the option of submitting separate written statements, documents, or materials for inclusion or citation in an appendix of the report.
After finalizing the summaries, the research team aggregated the comments into a single database, combining both comments "For Attribution" and "Not For Attribution," and then categorized the comments by topic and content. The Current and Prospective Uses of CalSim II section presents the range of analysis to which users apply CalSim II. The section entitled Interviewee Thoughts and Suggestions describes remarks regarding the perceived strengths and weaknesses of CalSim II. The results presented below are based on what was heard during the interviews and, to the extent possible, do not contain the opinions of the research team. In the subsequent section entitled Discussion of Results the interview team presents its own thoughts on CalSim II and its future management and development.

CURRENT AND PROSPECTIVE USES OF CALSIM II
Current uses of CalSim II include policy planning studies, system operations, facility planning, regulatory compliance, model development, water management, impact estimation, and policy evaluation. Interviewees often use CalSim II with other models, as its output serves as input to numerous economic, hydrodynamic, water quality, operations, and other water planning models at both state and local levels. Table 2 presents a sample of interviewees' current and prospective uses of CalSim II; a full listing and discussion appears in Ferreira et al. (2004).

Evaluation of Management Options
Water Transfers x x Conjunctive Use x x Groundwater Banking x x

Other
Gaming Exercises x x Hydropower Generation x x

INTERVIEWEE THOUGHTS AND SUGGESTIONS
Most interviewee comments relate to CalSim II's strengths and weaknesses, suggestions regarding CalSim II technical support and development, and broad conclusions about the model's effectiveness in meeting the diverse goals of the many users of CalSim II results. Interviewee thoughts and suggestions were classified according to five major categories and 36 subcategories ( Table 3). Some of the most prominent themes that emerged from the hundreds of individual comments are summarized below according to five major categories presented in Table 3. The comments in the Mission section highlight the purposes and uses of the model. Administration refers to how DWR and USBR manage, direct, and supervise CalSim II and related activities. Implementation refers to how the CalSim software is applied to the SWP/CVP system. Inputs refer to the data required by CalSim II for each model run. Finally, Software refers to the general water resources simulation software package, CalSim, and is not specific to its application to the SWP/CVP system. Many other comments, not mentioned below due to limitations on space, appear in the larger report (Ferreira et al. 2004).

Mission
Prior to CalSim II, DWR and USBR had independent models of the Central Valley projects (DWRSIM and PROSIM, respectively). The two models had different sets of hydrologic data and treated project operations differently. There is wide agreement that cooperation between DWR and USBR has improved greatly as a result of their joint modeling effort to develop and maintain CalSim II. In addition, the use of a single, standard modeling tool and data set has greatly improved the general modeling environment in the California water community. Work now focuses more on substantive issues, rather than on differences between competing models.
While there is consensus that CalSim II represents a step forward, there is also consensus that it needs further improvement in a variety of areas. Many interviewees assert that CalSim II developers did not think through the questions that CalSim II would be asked prior to building the model, and so it is poorly suited to address many of the questions for which interviewees need answers. However, many see CalSim II as the only tool available for such questions, especially for modeling the CVP and SWP systems. The limited (or seemingly limited) modeling options for California water mangers leads to the perception that CalSim II is often misused, misapplied, or over-stretched. Some feel A primary area of concern among interviewees is CalSim II's ability to perform comparative and/or absolute analyses. Comparative modeling examines differences between multiple model runs to evaluate the effects that varying a condition, facility, or operating policy will have on the system, while absolute (or predictive) modeling directly estimates what is likely to happen to the system given a single set of inputs. There is general agreement among interviewees that CalSim II is an appropriate tool for comparative studies, but there is no such consensus regarding absolute studies. Many interviewees feel that using CalSim II in absolute mode is risky and/or inappropriate, but they have no other option because there are no other agency-supported alternatives. To that end, many interviewees want DWR and USBR to either improve CalSim II's predictive capabilities or create a predictive companion model. If users are to apply CalSim II in an absolute mode, many believe that detailed documentation of known limitations and weaknesses, a better understanding of the uncertainty associated with results, and additional effort towards the calibration and testing of the model are imperative.
The huge range of expectations that the California water community has for CalSim II exacerbates this problem of perception. Model developers promised that CalSim II would be easy to use and accessible; in reality, it is a complex model of a complex system that requires significant expertise to run and understand. As a result, only a few individuals concentrated in DWR, USBR, and several consulting firms understand the details and capabilities of CalSim II. Thus, much of the rest of the water community feels left in the dark regarding what CalSim II can do, how to use it, and where to find further guidance. This widespread confusion and uncertainty has eroded CalSim II's credibility outside the small circle of knowledgeable users, as it is difficult to trust a tool that one is unfamiliar with and does not understand. Many indicate that reducing these uncertainties would improve the model's credibility.

Administration
Interviewees commonly mentioned a need for more people who can run CalSim II. The current need for model runs outstrips the number of people who can produce them. This situation is likely to worsen as the demand for CalSim II runs continues to grow. CalSim II's complexity is daunting to new and potential users, and so very few individuals can conduct an entire model study and produce good quality CalSim II runs. This shortage of expertise means that DWR and USBR may be unable to produce CalSim II runs quickly, reducing the usefulness of the model, as it is effectively inaccessible due to the lack of qualified modelers. In addition, the narrow circle of knowledgeable CalSim II users contributes to the perception that CalSim II is a "closed shop" available only to a few insiders. Finally, a small group of users limits the power of CalSim II as an analytical tool, as some see CalSim II's potential power and utility expanded by having a broad spectrum of groups representing different perspectives on water management debates. There is also concern that CalSim II analyses are considered "good" or "acceptable" only with the approval of a select group of individuals who are very familiar with California's water system. A larger pool of users is likely to broaden this circle and dilute the influence of individuals. In the absence of expanding this group, or until the number of experienced users has increased sufficiently, there may be value in creating a standing review group or some other method to certify studies. In general, interviewees agree that DWR and USBR should actively seek to expand the group of expert users, especially to include non-agency and non-consulting users.
To further this goal, many interviewees recommend that DWR and USBR create a centralized source of support for CalSim II users. They would like a help desk or website to provide information on assumptions made in the model and guidance regarding model code, logic, and structure. Tutorials for running CalSim II and interpreting its results, software utilities with which to download data and perform statistical analyses of results, answers to common questions, and results from a sample CalSim II run would also further this cause. In addition, the agencies should expand the existing CalSim II training course to address both the logistics of running CalSim II and the subtleties required to understand the meaning of its output and its appropriate application. A well-publicized user group also could provide many of these services, distributing information to model users from many interest groups efficiently, and thereby expanding the skill base and reducing the perception of CalSim II as a "closed shop." Similarly, many interviewees feel that some CalSim II managers are defensive in the face of criticism and that including more stakeholders in the development process or providing a forum for input from model users will enhance CalSim II's acceptance and credibility.
All users agree that CalSim II needs better documentation of the model, data, inputs, and results. CalSim II is data-driven, and so it requires numerous input files, many of which lack documentation. Documentation of assumptions is spotty and very technical when it exists, making it difficult for anyone other than model developers to understand how CalSim II arrives at its results. Poor documentation of the conceptual model means that it takes a long time for users to answer seemingly trivial questions and it is difficult for new users to learn how to use the model at all. Overall, the lack of clear and comprehensible documentation increases the likelihood of misunderstandings regarding how the model functions and it contributes to the common impression of CalSim II as a "black box" whose inner workings are beyond the comprehension of most users. This also makes CalSim II runs difficult to duplicate, eroding the model's credibility.
CalSim II is still relatively new and so many users are unsure of and thus uncomfortable with its limitations. They want more information on the model's limitations, including a clear description of what the model does and does not do well. In addition, information on the uncertainty associated with CalSim II results in the form of error bounds, ranges for individual values, or statistical parameters (e.g., mean, variance, etc.) would inform users about the limitations of specific outputs, which is particularly important when users run CalSim II in absolute mode.
There is considerable debate about the current and desirable state of CalSim II's calibration and verification. Some efforts have been made to calibrate the model, but many interviewees express concern that this effort is insufficient if the model is to be run in absolute mode. In addition, DWR and USBR have released a benchmark study to provide a baseline case from which users create alternative scenarios and to which they compare results of alternative runs. This benchmark study has changed with ongoing modifications to CalSim II. Many model users and potential users look forward to a complete, unchanging benchmark study to provide a stable point of reference for other analyses.

Implementation
CalSim II is at once both too simple and too complex. Its representation of the SWP and CVP includes many simplifications that raise concerns regarding the accuracy of results. At the same time, CalSim II is so complex that it is difficult to understand and requires several hours to run. Interviewees express numerous concerns about specific details of CalSim II's implementation, only a few concerns which seemed more prominent and informative are addressed here.
CalSim II's complexity reflects the complexity of the California water system. However, this makes the model cumbersome and difficult to learn. The difficulty in learning and running the model has been a source of frustration to many users and potential users, and there is a common consensus among respondents that CalSim II should be more user-friendly so that stakeholders can run the model without hiring consultants. Just as CalSim II is complex and difficult to understand, so are its results. Many interviewees indicate that interpreting CalSim II results requires not only experience with the model, but also knowledge of the CVP/SWP system and of linear programming. Model users require significant time to determine if results are reasonable and very little guidance on this topic is available from model developers. Also, some claim that there are no specific criteria to define a "good" model run or post-processing tools to help visualize, interpret, correct errors, and obtain answers to common questions. There is additional concern that CalSim II's formulation should be more robust so that runs are not user-dependent. Starting from the same point, different model users likely will produce different CalSim II outputs because during a CalSim II run, the model user generally views intermediate results and adjusts model parameters until he/she reaches an acceptable result. This adds to inconsistencies across CalSim II runs, making results more difficult to interpret. Finally, CalSim II's complexity and its many layers have resulted in a model that requires several hours to run, frustrating planners who need to explore many refinements to alternatives.
Many interviewees are concerned that CalSim II's monthly time step cannot capture hydrologic variability adequately and thus does not compute water exports and export capacity accurately, both of which are significant factors in system operations. The model's inability to capture within-month variations sometimes results in overestimates of the volume of water the projects can export from the Sacramento-San Joaquin Bay-Delta and makes it seem easier to meet environmental standards than it is in real operations. Many of the system's operations function on a shorter time scale and so CalSim II cannot represent them well given its current formulation. On the other hand, it is unclear if reducing the time step would be either more accurate or more useful, given the additional data and assumptions that would be needed to characterize the system. Some fear that moving to a daily time step might worsen some problems due to questions regarding the precise timing of short events.
Interviewees cannot always determine the parameters to which CalSim II is highly sensitive or its overall stability and sensitivity. They feel that the linear programming formulation allows multiple solutions, which can differ considerably. Small changes in CalSim II input can result in large changes in model results, causing difficulties in impact analyses and the defensibility of model results. In addition, some users note that the multiple layers of regulations and operational agreements included in CalSim II may obscure the effects of the change to the system being modeled.

Inputs
Many interviewees indicate that CalSim II represents demands simplistically using out-of-date values and calculations. Specifically, they believe that demands should be based on land use and should be sensitive to economic factors such as the unit price of water. Without a better basis for the demands in CalSim II, many question the model's validity and capabilities. Some interviewees also want to see further improvement in CalSim II's representation of hydrologic processes. They feel that it is weak enough to undermine the entire model, as errors in this input propagate through each layer of the model. Many claim that CalSim II's hydrology uses data and methods that are decades out of date and rely on too coarse a geographic scale. In addition, some feel that development of a hydrology should be based on land use patterns and include thorough documentation. Despite these significant concerns, interviewees agreed that CalSim II's joint hydrology (agreed upon by both DWR and USBR) is an improvement over those used by each agency for its previous model.

Software
Model users express general frustration with CalSim II's commercial linear programming (LP) solver. They contend that it provides little information on the location of infeasibilities, so that even a knowledgeable individual may need many days to debug a run. In addition, the solver sometimes produces non-unique solutions and running identical scenarios on different computers seems to generate different results. Several model users state that the solver does not provide any of the sensitivity analysis that LP solutions usually offer and gives no indication of which parameters are constrained, so that users have to search for this information on their own. However, many interviewees feel that the use of an optimization engine for CalSim II is a step forward from previous models and that it is appropriate given the regulatory structures that it tries to model. Others are unsure of how the optimization engine works within CalSim II, while some feel that an optimization approach does not make sense given the many constraints of the SWP and CVP systems.
Individual users are developing their own post-processing techniques, creating the potential for inconsistencies between analyses. They would like visual tools with which to present and compare multiple CalSim II runs. Interviewees expressed interest in tools, both computational and especially visual, that would make it easier to compare results across runs.
Users would like a more intuitive, geographically referenced interface to facilitate the understanding of both inputs and outputs. It would be helpful if the interface could show the current CalSim II schematic, allowing a user to click on a node and see relevant information including input data, metadata, water balances, and information on the location of relevant equations.
Interviewees applaud CalSim II's inherent transparency as a data-driven model. However, some find the vast number of input files required by CalSim II daunting, thus reducing the effective transparency. In addition, CalSim II includes no automated quality control mechanisms for its many input files, resulting in a time-consuming, generally manual process for setting up a CalSim II run that leaves substantial room for error. Interviewees largely agree that CalSim II would be easier to use if it had a simpler and more coherent data management system. Particularly desirable functions include the archiving of calculation files and the ability to conduct multiple traces of dependencies.

DISCUSSION OF RESULTS
The section above is a brief distillation of the remarks heard during the interview process, absent, to the extent possible, of the research team's opinions. They include many of the most common and interesting points raised throughout the interviews. During the analysis of the many interviews, the research team developed its own thoughts regarding CalSim II and its future development and management. While these clearly are informed by the interviews, this section represents the opinions of the research team. Some of these conclusions are similar to those in the report of a subsequent external peer review panel (Loucks et al. 2003). Our conclusions below are organized around three areas.

Broader involvement in development and use
CalSim II is a significant improvement over previous models of the CVP and SWP systems. A publicly available model, CalSim II uses a modeling approach that affords a flexible, expandable, data-driven, and more transparent modeling framework than alternative and previous models. These modeling features are a remarkable achievement for the developers of CalSim II. DWR's and USBR's agreement to use a single model and underlying data sets has facilitated this significant accomplishment. This consensus has allowed the two agencies to devote resources to develop a single tool rather than critiquing each other's model, as seemed common in the days of PROSIM and DWRSIM.
Despite the advancements of the CalSim II modeling efforts, the consensus between DWR and USBR needs to extend beyond the two agencies. Although both agencies have made attempts to include outsiders in the model development process, CalSim II might have been a much better model had a broader range of stakeholders been more integrally involved during its development. The knowledge and expertise that stakeholders could have brought to the fore would have allowed model developers to better represent the operations and water demands of many local water agencies, particularly how local operations interact with and affect the CVP and SWP systems. This would allow model developers to implement more realistic water demands, in terms of both land use and alternative management options available at the local and regional levels. More importantly, the inclusion of stakeholders in the early stages of model development would have provided developers with crucial insight regarding current and prospective modeling needs of the water community and helped broaden the model user community.
The disconnect between CalSim II developers and the broader California water community is one of the greatest obstacles to CalSim II's acceptance. In part, this obstacle stems from the limited institutional charge of model developers to model only the CVP and SWP systems, and not California water management more generally. Broader and more serious effort on the part of CalSim II developers is needed to raise CalSim II's credibility among stakeholders. To gain credibility among stakeholders it is imperative that model developers see CalSim II as "outsiders" see it.
That is, DWR and USBR should more fully commit to communication with the wider California water community through a well publicized, open, and available channel. This communication channel should be a way for outsiders to provide feedback to DWR and USBR regarding their modeling needs and for model developers to provide information and assistance to model users. For CalSim II to be effective for policy and planning purposes it must be seen as more widely accepted by stakeholders, something that seems unlikely to happen unless (i) stakeholders are involved in ongoing model development, (ii) CalSim II is seen as a useful model to stakeholders, and (iii) more people around the state are comfortable using the model and interpreting its results.

Continuous improvement for contemporary problems
A significant consequence of not including more stakeholders in the early phases of CalSim II development is that the current versions of CalSim II are ill-suited for many of the current analysis needs of the broader water community, particularly absolute applications. Current California water management differs considerably from a decade or two ago, when models that were designed for comparative applications were adequate for most analysis needs. Compliance with legislative provisions regarding both the environment and water availability for new land development requires much more accurate and non-comparative quantitative results than before. While CalSim II developers like to promote it as a model best used for comparative analysis, there is a clear need for a model for absolute purposes. Many local agencies and other stakeholders (including DWR) are employing CalSim II to develop non-comparative, absolute estimates of short-and long-term water deliveries. Therefore, given the issues facing California's water resources and the analysis needs of the water community, it seems unrealistic to expect that modelers only will use CalSim II for comparative analysis. Under the mantra of "the best available tool" (in this case the only available tool), CalSim II is being used, and will continue to be used, for many other types of analyses for which it may be ill-suited, including in absolute mode. Considering the setting in which CalSim II exists, its developers should work toward a model appropriate for absolute applications.
For use in absolute applications, CalSim II needs adequate calibration and testing against recent historical data. (Some very preliminary exploratory work has begun in this direction (DWR 2003).) As with any other type of study, calibration and testing results must be accompanied by a self-critical analysis indicating where the model performs well and where it does not, as well as how this will affect studies that use CalSim II results, both in comparative and in absolute mode, and why. This type of self-critical information is essential. Only when model limitations are understood better will CalSim II gain broader and deeper credibility and acceptance within the water community, and needed improvements can be better pursued.
In addition to the need for a model that can be used in absolute mode, there is also widespread demand for a model that encompasses more than just the SWP and the CVP elements of California's inter-tied water system. The desired model or modeling framework would include not only more of California geographically, but also represent a wider range of water management opportunities and options. For CalSim II to be a truly statewide model it needs to cover the Bay Area, Tulare Basin (including the Friant-Kern and Madera canals, eastside San Joaquin reservoirs, and Millerton), Yuba River Basin (for potential water transfer opportunities), Colorado River, Colorado River and Los Angeles aqueducts, and local Southern California projects.
Coupled with a need for greater geographic coverage, CalSim II should include management options available in California at the regional and local levels. Inter-and intra-agency water transfers are now commonplace, as are other management options such as groundwater banking, conjunctive use, desalination, and water conservation. Consequently, to effectively simulate the array of water operations available within the State, CalSim II needs to include a wider range of management options, facilities, and regions. It is vital that those involved in the management of California's water be able to analyze how local, regional, and state facilities and options best go together. California does not currently have a model or modeling framework capable of such integrated analysis, to parallel the kinds of integrated management thinking being pursued at local, regional, and statewide levels.

Accessibility
Along with serving the needs of the water community, acceptance of CalSim II will increase if more people are able to use and understand the model. Consequently, DWR and USBR need to widen the pool of model users (a problem DWR and USBR managers often note). Unfortunately, the current narrow circle of knowledgeable CalSim II users contributes to the perception that CalSim II is a "closed shop," available only to a few insiders. This perception also raises concerns about conflicts of interest, as skills on which many diverse stakeholders rely are concentrated in the hands of a few consulting firms, DWR, and USBR.
To widen the group of CalSim II users, DWR and USBR should provide a much greater level of user support than presently available. More frequent training workshops, a comprehensive manual/user's guide, a help desk, and online tutorials could supply much needed assistance to current and prospective CalSim II users. Better pre-and post-processing tools should be available to make the input process more automated and less prone to user error and to facilitate comparison, retrieval, viewing, and result interpretation.
Such attempts, however, should not replace thorough model and data documentation and version control, something that CalSim II currently lacks. CalSim II is unavoidably a very complex model and thus requires comprehensive documentation to enable users to understand the model framework and, more importantly, the sources and methodology used in the derivation of input data, including their limitations. It is difficult to attain credibility when the model and data are seen as impenetrable, particularly when model results run counter to many people's understanding of the system. Counterintuitive results are not necessarily infrequent or bad outcomes for a complex model of a complex system serving diverse stakeholders.
Model documentation should include information regarding appropriate uses of the model, data and model limitations, and error bounds on output values specific to the various purposes for which the model can be used. An often-mentioned frustration of many would-be model users is the lack of guidance on (i) how to appropriately interpret model results for various applications and (ii) what constitutes an acceptable model run.
As with any model, it is much easier to perform a good run if the user thoroughly understands what is being modeled. Understanding California's water sys-tem allows the model user to determine whether or not a particular run is "good" and to interpret its results. However, this should not be an excuse not to provide guidance on determining what constitutes a good model run and what must be done to attain a good run. If there are people who can evaluate a set of model results and determine if they are appropriate (and if not, how to modify model inputs to achieve adequate results) then that knowledge should be made available in the public domain, perhaps in the form of a post-processor. Such a post-processor should include guidelines for the appropriate interpretations of model results for various types of applications, including the use of monthly model results to assess impacts that depend on operations and processes that occur at a sub-monthly time-scale. Such a post-processor might not be a final certification of a model run, but would provide an initial screening.
A major problem facing CalSim II developers is that the system they try to model is extremely complex, particularly in light of numerous environmental requirements that must be modeled sequentially. While some people criticize CalSim II for doing too much and therefore being too complex, others believe that CalSim II is not comprehensive enough. Consequently, CalSim II is simultaneously seen as both too complex and too simple. This apparent dichotomy can only be resolved if CalSim II is made truly modular. Modularity would allow model users to turn features, regions, or layers of disaggregation on or off depending on their modeling needs. Modularity could also reduce model run time for many purposes and allow model users to apply CalSim II more efficiently in the early stages of screening alternatives.

THEORETICAL CONTEXT
The theme of opening CalSim II to a broader set of users is consistent with a growing body of literature on the value of combining policy and technical processes, rather than letting one lead the other (Sabatier 1999). While much of this literature focuses on how decision makers plan large infrastructure projects or manipulate natural resource systems, their lessons apply to the development and use of CalSim II, a complex model with significant policy implications.
The importance of including affected parties is now broadly accepted in project planning (WCD 2000; Delli Priscoli 2004). Beginning with research in ecosystem dynamics in the 1970s (Holling 1978), the concept of adaptive management and the integral use of computer models in environmental management and policymaking has grown to influence a variety of disciplines. Traditional methods of centralized planning for projects that affect a broad array of stakeholders based on technical expertise have largely given way to more holistic approaches that solicit input from a wider variety of sources and perspectives (NRC 2004). Within water resources management, this approach has been applied to projects as varied as restoration of the Everglades in Florida and the planning and construction of high dams in developing countries (NRC 2003;WCD 2000). The vast majority of the stakeholders interviewed about CalSim II expressed some interest in a more inclusive process for the ongoing development and improvement of the model. This broader literature supports many interviewees' assertion that CalSim II would benefit from their input, as the inclusion of multiple stakeholders in this process is likely to improve both the performance and acceptance of a complex project such as CalSim II (Lee 1993).

DISCONTENT AND MODEL DEVELOPMENT
Based on the concerns voiced by the interviewees during the interview process and on the impressions of the research team there exists significant concern regarding the abilities and applicability of CalSim II to California's large inter-tied water system. A simple reading of the summaries from the interviews could leave one with the impression that discontent with CalSim II in the water community is a serious impediment to the model's success. However, discontent with an analytical tool is not necessarily unhealthy or avoidable. Voiced concerns are a sign that the model is being used, produces useful insights, and encourages more systematic discussions of system details. Concerns also often provide a positive basis for model improvement.
Investments, both in terms of money and time, in analytical tools for decision-making usually arise from discontent with unaided decision-making. Technical development often follows six stages: 1. Informal statements of concerns; 2. Formal statement of concerns; 3. Assessment of concerns; 4. Plans to address concerns; 5. Actions to address concerns; 6. More (and hopefully different) concerns (repeat step 1).
The development process is by no means linear (BDMF, 2000). Stages can overlap and there is still the likelihood that new concerns will arise even after the effort is made to address the original concerns. Thus the development process is circular, reflecting evolution of the model, greater understanding of the system, and changing thoughts regarding management and policy problems.
If the technical end product was useless, its audience would ignore it entirely and the cycle would end without having successfully addressed the concerns. Many concerns will appear internally within the technical development team working on the products; such comments usually require less communication effort since they are already internal to the development team. New concerns also arise as the field problems to which the model is applied change. Such concerns are a sign of success, as the model is considered worthy of being stretched or adapted to address new problems. Finally, some concerns indicate great success from a model when they arise because model users demand further refinements as they ask more probing questions of the system.
In the case of CalSim II both the user community and the development team have raised concerns. Some of these concerns are quite positive in that the model is being asked to address new and expanded problems in more precise ways, going beyond the model's original narrow SWP/CVP scope. Ultimately, the broader the range of individuals involved with the on-going development of the model, the better the product likely will be. The fact that so many individuals have concerns regarding the uses and applicability of CalSim II is a sign that the model is being used and is worthy of ongoing applications, discussions, and development.