Implications of a regional resource adequacy program on utility integrated resource planning

Author(s): Carvallo Bodelon, Juan Pablo; Zhang, Nan; Leibowicz, Benjamin; Carr, Thomas; Galbraith, Maury; Larsen, Peter H | Abstract: LBNL collaborated with the Western Interstate Energy Board and the University of Texas, Austin to investigate the implications of a regional resource adequacy (RA) program on utility integrated resource planning. This report is focused on an active policy discussion to develop a novel voluntary program to share capacity resource and improve RA in the Western Interconnection. This report identifies two RA components of IRP that will be highly impacted by a regional RA program: resource capacity accreditation and RA targets. There are at least four resources that will require specific attention for their capacity credit calculation: (1) variable renewable resources, (2) demand-side resources, (3) hydropower, and (4) contracts. It will then be necessary to decide on a RA target reliability metric (e.g., a planning reserve margin) that is at least the minimum requirement in IRPs to ensure consistency in RA requirement calculations. The report finds that load forecasting and transmission expansion analyses will be moderately impacted, but that most of IRP components will not be significantly impacted. The report includes (i) a review of traditional resource adequacy practices in IRP; (ii) a case study of an existing regional RA program that interacts with IRP (Southwest Power Pool) and (iii) an overview of the NWPP regional resource adequacy proposal that is the object of this work. This paper does not (1) advocate for or against a regional RA program for the NWPP, (2) make detailed design recommendations for this program, or (3) assess its benefits and costs. This paper addresses three research questions: • How would typical IRP processes change if an LSE joined a regional RA program? • With a new regional RA program, which RA elements would remain local (i.e. within IRP) and which would become regional (i.e. within the RA program)? • How much control would LSEs and states retain over their utility resource mixes considering the influence of a regional RA program? This paper is primarily written for state regulators, public utility commission staff, and resource planners from states in the NWPP footprint that are pondering how their IRP guidelines and regulations may need to adjust to operate jointly with a regional RA program. The content of this paper may also help the NWPP RA program developer as it interacts with potential member states and utilities to understand what aspects of energy policy may be influenced by the program under development. This work was funded by the Energy Resilience Division of the U.S. Department of Energy’s Office of Electricity under Lawrence Berkeley National Laboratory Contract No. DE-AC02-05CH11231.


List of
.1 IRP RA components, impact from a regional RA program on these components, and how control of these components is allocated .. Implications of a regional resource adequacy program on utility integrated resource planning │v

Executive Summary
Resource adequacy (RA) refers to the ability of an electric power system to meet demands for electricity using its supply-side and demand-side resources (NERC, 2011). Monitoring and maintaining RA is becoming increasingly complex and challenging due to plant retirements and higher penetration of variable renewable energy resources that translate to higher uncertainty with the amount of generation that will be available during periods of peak demand. This challenge is becoming particularly acute in the Western United States due to states' environmental policy objectives and evolving resource economics that are prompting impending retirement of coal plants (NWPCC, 2018). A recent study showed that the Pacific Northwest region (PNW) could present RA issues as early as 2020 and highlighted the need for substantial reform in resource adequacy practices to meet reliability standards in the next decade (E3, 2019).
As a response challenges in the PNW region, the Northwest Power Pool (NWPP) is developing a proposal for a voluntary regional RA program. This regional program may overlap with states' existing integrated resource planning (IRP) processes that also assess and address resource adequacy issues. The NWPP proposal acknowledges this potential overlap, focusing on their differences, and how they complement each other (NWPP, 2019). States exercise control over resource planning through IRP regulations. This paper examines the impact of a regional RA program on electric utility IRP. Additional key questions are how much control over resource adequacy participating utilities will have to relinquish and what the impacts are on other aspects of state energy policy. Accordingly, this paper focuses on how joining a regional RA program may impact electric utility IRP processes and highlights key resource planning components that may be affected.
This paper covers three topic areas in its analysis. First, it documents traditional resource adequacy practices in IRP by examining plans from 11 Western and Midwest U.S. load serving entities (LSEs). Second, it develops a case study of an existing regional RA program that interacts with IRP through report analysis and interviews conducted with Southwest Power Pool staff and state officials. Finally, it presents the NWPP regional resource adequacy proposal that is the object of this work. This paper does not (1) advocate for or against a regional RA program for the NWPP, (2) make detailed design recommendations for this program, or (3) assess its benefits and costs. This paper addresses three research questions:  How would typical IRP processes change if an LSE joined a regional RA program?
 With a new regional RA program, which RA elements would remain local (i.e. within IRP) and which would become regional (i.e. within the RA program)?
 How much control would LSEs and states retain over their utility resource mixes considering the influence of a regional RA program?
This paper is primarily written for state regulators, public utility commission staff, and resource planners from states in the NWPP footprint that are pondering how their IRP guidelines and regulations may need to adjust to operate jointly with a regional RA program. The content of this paper may also Implications of a regional resource adequacy program on utility integrated resource planning │vi help the NWPP RA program developer as it interacts with potential member states and utilities to understand what aspects of energy policy may be influenced by the program under development.

How would typical IRP processes change if an LSE joined a regional RA program?
IRP processes will not fundamentally change when an LSE joins a regional RA program. However, some key IRP assumptions or resource adequacy components will be impacted. This report identifies two resource adequacy components of IRP that will be highly impacted: (1) RA targets and (2) resource capacity accreditation. Resource capacity credit will require much more alignment between IRP and the NWPP RA program. If IRP and regional RA capacity accreditation for the same resource differ, there is a risk that an LSE would be adequate at the local-level, but not at the regional level. For this reason, the LSE would have to justify additional investment outside its IRP recommendations to comply with regional resource adequacy requirements. Furthermore, states have historically assigned different capacity credit factors for similar resources-especially for wind, solar, and demand response-which may create friction among members if some states recognize higher or lower capacity than others for similar resources. There are at least four resources that will require specific attention for their capacity credit calculation: (1) variable renewable resources, (2) demand-side resources, (3) hydropower, and (4) contracts. It will then be necessary to decide on a RA target reliability metric (e.g., a planning reserve margin) that is at least the minimum requirement in IRPs to ensure consistency in RA requirement calculations. Two IRP components will be moderately impacted by an LSE joining a regional RA program: (1) transmission expansion and (2) load forecasts. Transmission expansion studies typically focus on the LSE's local power system and not all IRPs include a regional analysis to gauge the deliverability of Implications of a regional resource adequacy program on utility integrated resource planning │vii resources outside of the LSE's service territory. These limitations of current IRP processes could hinder the pooling of resource adequacy resources across the NWPP footprint, which is one of the main sources of cost savings. From an IRP perspective, the main challenge will be how to assure that the transmission expansion assumptions built into each IRP are consistent with the assumptions made at the regional-level for RA calculations. Load forecast could be delegated to individual LSEs, but the regional RA program would need to standardize its statistical methods and potentially require additional information if regional coincident peak demand were used for RA requirement calculations.

Which RA elements would remain local (i.e. within IRP) and which would become regional (i.e. within a new RA program)?
This report finds that for an efficient and effective operation of a regional RA program, states in the footprint will need to defer to the program's definitions of resource adequacy targets (e.g. the PRM) and resource capacity accreditation. States would effectively surrender control over those two assumptions and let the regional program define them, incorporating them exogenously in their IRP processes. Stakeholder involvement processes will be critical to give states voice in these collaborative decision processes (see Section 7.3.3).
In addition, states will need to develop a shared agreement on the processes to produce load forecasts and to define transmission expansion. These elements could continue to be developed by the LSE under state IRP mandates, but coordination of input data, modeling assumptions, and outcomes will be needed with the regional RA program.

How much control would LSEs and states retain over their utility resource mixes considering the influence of a regional RA program?
In general, FERC guidelines for Regional Transmission Organizations (RTOs) and Independent System Operators (ISOs) strive to allow states the right to decide their resource mix. The regional RA program defines the capacity needs to ensure reliability, but does not select the resource employed to meet those needs.
However, an open question closely related to capacity credit determination is how surrendering the control over how much capacity to recognize for certain resources would affect the resource portfolio choices in IRP. As mentioned, states whose power systems are managed by an RTO retain the right to determine their resource mixes. However, the capacity contributions of resources do affect the leastcost calculation and can indirectly impact the resource selection. For example, if a resource's contribution to peak demand were adjusted from 50% to 25% it would require twice the level of investment on that resource to meet the same peak demand contribution. This adjustment would certainly affect the relative economic performance of this resource in a least-cost analysis and subsequently alter the portfolio outcomes.

Introduction
Resource adequacy (RA) refers to the ability of an electric power system to meet demands for electricity using its supply-side and demand-side resources (NERC, 2011). Monitoring and maintaining RA is becoming increasingly complex and challenging due to plant retirements, higher penetration of variable renewable energy resources, and COVID-related load fluctuations that translate to higher uncertainty on the amount of generation that will be available during periods of peak demand. This challenge is becoming particularly acute in the Pacific Northwest region (PNW) due to states' environmental policy objectives and evolving resource economics that are prompting impending retirement of coal plants (NWPCC, 2018). A recent study showed that the region could present RA issues as early as 2020 and highlighted the need for substantial reform in resource adequacy practices to meet reliability standards in the next decade (E3, 2019).
The task of assessing, monitoring, and planning for RA has typically been performed by balancing area authorities (BAAs). Traditionally, BAAs were electric utilities, most of which developed their RA assessments as part of their integrated resource planning (IRP) processes. Over time, Regional Transmission Organizations (RTOs) and Independent System Operators (ISOs) have adopted BAA roles for the load serving entities (LSEs) in the regions in which they operate. Compared to individual LSEs' RA assessments, regional RA programs can exploit resource, load, and transmission diversity given their expansive footprints and achieve cost savings by pooling capacity resources. While RTOs and ISOs determine RA targets and monitor member compliance to meet these targets, states retain authority over resource planning to meet those targets (i.e., the future resource mix) and how to allocate costs, among other choices.
As a response to challenges in the PNW region, the Northwest Power Pool (NWPP) is developing a proposal for a voluntary regional RA program. The NWPP proposal acknowledges the potential overlap with states' IRP processes, focusing on their differences and how they complement each other (NWPP, 2019). In contrast, this paper is focused on how a regional RA program and IRP processes overlap and highlights key resource planning components that may be impacted when a utility joins a regional RA program. For example, one potential technical conflict between IRP and a regional RA program is that both are focused on producing a capacity requirement, or RA target, for IRP-regulated entities. IRP produces an RA target by combining the LSE's peak demand and a designated planning reserve margin. A regional RA program also produces an RA target based on a different set of methods and assumptions. It follows that it would be problematic for an LSE to meet two different capacity requirements.
The current NWPP proposal upon which this paper is based is evolving and some of the issues found in this paper may be addressed as the program is refined. Regardless, one critical question for states in the NWPP footprint whose utilities may join a regional RA program is how much control over resource adequacy they will have to relinquish and what the impacts are on other aspects of state energy policy. States exercise control over resource planning through IRP regulations, and hence this paper is focused on the interactions between IRP and a regional RA program. This paper does not (1) advocate for or against a regional RA program for the NWPP, (2) make detailed design recommendations for this program, or (3) assess its benefits and costs. This paper addresses three research questions:  How would typical IRP processes change if an LSE joined a regional RA program?
 With a new regional RA program, which RA elements would remain local (i.e., within IRP) and which would become regional (i.e., within the RA program)?
 How much control would LSEs and states retain over their utility resource mixes considering the influence of a regional RA program?
This paper is primarily aimed at state regulators, public utility commission staff, and resource planners from states in the NWPP footprint that are pondering how their IRP guidelines and regulations may need to adjust to operate jointly with a regional RA program. LSEs that do not file IRP (e.g., Electric Service Suppliers) that will join the NWPP RA program would also benefit from learning about the resource adequacy planning components of the program. The content of this paper may also help the NWPP RA program developer as it interacts with potential member states and utilities to understand what aspects of energy policy may be influenced by the program under development. Finally, this paper should be useful to other RTOs/ISOs whose utilities are required to conduct and file IRP processes and improve the connection between these processes and regional RA assessments. This paper focuses on U.S. IRP examples, although some Canadian provinces also produce IRP and could benefit from these findings.
The remainder of this report is organized as follows. Section 2 summarizes fundamental RA principles and the standard elements of an RA assessment. In Section 3, we review a sample of 11 IRPs from the Western and Midwest United States and report the methods that utilities employ to assess and plan for RA. Section 4 provides a detailed description of the current design of a regional RA program proposed by the NWPP. In Section 5, we discuss the Southwest Power Pool's (SPP) experience to understand how its regional RA program interacts with the IRP processes of its member LSEs. In Section 6, we identify and discuss how four specific IRP components identified in Section 3 would be impacted by a regional RA program. Section 7 concludes the report by answering the three research questions, summarizing the most important findings, and suggesting follow-up research activities.

Resource adequacy principles and current practices
This section defines resource adequacy and explains its traditional components and calculation methods. The section concludes with a summary of resource adequacy assessments performed for the Western United States.
Resource adequacy is the ability of supply-side and demand-side resources to meet the aggregate electrical demand including losses (NERC, 2011). Resource adequacy assessments typically focus on having adequate generation. However, a power system must also have adequate transmission and distribution lines to deliver power to load points, and may also consider demand-side resources. Ensuring that their Resource adequacy is the ability of supply-side and demand-side resources to meet the aggregate electrical demand including losses (NERC, 2011).
power systems are resource adequate is a crucial objective for utilities, regional transmission organizations (RTOs), and regulators who aim to provide affordable electricity to customers while maintaining high reliability and avoiding costly outages. Resource adequacy has become a prominent topic in both academia and industry due to its importance and the challenge of achieving it in a rapidly evolving electric sector.

Supply and demand
Annual peak demand is the focus of resource adequacy because this is when the maximum capacity is required, and hence when the risk and impact of a shortage tend to be highest. Electricity system planners forecast the approximate annual peak demand during the planning horizon, and most utilities use the forecast to estimate their load obligation and determine how much capacity should be available in the system. The peak demand forecast is usually produced based on some combination of historical load data as well as geographic, economic, and forward looking climate change factors.
On the supply side, resource adequacy is provided by the facilities that generate electricity and the transmission and distribution network that delivers power to customers. Supply resources are typically categorized by generation technology or fuel (e.g., coal, natural gas, and wind), whether they are selfowned or contracted, or whether they operate as baseload, variable, or peaking generators. These categories correlate with resource availability and their capacity to meet peak demand.
Due to the uncertain availability of supply resources and variation in loads, electricity system operators maintain reserves to ensure that demand can be met even when load is higher than expected or resources experience unplanned disruptions. Reserves are typically categorized as planning reserves and operating reserves. The planning reserve refers to the additional capacity procured beyond the amount required to satisfy the expected peak demand (NWPP, 2019). In other words, the planning reserve constitutes a system design buffer that allows the system to cope with unexpected, adverse conditions on an annual or longer timescale. The operating reserve refers to the various ancillary services that operators procure during daily operations on timescales of minutes to hours. For example, these services can be called upon to respond to load variations (regulating reserve) or short-term, unforeseen events such as the forced outage of a generating unit (contingency reserve).

RA metrics
Utilities and regional regulatory bodies use a variety of metrics to determine the level of resource adequacy that is sufficient and to track the actual status of resource adequacy on a power system. RA metrics can be used as a resource adequacy target that an entity must reach, or to describe the status of a power system or outcome of the planning process. Assessing whether a system would actually achieve a desired reliability target is inherently a probabilistic problem, but resource adequacy targets are often determined based on deterministic metrics which are more easily interpreted by utilities and monitored by regulators.
The planning reserve margin (PRM) is the predominant deterministic metric used to ensure that sufficient resources are available to meet projected load obligations over the course a determined timeframe. Generally, the PRM measures the percentage by which generation capacity exceeds the forecasted peak demand. For example, a 15% PRM stipulates that the system available capacity at the time of peak demand should be 15% higher than the peak demand. The PRM is popular among planners because it is intuitive to interpret and easily incorporated into capacity planning models. However, unless it is derived from probabilistic models to be consistent with probabilistic metrics, the PRM has its limitations as a metric to measure the resource adequacy of a system since it is unconnected to the underlying system risks. This is particularly true for electricity systems with considerable uncertain generation due to substantial variable resources whose capacity factors during peak load times are stochastic.
Probabilistic models consider stochastic scenarios related to uncertainties such as loads, variable resource capacity factors, and unplanned outages to determine how reliable an electricity system is in terms of avoiding power disruptions, measured by probabilistic metrics. While these metrics correspond more directly to conceptions of a reliable electricity system, they are less intuitive to calculate and monitor, and require more sophisticated methods. To compute probabilistic metrics, planners need to develop stochastic models and quantify probability distributions for uncertain loads, variable generation profiles, and extreme events. Table 2.1 lists some common probabilistic metrics that are used by utilities and regulators (North American Electric Reliability Corporation, 2018). These metrics are variously based on the frequencies and magnitudes of events. The choice of probabilistic RA metric is consequential, as research shows that there is no simple correspondence where realizing a targeted value of one metric necessarily translates into given values of the other metrics (Fazio and Hua, 2019).

Enforcement mechanisms in regional RA programs
Traditionally, resource adequacy targets and assessments were established at the individual utility level. However, after electricity sector deregulation in many regions of the United States and the Stochastic methods explicitly incorporate uncertainty to produce outcomes and/or optimal decisions that reflect the randomness inherent in the real world. In a stochastic model, parameters such as wind power output and load are often treated as uncertainties that take on different values with associated probabilities. This is in contrast to a deterministic method, which assumes that all parameters take on singular values with certainty.
development of ISOs and RTOs, there are now various enforcement mechanisms and cooperation schemes to encourage or require adequate investments in supply resources in competitive electricity market areas.
Energy-only markets work by compensating power that is actually produced. In principle, energy-only markets should not require any regulatory mandates to ensure resource adequacy because investors in peaking power plants can recover their costs by selling power at very high prices in the few hours of the year when they are needed. Currently, ERCOT is the sole energy-only market in the United States, serving roughly 90% of electricity load in Texas. In the Western Interconnection, the Alberta Electricity System Operator (AESO) has been operating an energy-only market since 2000. AESO studied the implementation of a capacity market to substitute its energy-only market in 2016, but abandoned the idea. Energy-only markets do not rely on an obligatory resource adequacy target to ensure system reliability. By contrast, they rely solely on energy market revenues to induce sufficient investment in generation capacity. The reliability level is then an output of market choices rather than an input to mechanism design. In some competitive markets, such as ISO-NE, NY-ISO, and PJM, centralized capacity markets complement energy markets by directly procuring enough generation capacity to maintain desired PRMs several years into the future. The market procures capacity through an auction and compensates generators who bid successfully for the capacity that they install as a result.
Compared to centralized capacity markets, bilateral capacity markets enable more local control over resource planning. The MISO, CAISO, and SPP systems all offer bilateral capacity markets. Bilateral capacity markets do not directly procure capacity as in a centralized capacity market, but they allow utilities to negotiate bilateral contracts. This allows a utility to supplement its self-owned resources with contracted resources owned by another party, and provides a potential revenue stream for generation owners who sell their capacity to utilities. The prices of these contracts are mainly determined by the supply and demand for capacity in the market, but the market administrator may set a price cap on capacity procurement or levy a penalty for utilities that do not meet their allocated shares of system need.

Utility RA assessments
All LSEs need to conduct RA assessments. Depending on the jurisdiction, these assessments may or may not fall under the umbrella of a regional RA program that coordinates RA planning among multiple utilities on a broader spatial scale. For individual regulated utilities subject to IRP requirements, RA assessments are typically part of-often implicitly-their IRP process.
An IRP typically covers more topics than an RA assessment. Utilities also use the IRP process to develop plans to meet a range of other objectives such as least-cost generation expansion, equal treatment of supply and demand-side resources, complying with environmental regulations, and promoting stakeholder involvement. A thorough RA assessment is fundamental to IRP because it ensures that the resource portfolios considered by the utility for future investments satisfy the necessary reliability standards. The role of IRP as a resource adequacy assessment platform is examined in detail in Section 3.

Regional assessments
LSEs generally rely on owned resources or long-term firm capacity contracts to meet their capacity requirements (Carvallo et al., 2020). LSEs rely less on market transactions 1 , in part because they are rarely able to rigorously analyze future regional market conditions to determine whether sufficient capacity will be available to purchase. Beyond the potential regional generation capacity deficit, there could also be hidden problems due to constraints in the regional transmission network that prevent power produced by a utility with excess capacity from reaching a utility facing a shortage. As a result, relying on market transactions might seem like a reasonable strategy for a single utility's planning, but not when examining the region as a whole. Therefore, several regions of the U.S. electricity system conduct regional assessments to monitor the RA status across several balancing areas. Regional assessments with varying degrees of comprehensiveness and authority are described below.
The North American Electric Reliability Corporation (NERC) is a non-profit international regulatory authority whose mission is to assure the effective and efficient reduction of risks to the reliability and security of the grid. As the largest and most diverse of the NERC regional entities, the Western Electricity Coordinating Council (WECC) encompasses 329 members including 38 BAs and is responsible for coordinating and promoting bulk electric system reliability in the Western Interconnection. WECC publishes information on RA planning and investigates how to maintain future resource adequacy considering growing loads, increasing coal retirements prior to the end of their established depreciable lives, and stringent carbon abatement goals, but it does not impose mandatory resource adequacy requirements on LSEs. Ultimately, LSEs and their regulators are responsible for ensuring RA in their service territories in the Western Interconnection.
There are several regional RA programs that are currently operating in the Western United States, and other potential future programs have been proposed. CAISO is a non-profit Independent System Operator (ISO) that manages about 80% of all electricity consumed in California. Following the California Electricity Crisis of 2000-2001, CAISO and the California Public Utility Commission (CPUC) developed an RA program (hereinafter referred to as the California RA program) that coordinates bilateral trades and procurement of RA capacity replacement among Investor Owned Utilities (IOUs) (Chatterjee and Oren, 2007). CAISO has the authority to procure backstop capacity under California's RA program and allocate costs, and it operates procurement mechanisms for capacity deficits for reliability targets and transmission grid needs. As California continues adding variable renewable resources, and particularly given the August 2020 forced power outage events, the California RA program may play an increasingly important role to assist utilities to enjoy the economic benefits of resource diversity and capacity sharing while maintaining high reliability standards.
The Pacific Northwest has assessed resource adequacy since the late 1990s. The Northwest Power and Conservation Council (NWPCC), whose member states include Idaho, Montana, Oregon, and Washington, recently assessed the adequacy of the power supply for the 2024 operating year and used its GENESYS model to test whether the 5% LOLP reliability target is met (NWPCC, 2018). They only considered existing resources as well as planned resources that are sited and licensed, and found that the power supply in the Northwest will become inadequate beginning in 2021 with an estimated LOLP of 7.5%, due to considerable retirement of coal-fired generating capacity prior to the end of their established depreciable lives.
The Northwest Power Pool (NWPP) is currently developing a proposal for a regional RA program to address potential capacity deficits. In 2019, NWPP hired E3 to conduct and publish an RA assessment in the Pacific Northwest (E3, 2019). The report analyzed the challenges of ensuring RA in a decarbonized power system with high renewable penetration and investigated the RA needs under various scenarios in multiple years. The proposed NWPP regional RA program is presented in detail in Section 4.
Finally, the SPP operates a regional RA program for LSEs participating in its wholesale electricity market, which spans all or parts of 14 U.S. states. LSEs consist of a mix of publicly-owned and IOUs as well as independent power producers. The PRM is established at 12% for each LSE or a level close to 10% if the LSE has at least 75% hydro resources in its generation mix (SPP, 2019a). SPP estimates the RA requirement for each LSE and verifies its compliance for summer and winter seasons each year. SPP is an interesting case study for this paper, because many LSEs in its footprint are required to conduct IRP while also complying with SPP RA requirements. The experience of SPP and details of its RA program are reported in Section 5.

The role that IRP plays in resource adequacy
This section investigates how LSEs currently assess RA within the context of IRP. We conduct a detailed review of a sample of 11 IRPs from LSEs across the Western and Midwest United States, including LSEs who are and are not part of regional RA programs. Table 3.1 provides the names, IRP submission years, service territory boundaries, and numbers of customers for the 11 LSEs included in our sample. We selected these LSEs because they are geographically distributed across the Western United States, vary in size, and differ according to whether they fall within the jurisdiction of a regional RA program or not. It should be noted that the sample includes two LSE members of SPP-OG&E and KC-BPU-which file IRPs and are also part of a regional RA program. We acknowledge that our sample is a selective set of IRPs and that the selections partially reflect our own familiarity with and interest in these LSEs. Some IRPs separately and explicitly convey their RA assessments (in which case they might use alternate terminology, such as "reliability analysis"). In other IRPs, RA is certainly assessed, but it is incorporated into other parts of the IRP (such as the construction of candidate resource portfolios) in a manner that often obscures the RA assessment and makes it difficult to identify. Nevertheless, we were able to study and categorize the approaches and methods that the 11 utilities in our sample employ for RA assessment.

Resource adequacy modeling approaches in the sample of IRPs
In this subsection, we report on the approaches that LSEs use in their IRPs to model different elements of RA, including RA targets, load obligations, supply resource portfolios, transmission expansion, and some elements that are becoming increasingly important. Table 3.2 lists the RA targets that the 11 IRPs aim to achieve over their planning horizons. The RA targets are most often specified in terms of a PRM, likely due to the simplicity of interpreting, calculating, and monitoring this metric. However, in several instances, the PRM target is itself the outcome of a more sophisticated analysis carried out to determine the PRM needed to achieve a desired maximum LOLP of one day of lost load every 10 years. Interestingly, Avista implements this process in reverse by specifying an RA target as 5% LOLP and then identifies the portfolio(s) necessary to satisfy an 18% PRM. We also find that KC-BPU and OG&E have lower PRMs, which were assigned to them by the SPP regional RA program. In 2016, the SPP Board approved the reduction of PRM from 13.6% to 12% due to the increased load and resource diversity facilitated through regional transmission planning and market operation. As a result, the capacity requirement in SPP was lowered by about 900 MW (SPP, 2016a). The regional RA program helps these SPP-based LSEs achieve the same or better reliability with lower PRMs by leveraging the diversity of all regional loads and resources in the SPP area. Comparing the PRM targets of LSEs that are not part of the same regional RA program is difficult because the exact meaning of each PRM depends on each LSE's definitions and assumptions. These can differ considerably, and in fact, standardizing assumptions and practices is an important role of a regional program. For example, some utilities may forecast their peak demands conservatively (i.e., project higher peaks), which would lead to relatively lower PRMs. Others may deal with load forecast uncertainty by specifying a higher PRM rather than building conservatism into the load forecast itself. Some utilities may include forced outages in their PRM calculations, while others consider them elsewhere.

RA reliability targets
It is also worth noting that the IRPs which translate between PRMs and probabilistic RA metrics do so using very different methods. A few LSEs provide detailed RA assessments in their IRPs. For example, Xcel Energy includes a stochastic analysis of the relationship between electric generating capacity reserve margin and the ability to maintain system reliability. They use the historical peak load as well as generation data to fit the LOLH as a function of reserve margin level, and then determine the PRM that is equivalent to a 1-day-in-10-year LOLP by interpolation.

Load forecast
Forecasting load is a crucial aspect of a utility's RA assessment. Utilities employ different methods to project peak demand and these forecasts drive the amount of capacity that the system must have available. Table 3.3 summarizes the methodologies and key inputs that utilities use in load forecasting, as well as the annual peak load growths that they project. The annual peak load growths generally fall between 0% and 2%, with APS being the exception with a 3.3% annual growth rate. In addition to the peak load forecast, the energy forecast is also crucial in RA since the maximum capacity power shortage may not coincide with peak demand. The energy demand forecast is particularly important for utilities that rely on a significant amount of generation from hydroelectric resources (e.g., utilities in the Pacific Northwest), because the total hydro energy available can vary considerably from year to year depending on meteorological conditions. Typically, the load forecast is segmented by customer class with the residential load forecast often serving as the most important component. The residential load forecasts are usually expressed as the product of the projected number of customers and projected usage per customer, which are treated separately. Utilities tend to apply statistical or regression models to forecast local population growth, which is highly related to population dynamics and local/regional/national economic trends. For the usage per customer forecast, some utilities develop end-use models which disaggregate residential usage per household into several electricity applications, while other utilities still rely on econometric forecasting models including factors like weather, consumption patterns, and personal income. By contrast, the forecasts for commercial and industrial loads are relatively easier to construct. Utilities often apply econometric models or regression models to forecast them, and those models are usually less complicated compared to the residential models. In addition, utilities prefer to apply individual forecasts for large-scale commercial and industrial customers or geography-specific customers like mining or irrigation. These forecasts are often established by the customers themselves or developed by specialized business managers with input data provided by customers.

Demand-side resources
Demand-side programs, including energy efficiency programs and demand response programs, help LSEs reduce the amount of energy and peak load they need to serve. These programs have been instituted as far back as the 1980s. Energy efficiency programs support customer-side investments that reduce the amount of energy required to maintain the same level of end-use energy service, and they can also help reduce the peak net demand if the energy savings are coincident with the peak. Typical energy efficiency programs include weatherization assistance for low-income households and the installation of energy-efficient LED bulbs. By contrast, demand response programs shift demand away from the peak hours of the day. Common demand response programs are interruptible loads, direct load control, and dynamic pricing.
All 11 utilities in our sample have ongoing demand-side programs, and more programs are proposed or in development to contribute to RA and other objectives. However, not every utility reports its estimated reduction of peak demand, and very few LSEs report the reduction disaggregated by its specific demand-side programs. Some utilities partially report the methods that they use to estimate the peak load reductions achieved through these programs. For example, Avista translates the peak savings attributed to demand response programs into a peak credit that differs depending on their durations. Generally, demand-side programs are projected to reduce the peak demand by 5-10% in 2030, but corresponding impacts on reliability are rarely reported.

Future resource portfolio
Once the load forecast is established, utilities can construct several alternative supply-and demandside resource portfolios that are deemed adequate according to their preferred RA metrics (e.g., peak demand plus PRM). Accordingly, we investigate the relationship between these resource portfolios and RA assessment from three perspectives, including the model, technology mix, and resource ownership.

Modeling approaches
We evaluate how the 11 utilities determine their preferred resource portfolios. All of the utilities apply capacity expansion models to develop and evaluate their preferred resource portfolios, and use deterministic or probabilistic methods to evaluate power system performance. Some utilities use proprietary capacity expansion models, while others use commercially-available software (e.g., Aurora). Additional modeling tools are also employed to support the supply resource portfolio. For example, OG&E relies on commercially-available software to calculate locational marginal prices and determine the costs and revenues for generators.
It is important to note that the reliability target is often determined prior to the construction of resource portfolios. In capacity expansion models restricted by PRM reliability targets, RA is enforced through a set of model constraints, i.e., the sum of available power from all generators is greater than or equal to the load forecast plus PRM, at multiple points in time. As RA assessment tools, these deterministic constraints are limited in that they cannot capture probabilistic loss of load events. Moreover, since the constraints will usually be binding for the preferred least-cost portfolio, utilities will build just enough capacity to hit the reliability target. Whether resources are truly adequate thus depends on the accuracy of that target. However, there is a general lack of transparency about how RA is actually assessed beyond the fact that the resource portfolios must satisfy RA-related constraints in the capacity expansion model, which is a general weakness of utilities' IRP processes.
Compared to a PRM-based capacity expansion model, a loss-of-load probability model is a more advanced tool for assessing the RA properties of alternative supply resource portfolios. Instead of deterministic PRM constraints, loss-of-load probability models simulate many scenarios based on multiple (and potentially correlated) uncertainties to ensure that a portfolio achieves acceptable RA performance. Avista uses its reliability assessment model-ARAM-to test the current resource portfolio's reliability metrics and the contribution of each resource. The ARAM model validates reliability by simulating 1000 potential scenarios with different loads, renewable power outputs, and forced outage rates. PGE applies E3's renewable energy capacity planning model (RECAP) to assess both capacity adequacy and capacity contributions in order to fulfill load obligations. The RECAP model requires inputs including an LOLE target and a wide variety of hourly load and generation behaviors, and it works by creating probability distributions for loads and resources via convolution methods. It then evaluates RA through sequential simulations with stochastic forced outage conditions.

Resource capacity credit
The technology mix of the existing and planned capacity varies significantly among utilities. The largest existing resources are still coal or gas plants for most utilities, but a significant fraction of these plants are expected to retire during the utilities' planning horizons. In many cases, these thermal power plants will be replaced by intermittent solar PV and wind resources, and battery storage, which raise new challenges for maintaining and evaluating RA.
Capacity value refers to the ability of a power plant to reliably meet peak demand. It is usually measured by equivalent firm capacity or a fraction of nameplate capacity. A high penetration of renewable generation complicates a utility's RA assessment (Ibanez and Milligan, 2014;Tanabe et al., 2017), since wind and solar PV have lower capacity values than conventional resources due to the variations of available power across time and space, which cannot be perfectly forecasted. Utilities must develop credible methods to estimate the capacity values of renewable resources, especially their capacity values during peak hours (Munoz and Mills, 2015;Zhou et al., 2018). In short, if the utility must build enough capacity to meet its peak load plus PRM, then it is critical to know how much capacity credit to assign to renewable resources that could be called upon to satisfy its load obligations. All 11 utilities in our sample include renewables as major supply resources and report their contributions to energy and peak demand. The methods they use to estimate capacity values are reported in Table 3.4. Some utilities present their estimated capacity values of renewables on peak, but do not clearly state how they are calculated. Other utilities provide detailed narratives on how they treat the intermittency of renewable energy. Avista applies a stochastic method combined with autocorrelation algorithms to capture the shapes of renewable generation profiles. For example, from a historical dataset of hourly wind outputs, they produce 15 different wind generation profiles and use them to estimate hourly supply.
The effective load carrying capability (ELCC) of a supply resource is defined as the amount of incremental load a resource can reliably serve, while also considering probabilistic parameters of unserved load caused by forced outages, load uncertainty, and other factors (SPP, 2019b). ELCC values depend on many factors including technology, location, historical load shape correlation, and unit size. For variable renewables, ELCC is more comprehensive than simple capacity values because the ELCC captures the correlations among the resource's own output, the outputs of other variable resources in the system, and the load. Several utilities do use the ELCC to assign capacity values to renewables, and it is a popular metric for assessing capacity credits. For example, SMUD uses modeling tools and datasets from NREL to produce hourly renewable outputs as net dependable capacities at the summer peak. Avista simulates hydro, load, wind, and forced outage rates to estimate the contributions of resources available to meet the peak determined by its LOLP target. Xcel estimates wind ELCC by replacing wind generation with a constant generator while maintaining the same LOLE performance. These IRPs reflect the widely perceived trend whereby the marginal ELCCs for both solar and wind resources decline as the amounts of these resources grow, since the remaining capacity need is less aligned with the generation profile .
Hydroelectric resources contribute significantly to meeting the peak demand in the Pacific Northwest region. Assessing the capacity contribution of hydroelectric requires consideration of a wide range of factors from annual variation in rainfall/snowpack to operational rules. Similar to solar and wind, Avista performs an ELCC study on both run-of-river and pumped hydro resources. PGE calculates hydro contributions to peak load satisfaction by building probability distributions of monthly generation based on historical flow conditions and data provided by NWPCC. Moreover, the nature of hydro generation prompts the need to evaluate energy resource adequacy in the Northwest, which is distinct from the typical focus on capacity in RA. Energy RA refers to there being sufficient supply of fuels and energyconstrained resources to satisfy demand over longer periods (months, seasons). Energy RA is an important issue in systems with substantial hydro resources due to the large inter-seasonal and interannual variations in resource availability. Currently, the energy limitations of resources are considered in capacity RA programs. Utilities in the Pacific Northwest may leverage pre-existing hydrological forecasting tools to evaluate the expected energy surplus or shortfall to support their RA assessments.

Market transactions
Utility-owned plants, which account for more than 80% of current generation mixes, are the most prevalent resource type (by ownership status) in the existing portfolios of most LSEs. Utility decisions to "buy versus build" are often informed by a reliability obligation, which is a strong driver for resource ownership (Carvallo et al., 2020). Nevertheless, firm capacity contracts and market transactions are a prevalent procurement and planning strategy, respectively.
Several utilities have commissioned studies into the availability and economic benefits of market transactions. Typical market study components include power supply, electricity price forecasts, power transfer ability, and other operating characteristics. Avista uses the AURORA model to simulate the operation of the WECC system and analyze market conditions. PGE hired E3 to model the Northwest regional power system and analyze its market position. SCE, PG&E, and SDG&E are participants in the California RA program. CAISO and the CPUC propose several provisions and enhancements to its RA program, including the consideration of systematic forced outage rates and the assessment of unforced capacity value based on the proposed RA capacity. Similarly, KC-BPU and OG&E indicate in their IRPs that thorough deliverability studies can help utilities estimate their market positions and support their RA assessments.
Utilities generally use firm capacity contracts as substitutes for their own capacity additions to meet load deficits. However, the future availability of market transactions is less certain than that of utilityowned resources whose commission is under the control of the LSE. Regional RA studies would help utilities ensure that their assumptions about the future availability of market transactions are compatible with each other, with planned capacity additions across the region, and with the transmission capabilities of the power system.

Transmission expansion
Utilities must develop robust transmission systems to enable the delivery of power from generation assets to loads. Excess generation capacity in one location cannot address a capacity deficit in another location if there is insufficient transmission capacity linking the two points.
All LSEs include regional transmission planning processes in their IRPs, but these transmission analyses vary considerably in their degree of detail. Some utilities perform detailed analyses of future transmission needs and typically find that existing transmission networks are not binding constraints on reliability in the present and immediate future. For instance, APS uses both probabilistic and deterministic approaches to assess the reliability of its transmission system and presents its expansion plan for transmission lines, transformers, and additional investments. TEP performs a sensitivity analysis involving transmission needs and the adoption of demand-side programs, and it determines that no additional transmission facilities are required. SMUD estimates its maximum transmission capacity needed to secure import capabilities to its territory and points out that no additional transmission is currently needed in order to support RA. PNM uses the SERVM model to assess the power it can transfer from or to its neighbors based on the topology of its transmission network, and uses results to generate import constraints based on transmission capabilities.
There are other utilities that rely on transmission impact analyses provided by their neighbors, or by the regional system operator (e.g., CAISO, SPP). For example, Avista coordinates its transmission planning activities with neighbors and determines its transmission plan together with multiple organizations including WECC, RC West, NWPP, ColumbiaGrid, and Northern Tier Transmission Group. OG&E's transmission system is directly interconnected to others through the SPP regional transmission organization, and OG&E depends on the SPP Transmission Expansion Plan to assess its transmission capabilities and needs. Their analysis confirms that the evaluation of transmission expansion is becoming increasingly important because resources are becoming more geographically diverse and shared among utilities (Luburić et al., 2018).

Emerging technologies
With the ongoing evolution of electricity technologies and market structures, a variety of emerging technologies are being incorporated into utility RA assessments. In addition to traditional demand-side programs, utilities often benefit from other demand-side resources, including distributed generation and electric vehicles. On the supply side, storage can work efficiently to shift loads away from the peak, mitigate renewable curtailment, and thus help LSEs address RA issues. These innovative technologies can all provide RA benefits, but they also introduce new challenges for utilities planning for RA.
Distributed generation, typically distributed solar, contributes significantly to meeting the peak load as a demand-side resource for some utilities. Distributed solar is often assigned the same capacity values as utility-scale solar resources. Most LSEs mention the reliability of distributed generation in their IRPs, and some of them project the expected capacity contribution explicitly in future years. Notably, PGE conducts a detailed study on how distributed resources will affect its load profile and capacity position. There are also a few IRPs that do not forecast distributed generation directly, but include it in the set of demand-side programs or as a factor built into the load forecast. For example, Xcel mentions the increasing trend in distributed generation as a major uncertainty in its customer demand forecasts. Normally, distributed generation should yield significant benefits for RA by reducing the net load that the utility must satisfy and relieving stress on congested transmission and distribution systems (Al-Muhaini and Heydt, 2013). However, in designing the utility's resource portfolio to cost-effectively achieve RA, it is important to accurately project distributed generation investments and their alignment with load and other generation profiles.
Electric vehicles (EVs) could play a significant role in future RA as their adoption increases. EVs could be a major source of load growth that requires utilities to add more capacity in order to maintain the same level of RA performance. EV charging could also be leveraged as a controllable load to limit the effect of their energy demand on the peak load and to absorb renewable energy during times of high supply relative to demand. Unfortunately, thoroughly accounting for EVs in an RA assessment is very difficult due to the lack of historical data and myriad related uncertainties. However, their increasing presence in IRPs over the past several years reflects growing recognition of their importance in the near future. Seven of the 11 LSEs mention the impacts of EVs in their IRPs and five of them consider EVs as an individual component. Avista and TEP use relatively simple methodologies to forecast EV load, while PGE, SCE, and SMUD apply more complex optimization models to forecast system-level EV load based on changing dynamics of competing vehicle, infrastructure, and consumer attributes. Their forecasts show that assumptions about EV adoption will have a significant influence on projected load obligations in the near future.
Energy storage could be used to improve reliability considerably, but its performance is difficult to evaluate at this stage. Storage could provide peaking capacity to satisfy peak loads, mitigate problems associated with ramp rates, and absorb renewable generation to prevent curtailment (Denholm et al., 2020;Stenclik et al., 2018). In addition, well-sited energy storage can reduce the need for new transmission and distribution assets (Xu and Singh, 2012). Most utilities include energy storage as a future resource and study its economic benefits. Some utilities conduct an ELCC analysis on storage resources with different durations, especially when they are coupled with solar PV. For example, PGE calculates ELCC values for four types of storage and indicates that storage resources with longer duration have higher contributions to peak demand. PacifiCorp finds that energy storage can provide nearly 100% ELCC as a substitute for peaking generation that only needs to supply power during short periods. However, similar to renewable energy, the ELCC value of storage declines significantly as more storage is deployed because storage does not generate energy. Utilities also state that, compared to the market electricity price and the cost of a CCGT plant, large-scale energy storage is not yet an economically competitive way to meet capacity needs.

Treatment of uncertainty in RA modeling within IRP
Reliability is inherently a probabilistic concept, and some utilities explicitly incorporate uncertainty into their RA assessments. However, formally integrating uncertainty into RA assessment requires advanced mathematical techniques and more comprehensive input data. Interestingly, while IRPs tend to account for a number of uncertainties, LSEs tend to focus more on economic uncertainties than RA uncertainties in their IRP analyses. Table 3.5 reports whether each IRP in our sample incorporates uncertainty into various elements of its RA assessment.

Load obligation
Load obligation uncertainties mainly include the uncertainty in the load forecast and the uncertainties in traditional demand-side programs, distributed generation, and electric vehicles. LSEs consistently account for risk in their load forecasts. All 11 utilities in our sample conduct scenario or sensitivity analysis on the load forecast. Some utilities evaluate the performances of their portfolios under high or low load growth sensitivities, while others construct alternative portfolios based on different load forecast scenarios. A few utilities explicitly report how their load forecast scenarios are developed. For example, OG&E establishes its peak demand forecasts according to weather scenarios based on historical weather data. Specifically, it produces a probabilistic range of forecast outcomes of peak demand levels based on different weather occurrence probabilities, with a maximum forecast reflecting weather whose peak demand would be exceeded only once in 30 years and an expected forecast whose peak demand would be exceeded once every two years. PNM uses the upper limit 95% confidence level of its load forecast interval to construct its high load forecast scenario, which reflects an assumption of optimistic economic growth. TEP simulates 60 years of historically calibrated weather scenarios to generate probabilistic peak forecasts and incorporate them into sensitivity analysis.
Demand-side programs, distributed generation, and transportation electrification can also have dramatic impacts on power demand. Their contributions to peak load will depend on a number of highly uncertain factors including technology and program adoption rates, distributed generation profiles, and customer behavior such as when they choose to operate appliances and charge EVs. However, only some of the utilities evaluate these uncertainties. APS and TEP consider expanded demand-side management portfolios in their IRPs, and they show that expanded demand-side programs can provide slight capacity benefits but cannot cover the capacity deficit caused by deferrals in additional capacity or earlier retirement of existing capacity. Avista constructs and analyzes different scenarios of rooftop solar and EV adoption trajectories over time. PGE investigates scenarios with different adoption pathways for energy efficiency programs, distributed generation, and EVs. The uncertainties in those elements can be incorporated into load forecast scenarios, but they can also be modeled stochastically as supply resources with seasonal, hours per year, and hours per day variation.

Power supply
Conventional resources still occupy large shares of the electricity supply mix across Western U.S. states. Though conventional resources are typically more stable, fossil fuel-fired power plant retirements, additions, and upgrades (often driven by fuel prices, environmental regulations, and technological upgrading of the fleet) must be viewed as important considerations in a utility's RA assessment (Wilkerson et al., 2014). Most utilities report explicit retirement schedules in their IRPs, and the counterfactuals in which the plants do not retire are often presented as alternative portfolio scenarios. In addition, several utilities develop scenarios or conduct sensitivity analysis on environmental policies, and investigate how the capacity forced to retire would be replaced to assure RA. For example, SMUD examines three long-term planning scenarios based on multiple GHG reduction goals in California, which would result in different efforts to maintain RA. The reliability challenges raised by retirements have also been referenced by other system operators outside the Western United States. For example, the increasing reliability risks associated with forced coal plant retirements have been discussed for PJM (Lueken et al., 2016) and MISO (Drom and McMurray, 2013).
LSEs are also challenged by many uncertainties related to renewable energy and storage. Nearly all of the 11 utilities consider the risks of solar and wind, but most of them only consider their potential investment cost reductions, which have limited impacts on the RA assessments. A few LSEs investigate the risks of increasing renewable penetration and its relationship with reliability. For example, PNM evaluates a range of capacity factors for new wind resources and characterizes the system reliability according to their geographic locations. About one-third of the utilities include sensitivities on the efficiency of storage, and they show that storage has great potential to reduce capacity needs when its cost falls or efficiency improves, especially under high additions of renewable generation resources (Denholm et al., 2020;Xu and Singh, 2012).
Market risks are treated as major uncertainties in utilities' IRPs, but beyond the economic consequences of market electricity prices, only half of the utilities include sensitivity analysis on the relationship between the capacity deficit and the level of market imports. For example, Avista conducts a sensitivity analysis regarding the level of imports in its regional resource study. The LOLP and required capacity additions are projected under different load and import levels, which show that the region is at risk without new resources unless load falls or imports increase. PGE constructs scenarios featuring different market availability of California electricity imports for winter and summer, capped by the 95 th percentile transmission capability constraints. PNM emphasizes the risk due to underlying market assistance and implements a reserve margin sensitivity analysis by capping the market assistance at different levels. The simulation indicates that the market interaction is a significant assumption and PNM's current 13% PRM target is not sufficient to meet a 0.2 events/year LOLEV requirement because the expected market imports are not guaranteed to be available.
Construction uncertainties refer to the risks of construction delays for new assets, and other future considerations for resource maintenance, plant upgrades, and transmission expansion. Only three of the 11 LSEs account for such uncertainties in their IRPs. For instance, APS discusses several construction risks such as permitting delays, equipment delivery delays, and liquidated damages, and explains how these uncertainties can be mitigated.

RA assessment methods in the academic literature and comparison to current practices
There is a significant body of academic research on RA and electricity system reliability. In this subsection, we briefly summarize some state-of-the-art RA assessment methods in the scholarly literature, and then compare them to the current practices we observed in our sample of 11 utilities.

Relevant literature
The literature on traditional RA assessment methods in IRP is quite large, and a group of emerging studies focuses on the treatment of uncertainty in RA modeling. Since uncertainty pervades all aspects of electric utility planning (Hirst and Schweitzer, 1990), utilities should ideally develop credible stochastic methods to determine the optimal resource portfolio while mitigating reliability risks and potentially satisfying a probabilistic RA target. Stochastic optimization approaches, which consider all possible scenarios and their respective probabilities, have been widely applied to incorporate uncertainties such as load growth (Ioannou et al., 2019) or renewable availability (Ding and Lee, 2016;Milligan et al., 2017) into RA modeling and decision-making. On the other hand, there are also some studies that use robust optimization models to analyze system reliability. Robust methods address uncertainties by evaluating RA performance in the worst-case scenario that is considered plausible (Gazijahani and Salehi, 2018;Lueken et al., 2016).
In addition, given that electricity systems involve interactions among utilities, customers, third-party generators, and other strategic agents, there are many studies that apply game theory methods to investigate the effects of various policies and strategies. Several papers employ game theory models to examine the designs of demand-side programs (Erkoc et al., 2015;Hurley et al., 2013;Yang et al., 2013), incentive mechanisms for distributed generation (Gazijahani and Salehi, 2018;Issicaba et al., 2012;Mitra et al., 2016), and contract negotiations (Mays et al., 2019), all of which influence a utility's RA position. These works indicate that utilities could benefit greatly from well-designed policies and strategies to support RA assessments.

Comparison
Though myriad advanced methods have been developed in the academic literature, they are usually not included (or at least they are not clearly reported) in utilities' IRPs and RA assessments. There are several possible reasons for these discrepancies. First, academic methods tend to optimize on a specific component in RA assessment, but real assessment needs to consider multiple objectives. Second, the IRP process has a strict timeline and a limited budget. The most advanced academic methods would require dedicated expertise and substantial time commitments that utilities may not have available, or even be able to hire as third-party consultants. Third, some of the most conceptually sophisticated methods in the literature may not be ready for practical deployment on large-scale, real-world problems. This situation can arise due to the high computational burden of solving advanced model formulations on the sizes of systems that are encountered in reality, as academic applications are sometimes limited to small test systems. Alternatively, even when a method is computationally tractable, it might require input data that are unavailable for a particular electricity system. Finally, regulators may prefer tried-and-true approaches.
It is worth noting that some advanced methods have been incorporated into commercially-available software programs that support utility RA assessments. However, the detailed models contained in these software products are generally not publicly-available to review. For this reason, we are unable to assess the methods these programs rely upon and see how they align with the most recent academic research.

The Northwest Power Pool RA program
The Northwest Power Pool (NWPP) Resource Adequacy Program is an emerging proposal to establish a regional resource adequacy process with binding commitments for the participating members of the NWPP. The NWPP RA program seeks to maintain reliability, increase transparency of the resource adequacy position for the region, and efficiently utilize reserves and resources. This section introduces the main design elements of the NWPP RA program and is based on publicly-accessible information through August 2020.

Background
The NWPP was originally formed in 1941 to promote coordination among its members. Today, the NWPP has evolved into a voluntary organization that promotes operational coordination among its 34  Figure 4.1).

Figure 4.1 Balancing Area Authorities in the NWPP (within red line) and WECC
In recent years, there have been concerns in the Northwest that the combination of increasing coal retirements, growing use of renewable generation, and the greater reliance on future market transactions to meet capacity targets are leading to a resource adequacy problem. A series of forward-looking studies over the past several years indicate that the Northwest's collective capacity margins are declining and increasing the risk of potential electricity shortfalls (BPA, 2019; E3, 2019; NWPCC, 2018). These studies indicate that the Northwest region could begin to face capacity shortages by the early 2020s followed by severe capacity shortages in the mid-2020s.
Concerns about resource adequacy prompted discussions among the Northwest electric utilities and the NWPP. In the summer of 2019, NWPP convened stakeholders and organized work groups to collect information and explore potential solutions. The outcome of this collaborative effort became the NWPP RA program. The NWPP has pursued this resource adequacy effort with the support of a group of funding partners (See text box next page). The NWPP RA program is moving forward by following a set of predefined phases. The initial coordinated work group effort-tasked with information gathering-was designated as Phase 1. The Phase 1 work groups reviewed existing regional studies on resource adequacy, reviewed existing resource adequacy planning practices of Northwest utilities, examined best practices of existing resource adequacy programs, investigated potential problems from fuel constraints and transmission deliverability, and communicated these results to the public. The consulting firm Energy and Environmental Economics (E3) developed a report that summarized the findings of the work groups and proposed potential options towards a collective regional approach to the resource adequacy concerns (NWPP, 2019). On October 2, 2019, the NWPP organized the Resource Adequacy Symposium in Portland to discuss the findings of the work groups and explore potential options for collective solutions. The Symposium marked the conclusion of the NWPP RA program's first phase.

Members of the NWPP
Phase 2A began in the fall of 2019 with the objective of developing a preliminary program design for the NWPP RA program. The project Steering Committee took the first step by developing a list of design objectives for the NWPP RA program. These design objectives, which are listed in Table 4.3, served as the foundational principles that would shape the initial design of the program.

Design objective
Ensure that Balancing Authorities and Load Serving Entities can continue to operate safely, efficiently, and reliably. Ensure that the recommended RA program and its components delivers investment savings through diversity benefits. Ensure RA program respects local autonomy over investment decisions and operations and continues to respect the rights and characteristics of individual utilities, transmission service providers, BAs and other entities through program design. Make recommendations that are acceptable within the current and evolving regulations and requirements of each applicable federal, state, and local jurisdiction. Ensure that the participation, evaluation, and qualification of resources is technology neutral. Ensure that all products and services transacted to meet the requirements of the RA program are well defined, voluntarily transacted through existing competitive market frameworks and accurately tracked. Ensure that the proposed RA program can be extended to other regions in the West. Ensure that entities that voluntarily choose to participate in the RA program equitably pay and receive benefits for services provided by the program. Ensure the RA program provides efficient long-term investment signals as well as a process for exit and entry of resources. Phase 2A work groups were formed to specify the key design elements that were consistent with the design objectives. A Stakeholder Advisory Committee was organized to provide periodic feedback on the proposed design elements for the NWPP RA program. Additionally, the NWPP RA program held public webinars to provide information and collect feedback on the proposed design elements emerging in the NWPP RA program. By July 2020, the NWPP completed its Phase 2A preliminary program design. The following section summarizes the framework and key elements of the NWPP preliminary RA program design based upon the public documents and presentations released during Phase 2A.

Forward showing program
The foundation of the NWPP RA program is the creation of a forward showing program. The general principle of a forward showing program is that each participating member of a regional resource adequacy program would be responsible to show that they have capacity resources to meet their respective expected peak load forecast plus a designated capacity margin. The total capacity across the participating members will match or exceed the expected regional peak load plus the desired capacity margin as long as all participating members demonstrate that they have capacity to meet their own expected peak loads. The challenge for the NWPP, or any entity developing a forward showing program, is to clearly define key parameters, establish incentives to promote compliance, identify the capacity resources in the region, and ensure that there is a transparent and verifiable enforcement.

Seasonal binding periods
The NWPP's preliminary design proposal calls for a forward showing program with two binding seasonal periods corresponding to a summer peak load and a winter peak load. The NWPP RA program proposal defines the winter season to start on November 1 and end on March 30. The summer season would begin on June 1 and end on September 30.
Member entities would be required to show they have sufficient resources to meet their respective binding seasonal targets seven months prior to the start of the winter and summer seasons. The specific dates for the start of each season and the showing deadline are as follows. For the winter season starting on November 1, member entities would need to make a demonstration of sufficient winter capacity seven months earlier on March 31. Similarly, for the summer season starting on June 1, member entities would need to demonstrate their respective summer capacity seven months earlier on October 31.
The NWPP RA program proposes a two-month cure period if members fail to meet either the binding winter or summer deadlines. Member entities that fail to meet the winter deadline on March 31, would be allowed to correct the deficiency by arranging for additional capacity by May 31. If an entity failed to meet the summer deadline on October 31, it could correct the deficiency by obtaining the required additional capacity by December 31. Member entities that fall short of their showing capacity target would have the option to add new capacity or obtain the equivalent amount of capacity through bilateral market transactions. Member entities that fail to address a capacity shortfall would be subject to binding financial penalties. The proposed penalty for failing to meet a binding showing obligation would be based on the cost of new entry (CONE). The CONE value times the gap to meet the capacity target would equal the total penalty value. Additional details will be worked out in the Phase 2B Detailed Design Phase.
The NWPP RA program proposes that a Program Administrator would be the entity that verifies the capacity targets for each entity and determines whether each member entity has met its winter and summer binding capacity deadlines.

Load forecast parameters
The NWPP RA program proposes a set of demand-side load forecast parameters to derive peak load forecasts in the winter and summer by the participating load serving entities and collectively for the region. Peak loads are based on the expected peak load in a season with a 50/50 or one-in-two probability 2 . Consistent with traditional planning practices, the expected load forecast is supplemented with contingency reserves, and a PRM that provides an extra buffer for the planning horizon. The combined expected load with contingency reserves and a PRM provides a measure of "pure capacity" needed to meet expected loads in the planning horizon.

Resource eligibility
Accurate accounting of the capacity resources is an important foundation when performing a resource adequacy analysis. The NWPP RA program proposes to require a registration and certification process for all resources. Under this certification process, the capacity contribution of an existing resource would be based upon its historical performance. The capacity contribution of future resources added after the implementation of the RA program would be determined by the Program Administrator.

Resource parameters
The NWPP RA program also proposes a set of supply-side parameters to measure capacity contribution of different types of resources in the supply and demand side. For thermal generators, the traditional approach has been to look to a generator's installed capacity with downward adjustments ("derates") for temperature and outages-the ICAP methodology. The NWPP RA program proposes to use the UCAP methodology, which provides a more comprehensive approach utilizing resource specific outage metrics. Variable energy resources such as wind and solar generators would be assigned a capacity value based on its ELCC methodology. The ELCC values would need to be developed from probabilistic modeling that account for seasonal and daily weather patterns, diurnal solar patterns, output correlation with other resources, and down time for maintenance and repairs. Run-of-river hydro generators would also use the ELCC method to derive its effective capacity contribution. Identifying the capacity contribution from large storage hydro in the Northwest is a more challenging modeling exercise given the interdependencies among dams and operational practices and constraints on these dams. The NWPP RA program proposes to enhance hydro modeling efforts to address the contribution from large hydro resources. The NWPP plans to develop additional parameters related to the capacity contribution of other resources including, but not limited to energy efficiency, demand response, batteries, and pumped storage.

Operational program
The NWPP RA program is designed to be a cooperative agreement among utilities in a region to share capacity resources to meet regional resource adequacy targets. Other regions that have developed resource adequacy programs have generally been built upon formal wholesale markets (e.g., PJM and SPP). There is not a formal wholesale market in the footprint of the NWPP and the NWPP RA program does not intend to create a centralized market operator. Rather, the operational program would impose constraints on transactions between members and non-members for capacity resources. Issues to be addressed in the operational program include the following:  When and how could a participant access pooled capacity resources?
 When and how might a participant be obligated to provide resources to other participants?
 Could a participant rely on imports from outside the region or provide export outside the region?
It should be noted that a preliminary framework has been developed for the import and export question. The NWPP RA program proposes the following modeling assumptions for purposes of forecasting load and resource balance in the NWPP footprint with imports and exports. Participants may be able to rely on imports to meet their capacity needs during hours of a binding season. These participants would need to demonstrate a regional load credit by showing an expected and reliable import level. The CAISO has performed historical data analysis of resource adequacy contracts and transfers that provides a basis for calculating expected future imports. By contrast, expected future exports by NWPP members are assumed to be a surplus that does not impact load or the planning reserve margin within the NWPP footprint. Expected exports are a surplus to the NWPP region and therefore unavailable to other entities in the NWPP. The NWPP RA program expects to have more robust data about forward contracted imports and exports as the NWPP approaches implementation of the RA program. This data would inform the modeling of expected imports and exports from the NWPP footprint.

Implementation
In July 2020, the NWPP RA program completed its Phase 2A work on developing a preliminary design of the RA program. The Phase 2B Detailed Design period builds upon the outline framework of Phase 2A and requires more specification and details for the complete design to implement a working program. The NWPP recently announced that it will hire SPP as a Program Developer to help manage the design and startup of the program (NWPP, 2020). Additionally, the NWPP intends to select a Program Administrator that would be responsible for carrying out the implementation and operations of the program.

Program administrator
The NWPP RA program is designed to be a voluntary contractual organization. Utilities in the footprint have the discretion to voluntarily join the NWPP RA program. Once a utility formally joins the program, however, it will be contractually committed to its privileges and commitments. The Program Administrator will play a critical role as the centralized administrator that verifies and enforces the requirements of the program. Specific duties would include the validation of member capacity showings, the validation or development of peak load forecasts, and the enforcement of deficient showings and capacity shortfalls. Failure to enforce the requirements of the program would undermine the intent and purpose of the RA program.

Phased implementation
The NWPP plans to gradually implement the requirements of the RA program over three distinct stages. Stage one of implementation would begin with holding two non-binding forward showings covering a winter and summer season. This would effectively be a trial run for all participants in executing a forward showing. The Program Administrator would communicate the requirements of the forward showing to the member participants. Participants would submit the required information such as expected loads and capacity resources according to established timelines. As a non-binding exercise, there would be no penalties imposed during the initial two showing periods.
Stage two of implementation would proceed with two binding forward showings that include a summer and a winter showing. The Program Administrator would initiate the requirements to the participants and the participants would respond with the required information. Unlike stage one, the Program Administrator would impose penalties on participants that did not comply with the requirements.
The third stage of implementation would launch the combined implementation of two binding forward showings (winter and summer season) and the use of a full operational program. The intent of the operational program would enable participating members to access pooled regional resources in a structured program. Specific details about how the operational program will actually work in practice are still under development. NWPP intends to implement stage three in 2024.

Governance and administration
Development of the NWPP RA program is currently overseen by the NWPP RA program Steering Committee consisting of representatives from the funding participants. As noted above, NWPP hired SPP to become the Program Developer that will guide and assist with the design process. A separate Project Administrator will be hired to carry out the showing program and the operational program. Additionally, the NWPP parties will likely create a new governance structure organization with bylaws and a governing board that will serve as the home for the NWPP RA program.

Institutional alignment within the Western Interconnection
The architects of the NWPP RA program are attempting to build a program that complements the unique history and culture of the Pacific Northwest and the Western Interconnection. Hydropower has been the dominant resource in the electric sector and the regional economy of the Northwest. The Northwest's electric sector institutions such as the Bonneville Power Administration, public utility districts, and municipal entities have historically been very independent and skeptical of organized electric wholesale markets. Formal organized electric markets are the exception in the West with ISO markets only in California and Alberta. Besides these markets, the Western Energy Imbalance Market is a real time market that was launched in 2014 and currently has 11 active participating utilities and 11 pending participants over the next several years. In contrast to other regional resource adequacy programs like SPP, the NWPP RA program will not be built upon a wholesale electricity market. The NWPP RA program is designed to pool and share resources using traditional bilateral transactions among its members. Such transactions to meet RA needs are expected to be infrequent and rare.
The establishment of the NWPP RA program should not significantly alter or modify how its members interact with other institutions and practices in the Western Interconnection. The Northwest Power and Conservation Council (NWPCC) performs a regional RA analyses in the Pacific Northwest region as part of its role to develop regional power plans. These analyses have been advisory and informative to the states, utilities, and regulators in the region. NWPCC will continue to have a mandate to develop RA analyses and power plans after the implementation of the NWPP RA program. The work of the NWPCC will be informative and likely complementary to the goals and tasks of the NWPP RA program.
WECC serves as the reliability entity in the Western Interconnection which enforces reliability standards and performs reliability assessments. WECC's mission is to effectively and efficiently mitigate risks to the reliability and security of the bulk power system in the Western Interconnection. In 2020, WECC identified resource adequacy as one of its top reliability risk priorities. WECC has also recently taken steps to improve its RA analysis of the Western Interconnection. WECC's RA analysis will likely continue to be advisory and informative to its stakeholders. WECC's regional RA analysis and the NWPP RA program could become mutually complementary by advancing the art of forecasting and analyzing RA in the NWPP footprint and the entire Western Interconnection.
FERC regulates the sales of electricity in interstate commerce. Some utilities in the NWPP footprint are subject to FERC jurisdiction and others are not. One of the NWPP working groups examined whether a fully functioning NWPP RA program would be subject to FERC jurisdiction. This work group observed that FERC jurisdiction can be triggered under the Federal Power Act by an agreement that affects the rates, terms and conditions of sales of electric energy for resale in interstate commerce or transmission of electric energy in interstate commerce. Since the NWPP RA program would impose binding commitments and financial penalties on the participating members, the working group concluded that FERC would likely have jurisdiction of components of the NWPP RA program. FERC jurisdiction would likely apply to the Program Administrator and the future governance structure of the NWPP RA program will have to meet FERC's independence criteria.
Under the independence criteria, individuals serving a NWPP RA program governing board could not be employed by member participants or have a financial interest in the member institutions. The WECC Board of directors is an example of a board the meets the independence criteria. The NWPP RA program would need to assemble a governing board with qualifications and experience of a major organization but fully independent of the member organizations. Finally, states will have an important role in the activities of the NWPP RA program. States have exclusive jurisdiction over the facilities used to produce electrical generation and have historically regulated resource planning and resource adequacy for investor owned utilities. How this may play out for the implementation of the NWPP RA program is examined in Section 6.

The Southwest Power Pool experience
This section describes the current implementation of the resource adequacy program at SPP, which is an interesting case study because its service territory includes several states that have had IRP regulations for more than a decade. It follows that the SPP experience of implementing an RA program in states with IRP can provide an understanding of how the two processes have interacted over time. This section is based on semi-structured interviews with SPP staff and with public utility commission staff conducted in July, 2020. The interviews are complemented by SPP publicly available documents as well as IRP reports. The SPP documents include the resource adequacy regulations in the Open Access Transmission Tariff as well as supporting transmission planning and reliability reports.
SPP is an RTO that serves all or parts of 14 states in the Southern and Midwest United States, including Arkansas, Iowa, Kansas, Louisiana, Minnesota, Missouri, Montana, Nebraska, New Mexico, North Dakota, Oklahoma, South Dakota, Texas and Wyoming ( Figure 5.1). The SPP operational territory encompasses 575,000-square-miles and includes more than 61,000 miles of high-voltage transmission lines, more than 750 generation resources, more than 4,800 transmission substations, and serves a population of 18 million people (SPP, 2019a).
As a transmission provider, SPP has the duty to serve its balancing area peak demand. The RA program ensures that the LSE 3 has enough capacity available for SPP to serve peak demand and enough reserves to maintain a predefined planning reserve margin. RA in SPP is regulated by Attachment AA to the Open Access Transmission Tariff, developed in early 2018. It includes conditions and responsibilities for each LSE as well as the transmission provider, market participants, and generator owners. Attachment AA also specifies an annual "Workbook submission" as a means for market participants to submit their relevant information in a standardized format.

Figure 5.1 SPP operational territory (source SPP, 2018a)
The SPP has implemented an RA Program that provides insights on navigating the challenges with existing IRP mandates. This section provides background information on how RA is assessed and commitments created in the SPP, the challenges with integrating this program into IRP processes, and how these challenges were addressed. The first section provides details of the RA program in SPP. The second section delves into the shared components between IRP and the SPP RA program and their interaction.

Resource adequacy in SPP
The SPP RA program distinguishes LSEs, market participants, and generator owners. Market participants are responsible for ensuring that the LSE they represent comply with the RA requirement. LSEs can apply to be a market participant or be represented by one. One advantage of splitting the role of the LSE and the market participant is that the SPP rules allow for a single market participant to represent more than one LSE meeting certain firm power transaction conditions. In this case, the market participant is responsible for complying with an aggregate resource adequacy requirement based on the load obligation of the represented market participant.
LSE resource adequacy requirements in SPP are calculated by augmenting the summer or winter season net peak demand by the target 4 PRM. The target PRM is the result of a loss of load expectation study performed by SPP at least every two years that employs criteria and assumptions agreed upon by the SPP members. In this particular case, the LOLE is based on the typical criterion of a 1-day-in-10-years expected outage time. The SPP target PRM had historically been 13.6%, but in 2016, the SPP Board approved a reduction to 12%. This decision was supported by a significant amount of transmission buildout as well as load and generation diversity (SPP, 2016b). Market participants that are responsible to comply with RA requirements must do so by annually reporting the amount of deliverable and firm capacity available to them to meet their winter and summer net peak demand obligations.
The peak demand obligations are based on load forecasts that are developed by each LSE, which are typically based on their resource planning reports. SPP does not impose any methodological constraint on the forecasts, but does require that it is a 50/50 or one-in-two forecast. SPP does not develop its own forecast for each LSE, but for LOLE Study purposes aggregates the peak demand forecasts assuming certain levels of load diversity to account for their non-coincident nature. SPP conducts a post-season validation where the load forecast is trued up with the actual outcome; the results of this exercise are analyzed by the Supply Adequacy Working Group to identify potential issues associated with over-or under-forecasting.
SPP market participants can meet their RA requirement with deliverable and firm capacity resources. SPP conducts an annual deliverability study for the summer season to determine how much capacity a given resource can deliver within the SPP balancing authority area. Firm capacity is determined by each LSE based on the financial commitments as well as the deliverability of its resources. This means that an LSE can meet its RA requirement with owned resources as well as supply-and demand-side contracted resources. In these cases, the LSE submits the standardized "workbook" with detailed information of the deliverability and firmness of each owned and contracted resources that will be used to meet RA requirements for the next season.
The different studies and decisions made by SPP for RA purposes are governed by the following annual timeline: 1. The process begins on July 1 st when SPP posts a formal timeline for resource adequacy requirement compliance.
2. In October 1 st , SPP confidentially releases the results of the Deliverability Study to each generator owner, which indicates the deliverability results for their resources 5 . SPP also releases the workbook template for market participants to report their compliance status.
3. Market participants have until February 15 th to submit their workbooks to SPP. Failure to do so may result in all of their RA requirement be declared deficient and for deficiency payments to be triggered.
4. By April 1 st , SPP notifies LSEs that have not met their RA requirements. LSEs have 45 days to address any deficiencies and avoid penalties called "deficiency payments". These payments apply to the fraction of the RA requirement that has not been met, and the revenues are distributed among the LSEs that will provide capacity to cover this deficit.
5. On June 15 th , SPP publishes the report with the status of each LSE's resource adequacy requirement.
The RA status report release in June documents how each LSE is complying with its RA requirement, as well as an aggregate resource adequacy outlook for the SPP balancing area. The report expands on each LSE's deliverable supply-and demand-side resources and compares it to the peak demand minus demand-side resources including DR and distributed generation for that LSE. The latest RA report shows that the SPP balancing area has a ~21% reserve margin for the 2020 summer season, which decreases to 12.5% by 2025.
The SPP neither facilitates the LSEs' meeting their RA obligations nor does it evaluate the economic decisions to meet those obligations. However, over time, the SPP has created different opportunities to help LSEs meet their obligations. For example, interviewees indicated that the deliverability study was designed to help LSEs know beforehand the exact capacity credit for each one of their resources. In addition, SPP made more flexible meeting the use of bilateral contracts by not requiring them to be backed up by firm transmission service agreements to be eligible to meet RA requirements.
While SPP has responsibility over the economic operation of the integrated system, it does not have responsibility over the financial investment decisions of the market participants. This distinction allows for states to retain the prerogative to implement policies that can be reflected in their regulated entities' investment decisions. In this next section, we evaluate the interactions between state's planning mandates -codified in their integrated resource planning guidelines -and the SPP planning and operation criteria.

Resource planning in SPP
It was discussed earlier that IRPs develop a resource adequacy assessment, and that IRP is, in part, a resource adequacy compliance mechanism for state regulators with respect to their regulated utilities. Ten out of the 14 SPP member states with utilities had IRP regulations implemented as of 2020, and nine of them have had these regulations in place for more than a decade.
In this section, we examine the relationship between the different IRP regulations mandated by states within the SPP footprint and SPP's RA program. The objective of this comparison is to understand how IRP and the SPP RA program share or not assumptions, methods, and outcomes. This section employs two methods. First, we build off the IRP analysis in section 3 and examine in more detail the latest IRP reports from selected LSEs in the SPP footprint. Second, we expand this information through unstructured interviews with public utility commission staff and utility resource planners. Table 5.1 reports the entities interviewed and/or whose resource plans that were examined for this section. In SPP, resource planning guidelines reflect the relationship between the regulated entities and the RTO. In states with thorough regulatory guidelines, the relationship with SPP will be defined in the statutes themselves. For example, in Missouri the IRP guidelines allow the utility to include the RTO transmission planning outcomes providing they create economic benefit to Missouri rate payers (i.e. these are not solely reliability improvement projects) (MO CSR, 2011, p. 11). The Missouri IRP guidelines direct their utilities to verify how their demand-side rates that produce load shifting would be assessed in the RTO's resource adequacy determinations and considered as demand response (MO CSR, 2011, p. 13). The 2018 IRP from a Missouri-regulated entity, KCPL, reflects these guidelines by making extensive reference to SPP's transmission planning outcomes and processes when responding to the IRP statutes.
In states with broader IRP regulations (e.g., Oklahoma), the relationship between IRP and SPP RA guidelines is mediated by an LSE's SPP membership, which includes duties and responsibilities over resource adequacy. In cases where it is not specified how an LSE should implement an IRP guideline, the LSE has the incentive to follow SPP guidelines to avoid penalties. For example, OG&E's 2018 IRP report states explicitly that "the objective of this IRP is to explore options to maintain OG&E's generation capability in accordance with the SPP planning reserve margin requirement of 12% in a manner that achieves the lowest reasonable costs to customers, improves reliability and maintains environmental balance." (OGE, 2018). The definition in OG&E's IRP shows how the utility retains the responsibility of securing least cost, reliable, and sustainable supply, but at the same time complies with SPP guidelines.
The IRP components reflect the way SPP and the state guidelines relate. None of the load forecasting sections in the IRP reports reviewed for this subsection mention SPP. This is expected given that SPP does not subject its members to any requirements regarding their peak demand forecasting besides the 50/50 guideline. In contrast, the IRP reports from LSEs in the SPP footprint rely extensively on SPP's transmission planning to frame their resource adequacy analysis regardless of the thoroughness of the IRP rules. For example, the OG&E 2018 IRP report states "OG&E provides input to the SPP planning process, and SPP is ultimately responsible for the planning of the OG&E system." (OGE, 2018). In another case, OPPD (Nebraska) explicitly mentions employing the capacity accreditation factors for wind as determined by SPP's resource adequacy guidelines.
Upon the recognition of SPP as an RTO, FERC determined that states within the SPP footprint would retain rights over cost allocation, financial transmission rights, planning for remote resources, and resource adequacy. In its FERC filing to review its resource adequacy policy, SPP stated that "As the Balancing Authority, SPP is the entity responsible to, amongst other things, integrate the resource plans of the Resource Planners within its region." (SPP, 2018b, p. 2). The integration of these plans rests on the close relationship between states and SPP through the various committees and working groups, especially the Regional State Committee. Interviews with public utility commission staff confirmed that SPP membership creates opportunities for extensive communication between regulatory bodies and regulated entities within the various SPP committees. The interviewees indicated that ongoing work among states lead to consensus even when there were initial disagreements on a range of topics. This relationship has made IRP and SPP guidelines naturally follow each other as evidenced from IRP reports and statutes.

Impact of the NWPP proposed RA program on IRP
Section 3 identifies the typical components of resource adequacy that are part of resource planning processes. These components are organized in the first and second column in Table 6.1. In this table, the third column reports our assessments of how much these components would be affected by the NWPP regional RA program. The last column reports the control allocation over different components of resource adequacy resulting from an LSE joining a regional RA program.
Based on our research, two components of IRP may be significantly affected by a regional RA program: target reliability targets and capacity accreditation assumptions. Two components of IRP may be moderately impacted: load forecast and transmission expansion. Subsections 7.1.1 to 7.1.4 highlight the ways in which IRP and the NWPP RA program may need to align around the four components.
The adoption of a regional RA program will shift the control over a specific resource adequacy component from local control (e.g., IRP) to regional control (e.g., regional RA program). We classify these control allocation shifts into three discrete categories. A "Local" classification reflects resource adequacy components that will remain under state authority and discretion of state regulators through their IRP regulations. The regional RA program should be designed to be effective while deferring to the state authorities on these components. A "Regional" classification indicates that the regional RA program will assume effective control over a resource adequacy component that has typically been part of IRP. Uniformity of these particular components is critical for a functioning regional program. Divergent assumptions or modeling practices by members of the program would undermine its functioning. Finally, the "Shared" classification reflects a compromise arrangement whereby some control remains in the states while the regional RA program has a complementary role in other aspects of the component. The motivation for agreement on common regional standard can be driven by stateof-the-art studies that promote consensus and political agreements for the common good, among others. There are three core technical issues to determine resource adequacy requirements for a pool of entities at a regional level. The first is the amount of regional capacity required to meet net peak demand plus a reserve margin. The second is how this capacity -the resource adequacy requirementis allocated among responsible entities. The third is what the compliance mechanism is and what the potential consequences of non-compliance are. The first of these issues has the most overlap with existing resource adequacy assessments conducted as part of IRP. This is largely because, at the IRP level, the resource adequacy requirement usually falls directly on the reporting LSE and compliance comes through regulatory oversight at the procurement phase. This section then focuses on the components of IRP related to capacity requirement determination.
It must be noted that we base our analysis on a typical IRP implementation to determine the impact of the NWPP regional RA program. States' IRP guidelines are diverse, and they implement and emphasize these typical components in different ways. The insights reported in this section should help states identify the specific elements in their jurisdictional IRP guidelines that would potentially conflict with a regional RA program that their regulated utilities could join, and drive deeper analyses on these elements.

Resource capacity credit
Determining the capacity position in IRP depends on two key assumptions: the capacity accreditation for existing and new resources (see Section 3.1.3.2), and the target reliability metric (see Section 3.1.1).
Resource capacity accreditation will require much more alignment between IRP and the NWPP RA program. A key potential issue is that the capacity contributions of resources depend on their regional penetration levels and the regional peak demand, neither of which can be determined within a single IRP . In addition, states have historically assigned different capacity credit factors for similar resources (especially wind, solar, and demand response), which may create friction among the members if some states recognize much lower capacity than others for similar resources. Finally, if IRP and regional RA capacity accreditation for the same resource differ, there is a risk that an LSE would be adequate at the local-level but not at the regional-level and would have to justify additional investment outside its IRP recommendations to comply with regional resource adequacy requirements.
There are at least four resources that will require specific attention for their capacity credit calculation: variable renewable resources, demand-side resources, hydropower, and contracts.
Standardizing capacity credit for variable renewable resources is relevant because the capacity contributions of wind and solar decrease when their penetration increases . It follows that a regional capacity accreditation that considers variable resources pooled across the region would most likely produce a lower capacity credit than a local IRP assessment. Therefore, if states grant capacity credits that are different than what the RA program recognizes, an LSE may have a capacity deficit with the RA program, but meet the state's capacity requirement even when subject to the same target PRM.
It is important to note that renewable resource capacity credit methodologies and outcomes in IRP differ substantially across LSEs and states . States may need to relinquish some control over the capacity credit of their solar and wind resources to make them consistent across the NWPP footprint, probably relying on a regional calculation that also accounts for the aforementioned joint impacts of aggregate penetration levels. Across SPP, for example, states reach consensus on capacity accreditation for renewable resources and their LSEs' IRPs incorporate these assumptions into their analyses.
Standardizing capacity credit for demand response, energy efficiency, and distributed generation follows a similar logic as with utility-scale, variable renewable resources. The main issue is that there is substantial variation in the treatment of DR and EE capacity credit in IRP across states, as evidenced from the analysis in Section 3.1.2.2. Similarly, there is also substantial variation in the treatment of distributed generation across IRP (Mills et al., 2016). There may need to be a common agreement on how to classify and treat DR, EE, and DG resources for resource adequacy assessments to allow LSEs to show these forms of capacity in their IRPs as well as in a regional program. Furthermore, it is likely that the capacity contributions of these demand-side resources would also decline with higher penetration of similar DR, EE, and DG measures across regions, something that is not currently calculated as part of IRP.
The resource adequacy contribution of hydropower has its own complexities, as has been recognized by the NWPP. Hydropower is generally regarded as an energy-constrained resource that can supply most of its nominal capacity as firm. However, in extreme drought events even this capacity may not be available. From an IRP perspective, this resource has similar challenges as variable renewable resources in that there will need to be a common methodological practice across hydropower-owning LSEs to determine the capacity contributions of these resources.
Finally, the capacity contributions of bilateral and market transactions in IRP also vary substantially across states (Carvallo et al., 2020). This feature may make contracts the more challenging resource to homogenize in a regional RA program. Note that the treatment of market transaction within IRP will not be fundamentally impacted by an LSE joining a regional RA program. However, it is possible that the treatment of capacity exchanges within pooled resources without an explicit market will require some sort of centralized system to standardize, aggregate, and verify the capacity contributions of contracts. More generally, deliverability of resources outside of the LSE's balancing area has not been treated systematically in IRP rules and will be a critical component of a regional program for market transactions and utility resources. Section 6.3 examines the issues arising from the treatment of transmission expansion.

RA targets
As indicated in Section 2, the target reliability metric serves two purposes: to set a common regional resource adequacy target and to track the status of resource adequacy in the region. In some cases, an LSE may derive a PRM to use as a target from a complex probabilistic analysis of its power system, and in other cases an LSE may simply decide on a rule-of-thumb PRM and use it as a target to determine a resource adequacy requirement.
Regardless of the source of the target, this value should be the same or higher for an individual LSE compared to that of the RA program. If the state PRM is higher that would just cause the LSE to be super-adequate compared to the regional requirement; regulators can choose this if they want to forego cost benefits of pooled resources to maintain higher local resource adequacy. Our interviews revealed that this practice is relatively common among risk-averse regulators across the SPP footprint. It would be infeasible for an LSE member of a regional RA program to meet the lower PRM between an IRP and the regional program. Hence, regulators may need to update their IRP guidelines to assure that the regulated LSE will use the same PRM as in the regional RA program.
The experience of SPP suggests that states agreed on a target PRM based on LOLE studies developed by SPP. These discussions took place at the Regional State Committee over several years, and the PRM was approved and adopted by states and reflected in their IRP rules once consensus on its benefits was reached. The main benefit of a consensus minimum PRM target across members of a regional RA program is that there are no potential conflicts over LSEs cross-subsidizing the resource adequacy of others LSEs in the pool. When a minimum margin is met, all members know they are contributing their fair shares to the region's resource adequacy, even if some LSEs or states decide to be super-adequate.

Transmission expansion
A key role of an RTO is to develop transmission expansion planning studies to inform development in the region. The ways these studies lead to actual investment vary, but LSEs do rely substantially on these plans to inform their supply-and demand-side investment decisions. As found in Section 5, IRPs from LSEs in the SPP footprint make extensive reference to the RTO's planning studies to ensure that the resource mixes proposed in their IRP reports are technically feasible.
In contrast, IRP processes in Western states have generally been overly focused on generation technology choices, and transmission expansion studies tend to be implicit and not clearly reported in all IRPs, as indicated in Section 3.1.4. Moreover, transmission expansion studies typically focus on the LSE's local power system and not all IRPs include a regional analysis to gauge the deliverability of resources outside of the LSE's service territory. These limitations of current IRP processes could hinder the pooling of resource adequacy resources across the NWPP footprint, which is one of the main sources of cost savings.
From an IRP perspective, the main issue is how to ensure that the transmission expansion assumptions built into each IRP are consistent with the assumptions made at the regional level. A recommendation would be to develop a process to collect transmission expansion assumptions from each NWPP member, based on what its IRP is assuming. NWPP or regional entities subject to FERC Order 1000, including the CAISO and Northern Grid, would then collect these transmission expansion assumptions and determine the reliability levels that they bring to the footprint. This is essentially the opposite of what most RTOs do, but this process could help establish a common understanding of transmission expansion across a large footprint. A complementary process would rely on current transmission expansion studies developed by the WECC and the PNWCC, but it is unclear how these outcomes could be incorporated into IRP.
An open question, which extends beyond IRP, is how much control over transmission expansion states would need to give up to ensure a pooled capacity resource mechanism works efficiently across the NWPP footprint. The need for common assumptions for both local and regional transmission expansion may conflict with a state's desire to retain full control over the timing and choice of transmission project expansion across the NWPP footprint. The allocation of costs of transmission expansion is also a matter of interest, but out of the scope of this paper.

Load forecast
The SPP experience shows that load forecasting can be left to the member entities in the regional program provided that they develop and share forecasts with standardized statistical characteristics. The main issue of using forecasts with different statistical properties is that they reflect different assumptions about risk. For example, if a regional program is aiming to manage a certain load forecast uncertainty level, then an LSE whose forecast reflects less risk aversion would be benefiting from the efforts of other LSEs that are aiming to hedge more against load forecast uncertainty. It follows that specifying the statistical properties of the forecast should be consistent with regional agreements on risk assessment and management. According to Section 2, the statistical properties of existing IRP forecasts by Western U.S. LSEs differ. LSEs may not need to petition to change their IRP rules if they are willing to produce a specific and separate forecast for the NWPP RA assessment. However, it would be more efficient if at least one of the existing forecasts created for the IRP was directly applicable to the regional RA assessment.
SPP does not attempt to calculate the coincident peak demand among its member LSEs for resource adequacy purposes, but aggregates the individual non-coincident forecasts to produce a regional estimate 6 . This approach may produce a slightly higher regional peak demand, but it has three benefits. First, this approach ensures that each member LSE maintains local resource adequacy by requiring them to meet their own peak demand. Second, it avoids the complexity of allocating fractions of a regional coincident peak demand to each member LSE, which may result in a contentious process. Third, it reduces the potential cross-subsidization that may arise when a regional peak demand is not coincident with an LSE's local peak demand. This is the case, for example, for North Dakota IOUs in the MISO resource adequacy program-according to one interviewee.
The obvious drawback of a simple aggregation of peak demands is that it does not account for the temporal diversity in load profiles. A regional coincident peak demand would probably be lower than the sum of all individual peak demands. It follows that savings accruing from using a coincident peak demand approach could be substantial if there is enough temporal diversity, which could be the case for the PNW. A regional coincident demand would require NWPP to collect each LSE's hourly demand forecast for an entire year and probably use power transfer distribution factors to correlate load to a reference node. These data would most likely not come from individual IRPs, but would rather be requested from LSEs via a separate process managed by the NWPP RA Program Administrator. The hourly data would need to be consistent with the data used for standard IRP peak demand forecasts for each LSE to ensure alignment between load forecasting and published resource planning assumptions.
As with the SPP, the NWPP is also proposing separate summer and winter peak load forecasts, which may or may not coincide with IRP forecasts. While some utilities do produce summer and winter peak demand forecasts, others may only be required by their IRP regulations to produce a single-season projection. Given that the methodologies to produce single-and dual-season forecasts are very similar, it should not be hard for all LSEs in the NWPP footprint to develop winter and summer season forecasts.
Ultimately, interviewees from public utility commission staff from SPP states indicated that LSEs have an incentive to develop IRP assumptions that are consistent with SPP's in order to fulfill their membership duties. IRP guidelines in these states are generally much more broad and flexible than the IRP rules in Western U.S. states. This flexibility makes it easier for LSEs to adapt their IRP analysis to align with SPP requirements. LSEs should be able to develop NWPP-aligned forecasts as part of their IRP processes and benefit from the public stakeholder engagement as long as IRP regulations in the NWPP states are based on a broad and flexible set of principles.

Summary of findings and additional research needs
This section begins by answering the three research questions framed in the introduction. Next, it summarizes the findings more broadly and suggests methodological improvements for RA assessments. It concludes by briefly presenting additional research topics on other aspects of regional RA programs and IRP that were not examined in this report.

How would typical IRP processes change if an LSE joined a regional RA program?
IRP processes will not fundamentally change when an LSE joins a regional RA program. However, some key IRP assumptions or resource adequacy components will be impacted.
This report identifies two resource adequacy components of IRP that will be highly impacted: (1) RA targets and (2) resource capacity accreditation. Resource capacity accreditation will require more alignment between IRP and the NWPP RA program. If IRP and regional RA capacity accreditation for the same resource differ, there is a risk that an LSE would be adequate at the local-level but inadequate at the regional-level.
Two components will be moderately impacted by an LSE joining a regional RA program: (1) transmission expansion studies and (2) load forecasting. LSE transmission expansion studies typically focus on the LSE's local power system and do not assess the deliverability of resources outside of the LSE's service territory. From an IRP perspective, the main issue will be how to assure that the transmission expansion assumptions built into each IRP are consistent with the assumptions made at the regional-level for RA calculations. Load forecasting may also require moderate adjustments. Load forecasting could be delegated to individual LSEs, but the regional RA program would need to standardize its statistical methods and potentially require additional information if regional coincident peak demand were used for RA requirement calculations.
With a new regional RA program, which RA elements would remain local (i.e. within IRP) and which would become regional (i.e. within the RA program)?
Efficient and effective operation of a regional RA program will require states in the footprint to defer to the program's definitions of reliability targets and resource capacity accreditation. States would effectively integrate control over those two assumptions and let the regional program define them, incorporating them as exogenous assumptions in their IRP processes. Ongoing stakeholder engagement opportunities will be a critical component of these collaborative decision processes (see Section 7.3.3).
In addition, states will need to develop a shared agreement on the processes to produce load forecasts and to define transmission expansion studies. These elements could continue to be developed by the LSE under state IRP mandates, but coordination of input data, modeling assumptions, and outcomes will be needed with the regional RA program.

How much control would LSEs and states retain over their utility resource mixes considering the influence of a regional RA program?
The regional RA program defines the capacity needs to ensure reliability, but does not define the resources that can be employed to meet those needs. However, there is the potential indirect effect that deferring to the region on determining the capacity credit for certain resources (e.g., variable, renewable generation) could affect the calculation of an IRP's preferred resource portfolio.

Summary of key findings
This paper examined the relationship between a regional RA program and state-controlled IRP processes with a focus on areas of overlap and potential conflicts. Regulators and policy makers in states within the NWPP footprint may have concerns about their electric utilities joining the NWPP RA program if states were to lose significant control over their IRP planning process. This section seeks to highlight the key findings from this analysis and provide context that is useful for policy makers and regulators.
Target reliability metrics are RA standards, which include the PRM and the Loss of Load Probability. These standards are adopted by LSEs, states, and regional RA programs when making their respective capacity position determination. The NWPP RA program proposes to adopt the target LOLP of 1-day-in-10-years. The SPP experience provides lessons about setting a regional RA program standard. After considerable deliberation, SPP adopted a regional standard that serves as a minimum amount that member states may adopt. Member states also have the option to set a higher standard that makes their LSEs become more adequate. The process of arriving at this consensus-within SPP-is also instructive. States came to this consensus through discussions in the SPP's Regional State Committee, which is a forum for member states to provide input on SPP's planning and operations.
Resource capacity accreditation refers to the assignment of a capacity credit to a specific type of capacity resource for planning and RA purposes. Capacity credits attributed to traditional thermal resources are generally well understood and straight forward. Four other resource types require a more complex approach to attributing a capacity credit: (1) variable renewable resources, (2) demand-side resources, (3) hydropower, and (4) contracts. A growing body of research shows that the capacity contribution of wind and solar can vary across regions and diminish with increasing penetration levels of the same resource. LSEs and states currently rely on different renewable resource capacity credit methods. This difference could produce different RA outcomes between states and a regional RA program. For example, if a state grants a higher capacity credit for wind power than the RA program, then the LSE may meet the state's RA capacity standard but have a capacity deficit with the RA program. This problem can be avoided if the states and regional RA program can reach consensus around capacity contribution values. The SPP experience is, again, an informative guide. SPP states reached consensus on capacity accreditation for renewable resources and their LSE's incorporate these assumptions into their IRP-related analyses.
Transmission factors into the calculation of regional RA in terms of ensuring the deliverability of power produced by a resource will be delivered to the LSE's load that contracted for the power. SPP performs an annual deliverability study for the summer season to determine how much capacity a given resource can deliver within the SPP balancing authority area. Firm capacity is determined by each LSE based on the financial commitments as well as the deliverability of their resources. If a similar system is adopted for the NWPP RA, then the Program Administrator would need to perform a similar type of deliverability study for its LSEs members. As observed in section 6.2, there is an opportunity to improve coordinated assumptions about the transmission system and potential transmission expansion projects that would be helpful for improved coordination among LSEs that perform IRPs. A process could be developed in the NWPP to collect transmission expansion assumptions from each NWPP member based on the LSE's IRP transmission expansion assumptions.
The SPP experience shows that load forecasting can be left to the member entities in the regional program provided that they develop and disseminate a forecast with standardized statistical characteristics. The NWPP RA program proposes a set of load forecast parameters to derive peak load forecasts in the winter and summer by the participating LSEs and collectively for the region. Peak loads are based on the expected peak load in a season with a 50/50 or one-in-two probability. The NWPP RA proposes to identify the "capacity critical hours" which refers to those hours with the lowest difference between loads and capacity. There are large potential savings to be gained if the NWPP RA develops a coincident peak load forecast across the NWPP region as opposed to non-coincident peaks of member LSEs. The amount of savings will depend upon the temporal diversity of loads across the region.

Recommendations to improve RA assessments
The review of the RA assessment methods employed by LSEs in the 11 IRPs analyzed in Section 3 elucidates several common methodological shortcomings. We propose the following recommendations to improve RA assessments that apply to an LSE or a regional RA program.
First, most LSEs do not thoroughly integrate their load forecasts into RA assessments in a manner that would capture the effects of load uncertainty on reliability. As uncertainty becomes more significant and central to RA, we suggest that LSEs (and regional RA programs) move toward the use of probabilistic RA metrics that circumvent the PRM as an intermediate step and can more accurately assess the RA implications of a given portfolio of supply-and demand-side resources.
Second, very few LSEs rigorously consider uncertainty in market imports and its effect on reliability performance (see Section 3.2.2). The market sensitivity analysis developed by Avista, which computed the capacity additions required to meet LOLP targets under multiple import levels, serves as a commendable example that other utilities could follow. Regional RA programs are well situated to conduct RA analyses of their whole footprints to ensure that LSEs' market import expectations are compatible, and to identify transmission investments that would greatly benefit regional RA.
Third, many utilities are not appropriately accounting for the effects of DG and EVs on net load profiles, even though they could become significant factors in the near future. LSEs essentially consider demandside programs, DG, and EVs separately, so that important correlations among these behind-the-meter activities are likely being missed as they are incorporated into load forecasts. LSEs could improve their RA assessments by explicitly considering the adoption and electricity use behaviors of "prosumers," instead of only viewing consumers through the load forecast.

Further research
This subsection introduces four research avenues that are not covered in this report. First, the monetary benefits of the NWPP program and its comparison with costs (including changes in IRP processes) need to be comprehensively estimated. Second, a regional RA program administrator may benefit from systematic, standardized, and widespread IRP assumption and outcome collection efforts. We provide initial ideas of the costs and benefits of this effort. Third, a regional RA program and IRP have fundamentally different time horizons, but they may overlap in the short-term for IRP regulations that require short-term action plans. It is not well understood how these time frames interact. Finally, stakeholder participation from IRP processes could enhance regional RA programs accountability and transparency. We draft research required to further this approach.

Estimate the net benefits from a regional RA program
The proposed NWPP RA program will likely produce significant economic benefits to its members who are now able to meet infrequent, short-term shortfalls in RA. The NWPP RA program would have the capability to tap into a diverse set of resources across a large geographic footprint. Pooling resources allows members to have access to lower cost resources through short-term transactions than if the member had to rely solely on its own capacity resources to meet the same resource adequacy target.
Additionally, the NWPP RA program could address the current concerns that LSEs in the Northwest have been relying on wholesale contracts to meet future RA needs without assurances that the resources are available or committed to other entities during a critical shortage period. The NWPP RA program proposes to require a registration and certification process for all capacity resources in the RA program. This ensures that the Program Administrator would have accurate accounting of capacity resources in the region and promotes a more efficient allocation of the regional capacity resources to the members.
An open area of research involves answering the following questions:  What are the economic benefits of the NWPP or other proposed regional RA programs?
 How do these benefits compare to the costs borne by states that will give up control over certain components of their IRP processes?

Systematic, standardized, and widespread collection of IRP assumptions and outcomes
SPP has established several standardized information collection processes for its member entities to share data needed for transmission, wholesale market, and resource adequacy studies, among others. It has also been recognized that IRP assumptions often differ across states and that they are not always clearly communicated in the reports (Wilkerson et al., 2014).
The development of a platform that allows systematic and standardized collection of IRP assumptionsfollowing the findings in Wilkerson et al. (2014) -would have immediate application to the NWPP RA program. This platform may also implement data collection processes for information that is usually used for IRP but not explicitly reported due to its volume, such as hourly load and generation assumptions, and transmission line power flow results. Further research would be required to identify the assumptions and types of data required for the different tasks that the NWPP Program Administrator would develop. In addition, research should explore the required policy changes in IRP regulations needed to enable standardized and efficient data collection and submission processes.

Different time horizons for RA Planning and IRP planning
Figure 7.1 illustrates an LSE's hypothetical load growth curve, load growth plus PRM target, and available capacity over the 10-year timeframe. The one-to two-year timeframe encompasses operational and near-term planning that are relevant for the regional RA program. By contrast, IRP planning is relevant for the 10-to 15-year timeframe and may entail larger capital investments to increase capacity. Not surprisingly, the issues identified and decisions around the near-term RA program are very different than those of an IRP with a 10 or 15-year timeframe.
The planning time horizon proposed for the NWPP RA program is a short-term period based on compliance with a showing period and subsequent deadlines to address RA target deficiencies. LSEs that need to add capacity for the showing period or address a deficiency would likely rely on market transactions within the framework of the NWPP RA program. It is unlikely that an LSE would pursue large capacity investments in the short-term timeframe. By contrast, IRP planning involves consideration of resource portfolios that may include large capital investments to meet expected load increases over a longer time horizon (e.g., 10-15 years). However, IRP action plans may lead to procurement of new resources in the near-to mid-term timeframe of 1-3 years. It would seem reasonable that new resources procured in this timeframe would be recognized by the RA program provided the new resource start-up dates were within the timelines of the RA program. In general, decisions made in the short-term timeframe of the RA program may be disconnected from decisions over the long-term timeframe of IRP planning. It is unknown what would happen if an LSE was required to procure capacity for the RA program and this capacity did not match the levels and timing of the IRP action plan. For this reason, a number of key questions remain:  How much would short-term, RA program-related decisions influence long-term selection of the preferred portfolio in an IRP?
 What aspects of the short-term, IRP action plan would need to be consistent with potential short-term procurement strategies for showing purposes in the regional RA program?

Stakeholder participation
One of the hallmark components of IRP is its platform for engaging stakeholders in long-term planning assumptions, processes, and outcomes. In many jurisdictions, LSEs are required to conduct several public meetings to present methodologies and assumptions, respond to questions, and facilitate a dialogue around ongoing concerns. State regulators often require the LSE to respond to suggestions and report how they incorporated the stakeholder's concerns in their long-term planning processes.
Other organizations that run RA programs may be a model of stakeholder involvement for the NWPP. For example, SPP has a governance structure that relies heavily on stakeholder involvement. SPP's Regional State Committee gives state regulators direct access to the highest level of decision-making within the organization, and it hosts a critical working group called the Cost Allocation Working Group.
Moreover, SPP has over 15 working groups composed of state regulatory staff and a diverse array of members. These groups work closely together on different components of the RTO's operation. In addition, SPP has several task forces-made up of a mix of regulators, utilities, merchant operators, and customers-that were created to address specific issues. The NWPP RA program could benefit from following the stakeholder involvement model demonstrated by SPP.
Research questions to explore stakeholder participation in the nascent NWPP RA program may include:  Should IRP stakeholder involvement be merged with the RA program stakeholder involvement?
 What are the benefits of the NWPP RA program to have extensive stakeholder involvement, and what would the main working groups or task forces to be deployed?
 What role would the states play in governance of stakeholder involvement in potential NWPP RA program working group or task forces?
This paper is primarily written for state regulators, public utility commission staff, and resource planners from states in the NWPP footprint that are pondering how their IRP guidelines and regulations may need to adjust to operate jointly with a regional RA program. The content of this paper may also help the NWPP RA program developer as it interacts with potential member states and utilities to understand what aspects of energy policy may be influenced by the program under development. Finally, this paper outlines four research topics whose development may help regulators, state energy officials, and the NWPP program developer continue to refine the details for the program's design and implementation.