EXECUTIVE SUMMARY
Expectations about technological change are central to the way policy details are set in a variety
of complex public problem areas, including health, welfare, safety, and environmental
protection. These details are established through a regulatory process designed to incorporate
scientific, technical, economic, and other expertise while following principles of good
governance such as transparency, analytical rigor, and the appropriate treatment of uncertainty.
An important element of most U.S. federal regulatory processes is cost-benefit analysis, which is
expected to employ “best-available” techniques that identify “changing future compliance costs
that might result from technological innovation or anticipated behavioral changes (OMB 2011).
Influential calls for the reform of regulatory analysis, however, locate the appropriate modeling
of technological change “at the frontier of economic research.” (Harrington, Heinzerling et al.
2009).
There are multiple reasons why it is difficult to appropriately model technical change in
regulatory analyses. First, even scholarly expectations of the influence of regulation on the rate
and direction of technical change are still strongly influenced by two opposing contentions. One
contention is that innovation can be induced to both public and private benefit by policy that is
well-designed, particularly with respect to stringency, technological neutrality, and uncertainty
(see, e.g., Porter and Van der Linde 1995, Ruttan 1997, Taylor, Rubin et al. 2005, Kemp and
Pontoglio 2011). The other contention is that policy interventions can be expected to hamper
innovation due to the influence of asymmetric information between the public and private sectors
(see, e.g., Stewart 1981). Second, although the Office of Information and Regulatory Affairs
(OIRA) in the Executive Office of Management and Budget (OMB) – the agency tasked with
providing oversight of the quality of U.S. regulatory analyses – has called for the use of best
available techniques to account for innovation in regulatory analyses, it has not suggested that
specific techniques be employed (which it has done in other areas of analytical methodology).
Third, there is no easily identifiable list of the technological change-dependent model
assumptions, inputs, and calculations used to generate estimates of the costs and benefits
associated with any given regulation. Fourth, regulated industries do not provide detailed guides
to their approaches to innovation, which vary by industry (e.g., the pharmaceutical industry does
not approach innovation the same way that the appliance industry does, although both are subject
to regulation).
Retrospective reviews of the accuracy of regulatory cost estimates point to the possibility that
there may be real value involved in trying to improve this situation. Over the past forty years,
about three-quarters of the 60+ U.S. regulatory cost estimates that have been retrospectively
reviewed have proven to be significantly inaccurate, where accuracy is defined as ex post costs
falling outside the range of +/- 25% of ex ante estimates, in keeping with a standard established
in Harrington, Morgenstern et al. (2000). As discussed in Simpson (2011), the majority of these
inaccurate regulatory cost projections are over-estimates of the costs of regulation, and
researchers have not been able to reject the null hypothesis that this robust finding is evidence of
systematic bias. The leading conjecture offered to explain this high level of cost-overestimation –
and its implicit corollary with respect to leaving societal benefits “on the table” – is that
rulemaking analyses are not currently able to accurately account for future technical change in
industries subject to regulation (Harrington, Heinzerling et al. 2009).
iii
The calls in recent presidential Executive Orders and in recommendations by the Administrative
Conference of the U.S. for periodic retrospective reviews of existing rules provide an
opportunity to start to address the situation with some rigor. The purpose of retrospective review
is to bring empirical evidence to bear on important topics that can inform the quality of future
regulatory efforts. The retrospective review project we report on in this report represents an
important first step in marshalling evidence to inform the expectations of technical change which
are embedded in the rulemaking process of a prominent area of national regulation. It takes a
layered approach to empirically considering regulation and innovation, starting with the tangible
outcomes of technical change and other factors that can be predicted through regulatory analysis
and observed in the marketplace, such as the price and quality of regulated products. It then
explores some of the subtler ways that the regulated industry’s approach to innovation may not
be reflected in the rulemaking process, to the potential detriment of analytical accuracy. In doing
this, it identifies some potentially valuable areas for future research that can inform refinements
to the rulemaking process as well as the broader scholarly debates about regulation and
innovation.
The area of regulation focused on in this project is minimum efficiency performance standards
(MEPS). Although this regulatory area has been particularly active in recent years because of its
relevance to U.S. climate change policy efforts, for the purpose of retrospective review it is best
understood in the broader historical context of U.S. energy policy. Since at least the 1970s, it has
been recognized that appliances and other energy-using products make significant contributions
to the energy consumption of the residential building sector (currently estimated at 22% of U.S.
energy use). The basic concept underlying MEPS is that it removes from the market certain
models of covered appliances and other energy-using products that do not meet specified energy
efficiency thresholds.
The covered products studied in this project are room air conditioners (“room ACs”),
refrigerator-freezers (“refrigerators”), dishwashers, clothes washers, and clothes dryers. These
five products, which heavily saturate U.S. households, are the full set of large domestic
appliances that were subject to MEPS regulatory analyses conducted by the U.S. Department of
Energy (DOE) prior to 2012. The research advantages of focusing on these appliances include
limiting selection bias, ensuring the likelihood of sufficient data for retrospective review, and
narrowing the focus to products that serve similar markets albeit with varying regulatory
histories. Note that we ground our retrospective analyses of these products in a broader market
and policy context that is informed by earlier retrospective reviews of MEPS that drew on lowerresolution
data.
The large amount of data that informs this project can be characterized as either ex ante
(expectation) data or ex post (observation data). The public rulemaking record provided a
considerable amount of useful material to inform ex ante data, both in the form of technical
support documents for specific rulemakings and petitions to reconsider final rules. Meanwhile,
the report drew upon several sources of ex post data, including: (1) extensive 2003-2011 U.S.
point-of-sale data on appliance models which we helped match to model energy use data (for all
but clothes dryers) in order to allow us to construct a monthly panel of model-specific prices,
quality characteristics, and market shares; (2) datasets we constructed from independent thirdparty
appliance testing and surveys that speak to the quality and reliability of all five products;
and (3) a dataset we constructed and coded that consists of the features identified in the product
manuals of 1,109 clothes washer models sold in the U.S. in 2003-11 (these models represented
iv
95% of the identifiable models in our point-of-sale data, which account for 29% of U.S. units
sold over that period).
The first set of analyses we report on demonstrated that purchased products in the U.S. surprised
regulatory expectations regarding cost and quality. Using the definition of significant misestimation
mentioned above, ex ante regulatory cost projections significantly over-estimated the
majority of model price observations across all efficiency levels. By product, significantly overestimated
observations were: > 95% of room AC observations; 54-66% of refrigerator
observations; 42-72% of dishwasher observations; and 50-81% of clothes washer observations
(clothes dryers had to be modeled in a different way, but the results were very similar). In
addition, we show that within-model (i.e., with fixed effects) average prices declined for the
refrigerators and clothes washers that U.S. consumers bought over the 2003-11 period.
Meanwhile, the monthly sales-weighted average energy use of these products was better than the
standard for almost all the data points of our study period, and in the one case for which we have
relevant data (clothes washers), we show that regulatory analyses underestimated how
enthusiastically U.S. consumers would buy highly efficient products. Our results show that these
better-than-expected price and efficiency outcomes did not occur to the general detriment of the
availability of products with high quality performance attributes other than energy use, despite
stakeholder concerns and the revealed concerns of the designers of the MEPS rulemaking
process in 1996. Instead, in most cases the statistically significant changes that occurred in thirdparty
quality variables assessed across MEPS events were quality improvements. Similarly, the
rate of significant repairs after five years of product ownership declined across our study period,
according to third-party surveys, which points to increased product quality over the period of
time products have been regulated by MEPS.
Although our analyses do not provide causal evidence that MEPS induced innovation to both
public and private benefit, as realized in these positive indicators of post-regulatory product
price, efficiency, and quality, our results align with this contention much more than they do the
contention that regulation hinders innovation. They also suggest that the mechanism by which
MEPS could induce innovation relates to product design. The significance of product design at
an aggregate level is apparent in our results in the stronger effect of product design versus
efficiency levels in differentiating the within-model product price declines mentioned above.
At a more disaggregate level, however, our exploratory feature-level analyses provided
important insights regarding two aspects of product design that may have relevance to the
rulemaking process followed for MEPS and to scholarship on regulation-induced innovation.
First, we identify a high degree of feature bundling in the clothes washer industry that may occur
in other appliances as well, either for technical and/or strategic reasons. Second, we observe that
product energy use is a “core” product performance attribute contributed to by many technical
features that also contribute to a range of other non-regulated product performance attributes. We
make the case that regulatory-imposed constraints on this core product performance attribute
have the potential to both reduce the search costs involved in product design while also
stimulating cascading problem-solving across technical issues related to non-regulated product
performance attributes, with the potential to enhance quality. Both insights would appear to be
promising avenues for future research to improve the MEPS rulemaking process, as preliminary
analysis does not support the idea that they are well-reflected in the current analytical process.
