Abstract: Residential energy efficiency programs play an important role in combating climate change. More precise quantification of the magnitude and timing of energy savings would bring large system benefits, allowing closer integration of energy efficiency into resource adequacy planning and balancing variable renewable electricity. However, it is often difficult to quantify the efficacy of an energy efficiency intervention, because doing so requires consideration of a hypothetical counterfactual case in which there was no intervention, and randomized control trials are often implausible. Although quasi-experimental econometric evaluation sometimes works well, we find that for a set of energy efficiency rebate programs in Northern California, a naïve interpretation of econometric measurement finds that rebate participation is associated with an average increase in electricity consumption of 7.2% [4.5%, 10.1%], varying in magnitude and sign depending on the type of appliance or service covered by the rebate. A subsequent household survey on appliance purchasing behavior and analysis of utility customer outreach data suggest that this regression approach is likely measuring the gross impact of buying a new appliance but fails to adequately capture a counterfactual comparison. Indeed, it is unclear whether it is even possible to construct a suitable counterfactual for econometric analyses of these rebate programs using data generally available to electric utilities. We view these results as an illustration of a limitation of econometric methods of program evaluation and the importance of weighing engineering modeling and other imperfect methods against one another when attempting to provide useful evaluations of real-world policy interventions.