Energy consumption in buildings accounts for a significant portion of the global energy use. Consequently, understanding building energy use is important. Data over the past decade show that the energy intensity (Joules/sqft) of commercial buildings has decreased. While some of the improvements (decrease in energy use) are easily measurable such as the use of more energy efficient lighting, impact of other modifications such as changes to the operation of the HVAC system or changes in the usage pattern of the building potentially due to external events are difficult to quantify. Simply comparing energy consumption prior and post change is not accurate as energy use is impacted by many factors including external weather conditions. In this paper, we present a case study to quantify the impact of external events on the energy consumption of a medium-sized office building. We adopt an approach based on counterfactual analysis. Towards this end, we first build two models based on Linear Regression and k-Nearest Neighbors to predict the daily energy use given different input features related to the weather. We determine the statistical features of the weather that are most predictive of energy use. We then use the models to determine a counterfactual baseline and thereby to accurately estimate the impact of the events. The results of the counterfactual analysis provide new insights on the impact of the events on energy consumption. The update to the building cooling system resulted in more energy savings than direct yearly comparison reveals. On the other hand, the tests of a MPC-based controller for the HVAC system saved less energy than determined by the direct yearly comparison. Finally, the results show that there no gains in terms of energy savings due to remote work during the COVID-19 pandemic. An increase in airflow setting in the HVAC system corroborates this finding and further validates the underlying model and the counterfactual analyses.