The leading source of weather-related deaths in the United States is heat, and future projections show that the frequency, duration, and intensity of heat events will increase in the Southwest. Presently, there is a dearth of knowledge about how infrastructure may perform during heat waves or could contribute to social vulnerability. To understand how buildings perform in heat and potentially stress people, indoor air temperature changes when air conditioning is inaccessible are modeled for building archetypes in Los Angeles, California, and Phoenix, Arizona, when air conditioning is inaccessible is estimated. An energy simulation model is used to estimate how quickly indoor air temperature changes when building archetypes are exposed to extreme heat. Building age and geometry (which together determine the building envelope material composition) are found to be the strongest indicators of thermal envelope performance. Older neighborhoods in Los Angeles and Phoenix (often more centrally located in the metropolitan areas) are found to contain the buildings whose interiors warm the fastest, raising particular concern because these regions are also forecast to experience temperature increases. To combat infrastructure vulnerability and provide heat refuge for residents, incentives should be adopted to strategically retrofit buildings where both socially vulnerable populations reside and increasing temperatures are forecast.