Conventional calculations of heating (and cooling) loads for buildings assume that conduction heat loss (or gain) through walls is independent of air infiltration heat loss (or gain). During passage through the building envelope, infiltrating air substantially exchanges heat wall insulation leading to partial recovery of heat conducted through the wall. The Infiltration Heat Recovery (IHR) factor was introduced to quantify the heat recovery and correct the conventional calculations. In this study, Computational Fluid Dynamics was used to calculate infiltration heat recovery under a range of idealized conditions, specifically to understand factors that influence it, and assess its significance in building heat load calculations. This study shows for the first time the important effect of the external boundary layers on conduction and infiltration heat loads. Results show (under the idealized conditions studied here) that (1) the interior details of the wall encountered in the leakage path (i.e., insulated or empty walls) do not greatly influence the IHR, the overall relative location of the cracks (i.e., inlet and outlet locations on the wall) has the largest influence on the IHR magnitude, (2) external boundary layers on the walls substantially contribute to IHR and (3) the relative error in heat load calculations resulting from the use of the conventional calculational method (i.e., ignoring IHR) is between 3 percent and 13 percent for infiltrating flows typically found in residential buildings.