Current reports of thermal expansion coefficients (TEC) of two-dimensional (2D) materials show large discrepancies that span orders of magnitude. Determining the TEC of any 2D material remains difficult due to approaches involving indirect measurement of samples that are atomically thin and optically transparent. We demonstrate a methodology to address this discrepancy and directly measure TEC of nominally monolayer epitaxial WSe2 using four-dimensional scanning transmission electron microscopy (4D-STEM). Experimentally, WSe2 from metal-organic chemical vapor deposition (MOCVD) was heated through a temperature range of 18-564 °C using a barrel-style heating sample holder to observe temperature-induced structural changes without additional alterations or destruction of the sample. By combining 4D-STEM measurements with quantitative structural analysis, the thermal expansion coefficient of nominally monolayer polycrystalline epitaxial 2D WSe2 was determined to be (3.5 ± 0.9) × 10-6 K-1 and (5.7 ± 2) × 10-5 K-1 for the in- and out-of-plane TEC, respectively, and (3.6 ± 0.2) × 10-5 K-1 for the unit cell volume TEC, in good agreement with historically determined values for bulk crystals.