Terrestrial open water evaporation is difficult to measure both in situ and remotely yet is critical for understanding changes in reservoirs, lakes, and inland seas from human management and climatically altered hydrological cycling. Multiple satellite missions and data systems (e.g., ECOSTRESS, OpenET) now operationally produce evapotranspiration (ET), but the open water evaporation data produced over millions of water bodies are algorithmically produced differently than the main ET data and are often overlooked in evaluation. Here, we evaluated the open water evaporation algorithm, AquaSEBS, used by ECOSTRESS and OpenET against 19 in situ open water evaporation sites from around the world using MODIS and Landsat data, making this one of the largest open water evaporation validations to date. Overall, our remotely sensed open water evaporation retrieval captured some variability and magnitude in the in situ data when controlling for high wind events (instantaneous: r2 = 0.71; bias = 13% of mean; RMSE = 38% of mean). Much of the instantaneous uncertainty was due to high wind events (u > mean daily 7.5 m·s-1) when the open water evaporation process shifts from radiatively-controlled to atmospherically-controlled; not accounting for high wind events decreases instantaneous accuracy significantly (r2 = 0.47; bias = 36% of mean; RMSE = 62% of mean). However, this sensitivity minimizes with temporal integration (e.g., daily RMSE = 1.2-1.5 mm·day-1). To benchmark AquaSEBS, we ran a suite of 11 machine learning models, but found that they did not significantly improve on the process-based formulation of AquaSEBS suggesting that the remaining error is from a combination of the in situ evaporation measurements, forcing data, and/or scaling mismatch; the machine learning models were able to predict error well in and of itself (r2 = 0.74). Our results provide confidence in the remotely sensed open water evaporation data, though not without uncertainty, and a foundation by which current and future missions may build such operational data.