Thermal cameras enable rapid non-contact detection of body temperature and are widely de- ployed for fever screening, but are inaccurate in unconstrained environments. Previous works have studied the impact of ambient temperature on thermal measurements, and proposed physics-agnostic correction of environmental effects. No previous studies have considered solar loading, the increase in skin temperature due to solar radiation. Solar loading results in spurious fever detection and is skin tone dependent, introducing inequity in non-contact fever detection. We propose a method to improve fever detection by removing the solar load- ing effect from thermal images of the face. We correct solar loading using only one frame of data using a physics-informed neural network that leverages the skin temperature forward model. Our model reduces solar loading mean absolute error (MAE) by 70.5% and achieves 100% specificity in fever detection. We ensure our model is robust by collecting a diverse dataset of 100 subjects with thermal and RGB images and skin tone measurements. Our works shows that it is possible to correct complex thermal perturbations to enable robust and equitable human thermography.